From 6739339f5376046658dafd1433e786f4069ba49f Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Fri, 28 Aug 2026 22:48:38 -0500 Subject: [PATCH] Replace MinIO backup target with versitygw (s3_backup_target role) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MinIO's open-source edition is de-prioritised (console removed, commercial pivot), so swap the off-cluster S3 backup target on the Proxmox host to versitygw v1.7.0 (github.com/versity/versitygw) — a maintained Go S3 gateway serving a POSIX filesystem as S3. Rename role minio_backup_target -> s3_backup_target and playbook backup-minio -> backup-s3. The role now: - decommissions any pre-existing MinIO (service/binary/env/mc-client/user/ object-layout), KEEPING the ZFS dataset Storage/k3s-backups and its mount - installs the pinned, checksummed versitygw binary (fetched+extracted on the controller, host has no internet), a dedicated versitygw system user, and a Type=simple hardened systemd unit running the posix backend rooted at /mnt/Storage/k3s-backups/data, bound to 192.168.1.9:9000 (same endpoint), plain HTTP / path-style - ensures the dataset has xattr=sa (versitygw stores object metadata in xattrs) - pre-creates the velero + cnpg buckets (bucket == top-level subdir) Credentials are REUSED: versitygw's ROOT key is set to the same scoped access_key/secret the cluster already uses, so endpoint + key + secret + bucket names are byte-identical to the old MinIO target. The GitOps SOPS secret (platform/backup/minio-credentials.sops.yaml) and all cluster-side Velero/CNPG config need ZERO change. The vaulted group_vars now expose s3_backup_target_access_key/_secret_key (same values, MinIO root creds retired). Verified live on the Proxmox host: versitygw active+enabled on 192.168.1.9:9000, MinIO fully removed, dataset intact; SigV4 ListBuckets with the reused key returns velero+cnpg; a throwaway in-cluster pod reaches http://192.168.1.9:9000 (versitygw S3 response). Re-run is idempotent (no host changes). Co-Authored-By: Claude Opus 4.8 --- .../group_vars/proxmox_hosts/README.md | 26 +- .../group_vars/proxmox_hosts/vault.yml | 52 ++-- infra/ansible/inventory/hosts.ini | 4 +- infra/ansible/playbooks/backup-minio.yml | 20 -- infra/ansible/playbooks/backup-s3.yml | 23 ++ .../roles/minio_backup_target/README.md | 46 --- .../minio_backup_target/defaults/main.yml | 62 ----- .../roles/minio_backup_target/tasks/main.yml | 263 ------------------ .../templates/bucket-policy.json.j2 | 15 - .../templates/minio.env.j2 | 11 - .../templates/minio.service.j2 | 28 -- .../ansible/roles/s3_backup_target/README.md | 66 +++++ .../roles/s3_backup_target/defaults/main.yml | 94 +++++++ .../handlers/main.yml | 4 +- .../roles/s3_backup_target/tasks/main.yml | 231 +++++++++++++++ .../templates/versitygw.env.j2 | 10 + .../templates/versitygw.service.j2 | 35 +++ 17 files changed, 499 insertions(+), 491 deletions(-) delete mode 100644 infra/ansible/playbooks/backup-minio.yml create mode 100644 infra/ansible/playbooks/backup-s3.yml delete mode 100644 infra/ansible/roles/minio_backup_target/README.md delete mode 100644 infra/ansible/roles/minio_backup_target/defaults/main.yml delete mode 100644 infra/ansible/roles/minio_backup_target/tasks/main.yml delete mode 100644 infra/ansible/roles/minio_backup_target/templates/bucket-policy.json.j2 delete mode 100644 infra/ansible/roles/minio_backup_target/templates/minio.env.j2 delete mode 100644 infra/ansible/roles/minio_backup_target/templates/minio.service.j2 create mode 100644 infra/ansible/roles/s3_backup_target/README.md create mode 100644 infra/ansible/roles/s3_backup_target/defaults/main.yml rename infra/ansible/roles/{minio_backup_target => s3_backup_target}/handlers/main.yml (62%) create mode 100644 infra/ansible/roles/s3_backup_target/tasks/main.yml create mode 100644 infra/ansible/roles/s3_backup_target/templates/versitygw.env.j2 create mode 100644 infra/ansible/roles/s3_backup_target/templates/versitygw.service.j2 diff --git a/infra/ansible/inventory/group_vars/proxmox_hosts/README.md b/infra/ansible/inventory/group_vars/proxmox_hosts/README.md index e34877f..bc6dd42 100644 --- a/infra/ansible/inventory/group_vars/proxmox_hosts/README.md +++ b/infra/ansible/inventory/group_vars/proxmox_hosts/README.md @@ -1,29 +1,27 @@ # group_vars for the `proxmox_hosts` group (the Proxmox hypervisor host) -Target of `playbooks/backup-minio.yml` (role `minio_backup_target`). The role needs the MinIO -credentials as **ansible-vault-encrypted** vars here — encrypted, so safe to commit; the passphrase -that decrypts them stays out of the repo (`ansible.cfg` → `vault_password_file`). +Target of `playbooks/backup-s3.yml` (role `s3_backup_target`, versitygw). The role needs the S3 +backup-target credentials as **ansible-vault-encrypted** vars here — encrypted, so safe to commit; +the passphrase that decrypts them stays out of the repo (`ansible.cfg` → `vault_password_file`). ## `vault.yml` (committed, encrypted) ```bash cd infra/ansible -ansible-vault create inventory/group_vars/proxmox_hosts/vault.yml +ansible-vault edit inventory/group_vars/proxmox_hosts/vault.yml # contents: -# minio_backup_target_root_user: "…" # MinIO root (host admin only) -# minio_backup_target_root_password: "…" -# minio_backup_target_access_key: "k3s-backup" # scoped key handed to Velero + CNPG -# minio_backup_target_secret_key: "…" +# s3_backup_target_access_key: "k3s-backup" # versitygw ROOT key = key handed to Velero + CNPG +# s3_backup_target_secret_key: "…" ``` -The **scoped** access key/secret is mirrored (SOPS-encrypted) for the GitOps side at -`platform/backup/minio-credentials.sops.yaml` — the two MUST stay in sync (same key/secret), since -the cluster authenticates to this MinIO with them. Rotating means updating BOTH and re-running the -playbook (recreates the MinIO user) + reconciling Flux. +versitygw authenticates every S3 request against this single **ROOT** key. It is mirrored +(SOPS-encrypted) for the GitOps side at `platform/backup/minio-credentials.sops.yaml` — the two MUST +stay in sync (same key/secret), since the cluster authenticates to this endpoint with them. Rotating +means updating BOTH and re-running the playbook (rewrites the env file + restarts versitygw) + +reconciling Flux. ## Notes - This is the **Ansible-bootstrap-layer** secret (host credential), distinct from the platform's - in-cluster **SOPS + age** secret (ADR-0007) — different layer, different tool, same values for the - scoped key. + in-cluster **SOPS + age** secret (ADR-0007) — different layer, different tool, same values. - `vault.yml` is a vault blob (not parseable YAML) — excluded from the yamllint gate like the other `**/vault.yml`. diff --git a/infra/ansible/inventory/group_vars/proxmox_hosts/vault.yml b/infra/ansible/inventory/group_vars/proxmox_hosts/vault.yml index 9e78e81..00a4bd8 100644 --- a/infra/ansible/inventory/group_vars/proxmox_hosts/vault.yml +++ b/infra/ansible/inventory/group_vars/proxmox_hosts/vault.yml @@ -1,29 +1,25 @@ $ANSIBLE_VAULT;1.1;AES256 -30633837396565613861623931383238383836356238326436633738613935326461346138373432 -6234643535303037313337363463633765326239376238380a333331333136396239313733366163 -33353036366461393530633139636430313736366338313331353163616135303930323038636134 -6366353362313537330a373863623861663537353539376664306338643539346139396130396632 -30336231363731616334383863326638636238323565316364303530396636373163393765376438 -63373263623063656662353434623339316238663535386465653164326236616265616638656639 -30613466656439396534316339646139343334656165323965323965313361633933633038376663 -31653539653763623132313533396333643964373032313333613432383433626338376431323731 -63316163363361303439313266356361613737333066643931656564633038306236363439666435 -39343536656336343834663464356566373564636531343534343365376332343832343235663339 -66383561306533323938613738376632626263306337346532643063373166333838333033346365 -32383136363636643163613334366631653238613166303962656463363663643830343531363533 -39333630323765643963396562363537396564666462306163336366643739313035636633646661 -61653637376535393034383631303563363539633265333161363334616239323866353162636632 -65663830323465393138306361333662383934346438663562316235353035633064643861353334 -63646435653639316439366139316539653833656561646439323766633139306565383565656261 -61323230306635343236646438336165643737626132613962633930653461393131303731623438 -63313265653166643334396461353231343235386434363438623034323762653433303932326630 -62633464316139353433366338313837646162323231363135353964623035366132643332366662 -38393465326363616139376465383738626636393639363338653437323832613439313138353033 -39393765613735666331636230363463306462373238623138316263326339366330333332643833 -33363764303939326662343665643963333463356638613231363834386132666339393132356538 -38303362643337366430303165633432666632343536383366653836663135383336316232393936 -66626632386537333739313432313564633866383132353862626433626366633562346464613066 -35376332303132326438313132303264313763376163396137356132306131373335346266646532 -31666230616461386339366461653661323138353662373866316639626134326339643231613337 -30343361643166353162363866653061313966663435373332353562656164663962326562333038 -35363834656361393930 +33623463393666373861613531626336396164356361643939326332626634663234653164313437 +3963326465326364643639633066636564656565333461360a383438323530663932326462376661 +64343766366132373436366336353338383365396563636138333333306330643337636532383662 +6561323836656637380a656436656538613263646461386635303762376163363161336436386263 +34653437386364363161653132343434333230393732373364383564336162643136636437376231 +39376463336666396462363431636161306632326134663038343364626232333838333039393338 +64313762343363666139393864366561316362633465323361633132373533356531353534396433 +62333131356630343764303438663766623737333137383264623739646539373535306136303530 +34663063383032643238373537663633343735613265653964356638306566636538363063656435 +65643666366665393238363037313337633366333338633163316533656134336463363630353961 +37656637343063373266643237663633613638303165656262313133313535313665336335333039 +30626537373530663430343637386263353765353538376263333130623536306635353966393561 +62313362373061303035376635646335303133613365326163643566333031356436326464616430 +34373433343733386666626234623064383063393330343034363036663632373865643636623961 +31666131636538303131376339623733353964653266356339313466616363646336663433666631 +61663564633931626639323235633332366338333837316535376330633534626537666166643262 +63646239656165303366626163613238393266336566316164356330393836323332336330633635 +64636636613164366338396666316137353837313639323630353234373936393335366364666361 +35356163363636623031333733653432636539396235633436313334333663343532303239373238 +37646230326164396339386339356663396661306165343630316261306331383632626662323234 +31396539313766356633333736383036623839626663383763626332626237373338653832646463 +34366434386139306336633964313939643565363838346163656236656561363865386136346530 +36316661323262323366646332316535306637316663656264303137653831656136633432643765 +61646433653530646339 diff --git a/infra/ansible/inventory/hosts.ini b/infra/ansible/inventory/hosts.ini index 9543573..198accc 100644 --- a/infra/ansible/inventory/hosts.ini +++ b/infra/ansible/inventory/hosts.ini @@ -38,8 +38,8 @@ localhost ansible_connection=local [vpn_egress_gateway] k3s-dev-3 -# The Proxmox HYPERVISOR host itself (NOT a k3s node) — target of playbooks/backup-minio.yml, -# which stands up the off-cluster MinIO S3 backup target for Velero + CNPG. Reached DIRECTLY +# The Proxmox HYPERVISOR host itself (NOT a k3s node) — target of playbooks/backup-s3.yml, +# which stands up the off-cluster versitygw S3 backup target for Velero + CNPG. Reached DIRECTLY # over SSH as root (its own 192.168.2.102:22, the pfSense NAT to the host's 192.168.1.9), so # it overrides the cluster-wide `ansible_user: k3s` from group_vars/all.yml. Distinct from the # `hypervisor` group above (that is `localhost` for host-prereqs.yml, a different machine). diff --git a/infra/ansible/playbooks/backup-minio.yml b/infra/ansible/playbooks/backup-minio.yml deleted file mode 100644 index 3cbe13b..0000000 --- a/infra/ansible/playbooks/backup-minio.yml +++ /dev/null @@ -1,20 +0,0 @@ ---- -# Stand up the off-cluster S3 backup target (MinIO) on the Proxmox host — the store Velero -# (R-STATE-03) and CloudNativePG (R-IDP-05) back up into. Host infra, additive + idempotent: -# ONE new ZFS dataset (Storage/k3s-backups), the minio user/service/binaries, and the -# velero + cnpg buckets with a scoped non-root access key. It never touches existing -# datasets, sharenfs, pvesm storage, VMs, or firewall. -# -# Credentials come from Ansible vault (inventory/group_vars/proxmox_hosts/vault.yml): -# minio_backup_target_root_user/_root_password, minio_backup_target_access_key/_secret_key -# The scoped key is ALSO staged for the GitOps side (SOPS) at -# platform/backup/minio-credentials.sops.yaml -# -# Prereqs: ansible-galaxy collection install -r collections/requirements.yml (community.general.zfs) -# Run: ansible-playbook playbooks/backup-minio.yml -- name: Provision the MinIO S3 backup target on the Proxmox host - hosts: proxmox_hosts - become: true - gather_facts: true # need ansible_facts.all_ipv4_addresses to discover the LAN bind addr - roles: - - role: minio_backup_target diff --git a/infra/ansible/playbooks/backup-s3.yml b/infra/ansible/playbooks/backup-s3.yml new file mode 100644 index 0000000..12e5e78 --- /dev/null +++ b/infra/ansible/playbooks/backup-s3.yml @@ -0,0 +1,23 @@ +--- +# Stand up the off-cluster S3 backup target on the Proxmox host — the store Velero (R-STATE-03) +# and CloudNativePG (R-IDP-05) back up into. Implemented with versitygw (Versity Gateway), which +# REPLACES the earlier MinIO target; the role also decommissions any pre-existing MinIO install so +# a re-run is a clean swap. Host infra, surgical + idempotent: the ONE ZFS dataset it owns +# (Storage/k3s-backups), the versitygw user/service/binary, and the velero + cnpg buckets. It never +# touches other datasets, sharenfs, pvesm storage, VMs or firewall. +# +# Credentials come from Ansible vault (inventory/group_vars/proxmox_hosts/vault.yml): +# s3_backup_target_access_key / _secret_key (versitygw ROOT key) +# The SAME key is staged for the GitOps side (SOPS) at +# platform/backup/minio-credentials.sops.yaml +# Endpoint + key + secret + bucket names are byte-identical to the old MinIO target, so the +# cluster-side Velero/CNPG config + the SOPS secret need ZERO change. +# +# Prereqs: ansible-galaxy collection install -r collections/requirements.yml (community.general.zfs) +# Run: ansible-playbook playbooks/backup-s3.yml +- name: Provision the versitygw S3 backup target on the Proxmox host + hosts: proxmox_hosts + become: true + gather_facts: true # need ansible_facts.all_ipv4_addresses to discover the LAN bind addr + roles: + - role: s3_backup_target diff --git a/infra/ansible/roles/minio_backup_target/README.md b/infra/ansible/roles/minio_backup_target/README.md deleted file mode 100644 index 809ec97..0000000 --- a/infra/ansible/roles/minio_backup_target/README.md +++ /dev/null @@ -1,46 +0,0 @@ -# roles/minio_backup_target - -Stands up a **single-node, single-drive MinIO** S3 endpoint on the **Proxmox host** to serve as the -off-cluster **backup target** for the k3s platform's **Velero** (volume/resource DR, R-STATE-03) and -**CloudNativePG** (Keycloak DB Barman object store, R-IDP-05). A backup target — not primary storage — -so a single drive is intentional. - -## What it does (all additive + idempotent) - -1. **ZFS dataset** `Storage/k3s-backups` — `compression=zstd`, `quota=100G`, - `mountpoint=/mnt/Storage/k3s-backups`, **no sharenfs**. Uses `community.general.zfs` so re-runs are - no-ops and existing datasets are never touched. -2. **`minio` system user/group** + a data dir (`…/k3s-backups/data`) it owns. -3. **MinIO + `mc` binaries** (pinned, checksummed). The Proxmox host has **no outbound internet**, so - the binaries are downloaded on the **Ansible controller** and copied to the host. -4. **systemd unit** `minio.service` (Type=notify, hardened), bound to the host's - **cluster-reachable `192.168.1.x` address** on port **9000** (console 9001) — auto-discovered from - facts, assert-guarded. -5. **Buckets** `velero` + `cnpg`, a **scoped non-root access key** limited by an IAM policy to just - those two buckets, plus the retained root creds for host-side admin. - -## Credentials - -Supplied via the vaulted `inventory/group_vars/proxmox_hosts/vault.yml` (ansible-vault). Nothing is -defaulted — a missing vault fails the play loudly: - -| var | purpose | -|---|---| -| `minio_backup_target_root_user` / `_root_password` | MinIO root (host admin only) | -| `minio_backup_target_access_key` / `_secret_key` | scoped key handed to Velero + CNPG | - -The **same scoped key** is staged for the GitOps side, SOPS-encrypted, at -`platform/backup/minio-credentials.sops.yaml`. - -## Run - -```bash -cd infra/ansible -ansible-galaxy collection install -r collections/requirements.yml # community.general (zfs) -ansible-playbook playbooks/backup-minio.yml -``` - -## Endpoint - -`http://:9000` — the S3 URL the cluster uses (path-style). Health: -`GET /minio/health/ready` ⇒ 200. Verified reachable from an in-cluster pod at bring-up. diff --git a/infra/ansible/roles/minio_backup_target/defaults/main.yml b/infra/ansible/roles/minio_backup_target/defaults/main.yml deleted file mode 100644 index 0a5bab6..0000000 --- a/infra/ansible/roles/minio_backup_target/defaults/main.yml +++ /dev/null @@ -1,62 +0,0 @@ ---- -# roles/minio_backup_target — a single-node, single-drive MinIO S3 target on the Proxmox -# host, used ONLY as an off-cluster backup bucket store for Velero + CloudNativePG (ADR-0003, -# R-STATE-03 / R-IDP-05). This is a BACKUP TARGET, not primary storage — single drive is fine. -# -# SAFETY: every task in this role is additive and idempotent. It creates ONE new ZFS dataset, -# ONE system user, ONE systemd service and two buckets. It never touches existing datasets, -# sharenfs, pvesm storage, VMs or firewall. See tasks/main.yml for the guards. - -# ── MinIO / mc release pins (Renovate-friendly; both are AGPL community builds) ─────────────── -# The Proxmox host has NO outbound internet (PiHole-only DNS, no default route to WAN), so the -# binaries are fetched on the ANSIBLE CONTROLLER and copied to the host (tasks/main.yml). -minio_backup_target_minio_version: "RELEASE.2025-09-07T16-13-09Z" -minio_backup_target_minio_sha256: "7c5bd8512c6e966455b1d198209358b2d191c77a83ab377c4073281065fb855f" -minio_backup_target_minio_url: "https://dl.min.io/server/minio/release/linux-amd64/archive/minio.{{ minio_backup_target_minio_version }}" - -minio_backup_target_mc_version: "RELEASE.2025-08-13T08-35-41Z" -minio_backup_target_mc_sha256: "01f866e9c5f9b87c2b09116fa5d7c06695b106242d829a8bb32990c00312e891" -minio_backup_target_mc_url: "https://dl.min.io/client/mc/release/linux-amd64/archive/mc.{{ minio_backup_target_mc_version }}" - -# Controller-side cache the binaries land in before being copied to the host (delegate_to localhost). -minio_backup_target_controller_cache: "/tmp/minio-backup-target-cache" - -# ── ZFS dataset (backup drive) ──────────────────────────────────────────────────────────────── -# Host convention is Storage/ mounted at /mnt/Storage/. NO sharenfs. -minio_backup_target_zfs_dataset: "Storage/k3s-backups" -minio_backup_target_mountpoint: "/mnt/Storage/k3s-backups" -minio_backup_target_zfs_compression: "zstd" -minio_backup_target_zfs_quota: "100G" -# MinIO writes its drive (incl. .minio.sys) into a subdir of the dataset, keeping the dataset -# root clean and owned by root. -minio_backup_target_data_dir: "{{ minio_backup_target_mountpoint }}/data" - -# ── Service identity ──────────────────────────────────────────────────────────────────────── -minio_backup_target_user: "minio" -minio_backup_target_group: "minio" -minio_backup_target_minio_bin: "/usr/local/bin/minio" -minio_backup_target_mc_bin: "/usr/local/bin/mc" -minio_backup_target_env_file: "/etc/default/minio" - -# ── Network binding ────────────────────────────────────────────────────────────────────────── -# MinIO's S3 API must bind the host address the k3s pods can reach (the 192.168.1.0/24 LAN; -# nodes are .50/.51/.52). tasks/main.yml auto-discovers the host's address in this CIDR prefix -# and asserts it exists; override minio_backup_target_bind_address to pin it explicitly. -minio_backup_target_lan_prefix: "192.168.1." -minio_backup_target_bind_address: "" # "" ⇒ auto-discover from facts (see tasks) -minio_backup_target_api_port: 9000 -minio_backup_target_console_port: 9001 - -# ── Buckets + scoped (non-root) access key ──────────────────────────────────────────────────── -# Velero and CNPG each get a bucket; a single non-root user scoped to ONLY these two buckets is -# handed to the cluster (SOPS). Root creds stay on the host for admin. -minio_backup_target_buckets: - - velero - - cnpg -minio_backup_target_policy_name: "k3s-backup-rw" - -# Credentials come from the vaulted group_vars (inventory/group_vars/proxmox_hosts/vault.yml): -# minio_backup_target_root_user, minio_backup_target_root_password, -# minio_backup_target_access_key, minio_backup_target_secret_key -# They are intentionally NOT defaulted here so a missing vault fails loudly rather than -# silently provisioning guessable creds. diff --git a/infra/ansible/roles/minio_backup_target/tasks/main.yml b/infra/ansible/roles/minio_backup_target/tasks/main.yml deleted file mode 100644 index 69220b7..0000000 --- a/infra/ansible/roles/minio_backup_target/tasks/main.yml +++ /dev/null @@ -1,263 +0,0 @@ ---- -# ── 0. Preconditions / safety asserts ───────────────────────────────────────────────────────── -- name: Assert MinIO credentials are supplied (from vaulted group_vars) - ansible.builtin.assert: - that: - - minio_backup_target_root_user | default('') | length > 0 - - minio_backup_target_root_password | default('') | length >= 8 - - minio_backup_target_access_key | default('') | length > 0 - - minio_backup_target_secret_key | default('') | length >= 8 - fail_msg: >- - MinIO credentials are missing. Provide minio_backup_target_root_user/_password and - minio_backup_target_access_key/_secret_key via inventory/group_vars/proxmox_hosts/vault.yml. - quiet: true - -# Determine the host's 192.168.1.0/24 address (the S3 endpoint the cluster reaches). If the -# operator pinned minio_backup_target_bind_address, honour it; else auto-discover and assert. -- name: Discover the host LAN (cluster-reachable) IPv4 address - ansible.builtin.set_fact: - minio_backup_target_bind_address: >- - {{ minio_backup_target_bind_address - if (minio_backup_target_bind_address | length > 0) - else (ansible_facts.all_ipv4_addresses - | select('match', '^' ~ (minio_backup_target_lan_prefix | regex_escape)) - | list | first | default('')) }} - -- name: Assert a cluster-reachable LAN address was found - ansible.builtin.assert: - that: - - minio_backup_target_bind_address | length > 0 - fail_msg: >- - Could not find a {{ minio_backup_target_lan_prefix }}x address on this host - (interfaces: {{ ansible_facts.all_ipv4_addresses | default([]) | join(', ') }}). - The cluster egresses onto that LAN — refusing to bind MinIO to an unreachable address. - Set minio_backup_target_bind_address explicitly if discovery is wrong. - quiet: true - -# ── 1. ZFS dataset (backup drive) — idempotent, never touches other datasets ─────────────────── -- name: Create/ensure the k3s-backups ZFS dataset (compression, quota, mountpoint; NO sharenfs) - community.general.zfs: - name: "{{ minio_backup_target_zfs_dataset }}" - state: present - extra_zfs_properties: - compression: "{{ minio_backup_target_zfs_compression }}" - quota: "{{ minio_backup_target_zfs_quota }}" - mountpoint: "{{ minio_backup_target_mountpoint }}" - -# ── 2. Service identity ─────────────────────────────────────────────────────────────────────── -- name: Create the minio system group - ansible.builtin.group: - name: "{{ minio_backup_target_group }}" - system: true - -- name: Create the minio system user (no login shell, home = data dir) - ansible.builtin.user: - name: "{{ minio_backup_target_user }}" - group: "{{ minio_backup_target_group }}" - system: true - shell: /usr/sbin/nologin - home: "{{ minio_backup_target_data_dir }}" - create_home: false - -- name: Ensure the MinIO data directory exists (owned by minio, inside the dataset) - ansible.builtin.file: - path: "{{ minio_backup_target_data_dir }}" - state: directory - owner: "{{ minio_backup_target_user }}" - group: "{{ minio_backup_target_group }}" - mode: "0750" - -# ── 3. Binaries — fetched on the CONTROLLER (host has no internet), copied to the host ────────── -- name: Ensure controller-side download cache exists - ansible.builtin.file: - path: "{{ minio_backup_target_controller_cache }}" - state: directory - mode: "0700" - delegate_to: localhost - become: false - run_once: true - -- name: Fetch MinIO server binary onto the controller (checksummed) - ansible.builtin.get_url: - url: "{{ minio_backup_target_minio_url }}" - dest: "{{ minio_backup_target_controller_cache }}/minio.{{ minio_backup_target_minio_version }}" - checksum: "sha256:{{ minio_backup_target_minio_sha256 }}" - mode: "0755" - delegate_to: localhost - become: false - run_once: true - -- name: Fetch mc client binary onto the controller (checksummed) - ansible.builtin.get_url: - url: "{{ minio_backup_target_mc_url }}" - dest: "{{ minio_backup_target_controller_cache }}/mc.{{ minio_backup_target_mc_version }}" - checksum: "sha256:{{ minio_backup_target_mc_sha256 }}" - mode: "0755" - delegate_to: localhost - become: false - run_once: true - -# Integrity is already guaranteed by the SHA256 verification get_url performed on the controller -# above; copy transfers that verified artifact faithfully. (The copy module's own `checksum` arg is -# SHA1, not SHA256 — deliberately not used here to avoid a false mismatch.) -- name: Install MinIO server binary on the host - ansible.builtin.copy: - src: "{{ minio_backup_target_controller_cache }}/minio.{{ minio_backup_target_minio_version }}" - dest: "{{ minio_backup_target_minio_bin }}" - owner: root - group: root - mode: "0755" - notify: Restart minio - -- name: Install mc client binary on the host - ansible.builtin.copy: - src: "{{ minio_backup_target_controller_cache }}/mc.{{ minio_backup_target_mc_version }}" - dest: "{{ minio_backup_target_mc_bin }}" - owner: root - group: root - mode: "0755" - -# ── 4. systemd service ──────────────────────────────────────────────────────────────────────── -- name: Write the MinIO environment file (root creds, bind address, data dir) - ansible.builtin.template: - src: minio.env.j2 - dest: "{{ minio_backup_target_env_file }}" - owner: root - group: "{{ minio_backup_target_group }}" - mode: "0640" - no_log: true - notify: Restart minio - -- name: Install the MinIO systemd unit - ansible.builtin.template: - src: minio.service.j2 - dest: /etc/systemd/system/minio.service - owner: root - group: root - mode: "0644" - notify: Restart minio - -- name: Enable + start MinIO (apply any pending unit/env changes now) - ansible.builtin.systemd: - name: minio - enabled: true - state: started - daemon_reload: true - -- name: Flush handlers so MinIO is (re)started before we configure buckets/users - ansible.builtin.meta: flush_handlers - -# ── 5. Readiness gate ───────────────────────────────────────────────────────────────────────── -- name: Wait for the MinIO S3 health endpoint to report ready - ansible.builtin.uri: - url: "http://{{ minio_backup_target_bind_address }}:{{ minio_backup_target_api_port }}/minio/health/ready" - status_code: 200 - register: minio_backup_target_health - retries: 30 - delay: 2 - until: minio_backup_target_health.status == 200 - -# ── 6. Buckets + scoped non-root user (via mc, idempotent) ───────────────────────────────────── -- name: Configure the local mc alias against the running MinIO - ansible.builtin.command: - argv: - - "{{ minio_backup_target_mc_bin }}" - - alias - - set - - k3sbackup - - "http://{{ minio_backup_target_bind_address }}:{{ minio_backup_target_api_port }}" - - "{{ minio_backup_target_root_user }}" - - "{{ minio_backup_target_root_password }}" - no_log: true - changed_when: false - -- name: Create the Velero + CNPG buckets (idempotent) - ansible.builtin.command: - argv: - - "{{ minio_backup_target_mc_bin }}" - - mb - - --ignore-existing - - "k3sbackup/{{ item }}" - loop: "{{ minio_backup_target_buckets }}" - register: minio_backup_target_mb - changed_when: "'created successfully' in (minio_backup_target_mb.stdout | default(''))" - -- name: Render the scoped bucket RW policy to the host - ansible.builtin.template: - src: bucket-policy.json.j2 - dest: "/etc/minio-{{ minio_backup_target_policy_name }}.json" - owner: root - group: "{{ minio_backup_target_group }}" - mode: "0640" - -- name: Check whether the scoped policy already exists - ansible.builtin.command: - argv: - - "{{ minio_backup_target_mc_bin }}" - - admin - - policy - - info - - k3sbackup - - "{{ minio_backup_target_policy_name }}" - register: minio_backup_target_policy_info - changed_when: false - failed_when: false - -- name: Create the scoped bucket RW policy (velero + cnpg only) - ansible.builtin.command: - argv: - - "{{ minio_backup_target_mc_bin }}" - - admin - - policy - - create - - k3sbackup - - "{{ minio_backup_target_policy_name }}" - - "/etc/minio-{{ minio_backup_target_policy_name }}.json" - when: minio_backup_target_policy_info.rc != 0 - changed_when: true - -- name: Check whether the scoped access key already exists - ansible.builtin.command: - argv: - - "{{ minio_backup_target_mc_bin }}" - - admin - - user - - info - - k3sbackup - - "{{ minio_backup_target_access_key }}" - register: minio_backup_target_user_info - changed_when: false - failed_when: false - -- name: Create the scoped non-root access key - ansible.builtin.command: - argv: - - "{{ minio_backup_target_mc_bin }}" - - admin - - user - - add - - k3sbackup - - "{{ minio_backup_target_access_key }}" - - "{{ minio_backup_target_secret_key }}" - when: minio_backup_target_user_info.rc != 0 - no_log: true - changed_when: true - -- name: Attach the scoped policy to the access key (idempotent) - ansible.builtin.command: - argv: - - "{{ minio_backup_target_mc_bin }}" - - admin - - policy - - attach - - k3sbackup - - "{{ minio_backup_target_policy_name }}" - - --user - - "{{ minio_backup_target_access_key }}" - register: minio_backup_target_attach - changed_when: - - minio_backup_target_attach.rc == 0 - - "'already in effect' not in (minio_backup_target_attach.stderr | default(''))" - failed_when: - - minio_backup_target_attach.rc != 0 - - "'already in effect' not in (minio_backup_target_attach.stderr | default(''))" diff --git a/infra/ansible/roles/minio_backup_target/templates/bucket-policy.json.j2 b/infra/ansible/roles/minio_backup_target/templates/bucket-policy.json.j2 deleted file mode 100644 index c29d70d..0000000 --- a/infra/ansible/roles/minio_backup_target/templates/bucket-policy.json.j2 +++ /dev/null @@ -1,15 +0,0 @@ -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": ["s3:*"], - "Resource": [ -{% for b in minio_backup_target_buckets %} - "arn:aws:s3:::{{ b }}", - "arn:aws:s3:::{{ b }}/*"{{ "," if not loop.last else "" }} -{% endfor %} - ] - } - ] -} diff --git a/infra/ansible/roles/minio_backup_target/templates/minio.env.j2 b/infra/ansible/roles/minio_backup_target/templates/minio.env.j2 deleted file mode 100644 index cf82a4f..0000000 --- a/infra/ansible/roles/minio_backup_target/templates/minio.env.j2 +++ /dev/null @@ -1,11 +0,0 @@ -# {{ ansible_managed }} -# MinIO server environment (single-node, single-drive backup target for k3s Velero + CNPG). -# Managed by roles/minio_backup_target — do not edit by hand. -MINIO_ROOT_USER={{ minio_backup_target_root_user }} -MINIO_ROOT_PASSWORD={{ minio_backup_target_root_password }} - -# The S3 drive lives on the ZFS dataset {{ minio_backup_target_zfs_dataset }}. -MINIO_VOLUMES="{{ minio_backup_target_data_dir }}" - -# Bind the S3 API (and console) to the cluster-reachable LAN address only. -MINIO_OPTS="--address {{ minio_backup_target_bind_address }}:{{ minio_backup_target_api_port }} --console-address {{ minio_backup_target_bind_address }}:{{ minio_backup_target_console_port }}" diff --git a/infra/ansible/roles/minio_backup_target/templates/minio.service.j2 b/infra/ansible/roles/minio_backup_target/templates/minio.service.j2 deleted file mode 100644 index 89fcbd9..0000000 --- a/infra/ansible/roles/minio_backup_target/templates/minio.service.j2 +++ /dev/null @@ -1,28 +0,0 @@ -{# {{ ansible_managed }} — MinIO systemd unit (roles/minio_backup_target) #} -[Unit] -Description=MinIO S3 backup target (k3s Velero + CNPG) -Documentation=https://min.io/docs/minio/linux/index.html -After=network-online.target -Wants=network-online.target -AssertFileIsExecutable={{ minio_backup_target_minio_bin }} - -[Service] -Type=notify -User={{ minio_backup_target_user }} -Group={{ minio_backup_target_group }} -EnvironmentFile={{ minio_backup_target_env_file }} -ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo 'MINIO_VOLUMES unset in {{ minio_backup_target_env_file }}'; exit 1; fi" -ExecStart={{ minio_backup_target_minio_bin }} server $MINIO_OPTS $MINIO_VOLUMES -Restart=always -RestartSec=5s -LimitNOFILE=65536 -TimeoutStopSec=30s -# Hardening (backup target holds no other roles on this host). -NoNewPrivileges=true -ProtectSystem=full -ProtectHome=true -PrivateTmp=true -ReadWritePaths={{ minio_backup_target_mountpoint }} - -[Install] -WantedBy=multi-user.target diff --git a/infra/ansible/roles/s3_backup_target/README.md b/infra/ansible/roles/s3_backup_target/README.md new file mode 100644 index 0000000..f6b0792 --- /dev/null +++ b/infra/ansible/roles/s3_backup_target/README.md @@ -0,0 +1,66 @@ +# roles/s3_backup_target + +Stands up a **single-node S3 endpoint** on the **Proxmox host** to serve as the off-cluster +**backup target** for the k3s platform's **Velero** (volume/resource DR, R-STATE-03) and +**CloudNativePG** (Keycloak DB Barman object store, R-IDP-05). A backup target — not primary +storage — so a single dataset is intentional. + +Implemented with **[versitygw](https://github.com/versity/versitygw)** (Versity Gateway): a +maintained Go S3 gateway that serves a POSIX filesystem as S3. It **replaces the earlier MinIO +implementation** (MinIO's community edition was de-prioritised — console removed, commercial +pivot). The role also **decommissions any pre-existing MinIO** install on the host, so a re-run is +a clean, idempotent swap. Endpoint, access key, secret and bucket names are kept **byte-identical** +to the previous MinIO target, so nothing downstream (the SOPS secret, Velero, CNPG) changes. + +## What it does (all additive + idempotent, surgical) + +1. **Decommission MinIO** (tolerant no-op on a fresh host): stop/disable/remove `minio.service`, + the `minio` + `mc` binaries, `/etc/default/minio`, the IAM policy file, the `mc` client config, + the `minio` user/group, and MinIO's on-disk object layout (`.minio`, `.minio.sys`) inside the + gateway-root dir. **The ZFS dataset and mount are KEPT.** +2. **ZFS dataset** `Storage/k3s-backups` — `compression=zstd`, `quota=100G`, **`xattr=sa`** + (required — versitygw stores object metadata in POSIX extended attributes and refuses to start + without them), `mountpoint=/mnt/Storage/k3s-backups`, **no sharenfs**. Uses `community.general.zfs` + so re-runs are no-ops and existing datasets are never touched. +3. **`versitygw` system user/group** + the gateway-root dir (`…/k3s-backups/data`) it owns. +4. **versitygw binary** (pinned, checksummed). The Proxmox host has **no outbound internet**, so the + release tarball is downloaded + verified + extracted on the **Ansible controller** and only the + binary is copied to the host. +5. **systemd unit** `versitygw.service` (Type=simple, hardened), `posix` backend rooted at the + gateway root, bound to the host's **cluster-reachable `192.168.1.x` address** on port **9000** + (same endpoint as the old MinIO) — auto-discovered from facts, assert-guarded. TLS off (plain + HTTP, path-style). +6. **Buckets** `velero` + `cnpg` — on the posix backend a bucket is simply a top-level subdirectory + of the gateway root, so they are pre-created owned by the service user. + +## Credentials + +versitygw authenticates every S3 request against a single **ROOT** access key/secret +(`ROOT_ACCESS_KEY` / `ROOT_SECRET_KEY`). This role sets it to the **same** scoped key/secret the +cluster already uses, supplied via the vaulted `inventory/group_vars/proxmox_hosts/vault.yml` +(ansible-vault). Nothing is defaulted — a missing vault fails the play loudly: + +| var | purpose | +|---|---| +| `s3_backup_target_access_key` / `_secret_key` | versitygw ROOT key = the key handed to Velero + CNPG | + +The **same** key/secret is staged for the GitOps side, SOPS-encrypted, at +`platform/backup/minio-credentials.sops.yaml` — the two MUST stay in sync. + +> Note: unlike MinIO, versitygw's single-binary posix backend has no separate "root admin" vs +> "scoped user" distinction without enabling an external IAM backend. For a dedicated backup target +> the scoped key **is** the root key — it only ever fronts the velero + cnpg buckets on this host. + +## Run + +```bash +cd infra/ansible +ansible-galaxy collection install -r collections/requirements.yml # community.general (zfs) +ansible-playbook playbooks/backup-s3.yml +``` + +## Endpoint + +`http://:9000` — the S3 URL the cluster uses (path-style). An unauthenticated +`GET /` returns `403 AccessDenied` once serving; a SigV4 `ListBuckets` with the root key returns +`velero` + `cnpg`. Verified reachable from an in-cluster pod at bring-up. diff --git a/infra/ansible/roles/s3_backup_target/defaults/main.yml b/infra/ansible/roles/s3_backup_target/defaults/main.yml new file mode 100644 index 0000000..c55140e --- /dev/null +++ b/infra/ansible/roles/s3_backup_target/defaults/main.yml @@ -0,0 +1,94 @@ +--- +# roles/s3_backup_target — a single-node S3 backup endpoint on the Proxmox host, used ONLY as an +# off-cluster backup bucket store for Velero + CloudNativePG (ADR-0003, R-STATE-03 / R-IDP-05). +# This is a BACKUP TARGET, not primary storage — a single drive/dataset is intentional. +# +# Implemented with versitygw (Versity Gateway, github.com/versity/versitygw): a maintained Go S3 +# gateway that serves a POSIX filesystem as S3. It REPLACES the earlier MinIO implementation (MinIO's +# community edition was de-prioritised); the role also decommissions any pre-existing MinIO install +# on the host so a re-run is a clean swap. +# +# SAFETY: every task in this role is additive/idempotent and SURGICAL. It touches ONLY the +# minio→versitygw swap: the ONE ZFS dataset it owns (Storage/k3s-backups), the versitygw system +# user, its systemd service + binary, and the two buckets inside that dataset. It never touches +# other datasets, sharenfs, pvesm storage, VMs or firewall. See tasks/main.yml for the guards. + +# ── versitygw release pin (Renovate-friendly) ───────────────────────────────────────────────── +# The Proxmox host has NO outbound internet (PiHole-only DNS, no default route to WAN), so the +# release tarball is fetched + verified on the ANSIBLE CONTROLLER, the binary extracted there, and +# only the binary is copied to the host (tasks/main.yml). +# Release assets: https://github.com/versity/versitygw/releases +# tarball : versitygw_v{ver}_Linux_x86_64.tar.gz (extracts to versitygw_v{ver}_Linux_x86_64/versitygw) +# checksums: checksums.txt (sha256) in the same release +s3_backup_target_versitygw_version: "v1.7.0" +s3_backup_target_versitygw_sha256: "dacec530fecaa8776feec0536998dd4bd78036e0d4db0da9b9189d96bb790e03" +s3_backup_target_versitygw_url: "https://github.com/versity/versitygw/releases/download/{{ s3_backup_target_versitygw_version }}/versitygw_{{ s3_backup_target_versitygw_version }}_Linux_x86_64.tar.gz" +# Path of the binary INSIDE the extracted tarball (GoReleaser lays it out under a version dir). +s3_backup_target_versitygw_archive_member: "versitygw_{{ s3_backup_target_versitygw_version }}_Linux_x86_64/versitygw" + +# Controller-side cache the tarball is fetched + extracted into (delegate_to localhost). +s3_backup_target_controller_cache: "/tmp/s3-backup-target-cache" + +# ── ZFS dataset (backup drive) ──────────────────────────────────────────────────────────────── +# Host convention is Storage/ mounted at /mnt/Storage/. NO sharenfs. +# xattr=sa is REQUIRED: versitygw's posix backend stores object metadata (ETags, etc.) in POSIX +# extended attributes and refuses to start if the root FS can't hold them. +s3_backup_target_zfs_dataset: "Storage/k3s-backups" +s3_backup_target_mountpoint: "/mnt/Storage/k3s-backups" +s3_backup_target_zfs_compression: "zstd" +s3_backup_target_zfs_quota: "100G" +s3_backup_target_zfs_xattr: "sa" +# versitygw's gateway root — a subdir of the dataset, keeping the dataset root owned by root and +# clean. Its top-level subdirectories ARE the S3 buckets. +s3_backup_target_gateway_root: "{{ s3_backup_target_mountpoint }}/data" + +# ── Service identity ──────────────────────────────────────────────────────────────────────── +s3_backup_target_user: "versitygw" +s3_backup_target_group: "versitygw" +s3_backup_target_bin: "/usr/local/bin/versitygw" +s3_backup_target_env_file: "/etc/default/versitygw" + +# ── Network binding ────────────────────────────────────────────────────────────────────────── +# The S3 API must bind the host address the k3s pods can reach (the 192.168.1.0/24 LAN; nodes are +# .50/.51/.52). tasks/main.yml auto-discovers the host's address in this CIDR prefix and asserts it +# exists; override s3_backup_target_bind_address to pin it explicitly. Port 9000 keeps the endpoint +# byte-identical to the previous MinIO install so nothing downstream (SOPS secret, Velero/CNPG) +# changes. (versitygw's own default is :7070 — deliberately overridden here.) +s3_backup_target_lan_prefix: "192.168.1." +s3_backup_target_bind_address: "" # "" ⇒ auto-discover from facts (see tasks) +s3_backup_target_api_port: 9000 +s3_backup_target_region: "us-east-1" + +# ── Buckets ──────────────────────────────────────────────────────────────────────────────────── +# Velero and CNPG each get a bucket. On the posix backend a bucket == a top-level subdirectory of +# the gateway root, so they are simply pre-created (owned by the service user). The single root key +# below has full access to all buckets. +s3_backup_target_buckets: + - velero + - cnpg + +# ── Credentials ────────────────────────────────────────────────────────────────────────────────── +# versitygw authenticates every S3 request against a single ROOT access key/secret (ROOT_ACCESS_KEY +# / ROOT_SECRET_KEY). We set it to the SAME scoped key/secret the cluster already uses, so the +# endpoint + access key + secret + buckets stay byte-identical to the previous MinIO target and the +# GitOps side (SOPS: platform/backup/minio-credentials.sops.yaml) needs ZERO change. +# +# Supplied via the vaulted group_vars (inventory/group_vars/proxmox_hosts/vault.yml): +# s3_backup_target_access_key, s3_backup_target_secret_key +# Intentionally NOT defaulted here so a missing vault fails loudly rather than silently +# provisioning guessable creds. + +# ── MinIO decommission (of any pre-existing MinIO install this role is replacing) ────────────── +# Paths/identities the OLD minio_backup_target role created, removed by the decommission block. +s3_backup_target_minio_service: "minio.service" +s3_backup_target_minio_bin: "/usr/local/bin/minio" +s3_backup_target_minio_mc_bin: "/usr/local/bin/mc" +s3_backup_target_minio_env_file: "/etc/default/minio" +s3_backup_target_minio_policy_file: "/etc/minio-k3s-backup-rw.json" +s3_backup_target_minio_user: "minio" +s3_backup_target_minio_group: "minio" +# MinIO's on-disk object layout left inside the gateway-root dir (cleared so versitygw starts +# fresh; there are no real backups yet so this is safe). The dataset itself is KEPT. +s3_backup_target_minio_leftovers: + - "{{ s3_backup_target_gateway_root }}/.minio" + - "{{ s3_backup_target_gateway_root }}/.minio.sys" diff --git a/infra/ansible/roles/minio_backup_target/handlers/main.yml b/infra/ansible/roles/s3_backup_target/handlers/main.yml similarity index 62% rename from infra/ansible/roles/minio_backup_target/handlers/main.yml rename to infra/ansible/roles/s3_backup_target/handlers/main.yml index a27bfd9..d90ff62 100644 --- a/infra/ansible/roles/minio_backup_target/handlers/main.yml +++ b/infra/ansible/roles/s3_backup_target/handlers/main.yml @@ -1,6 +1,6 @@ --- -- name: Restart minio +- name: Restart versitygw ansible.builtin.systemd: - name: minio + name: versitygw state: restarted daemon_reload: true diff --git a/infra/ansible/roles/s3_backup_target/tasks/main.yml b/infra/ansible/roles/s3_backup_target/tasks/main.yml new file mode 100644 index 0000000..0ccd765 --- /dev/null +++ b/infra/ansible/roles/s3_backup_target/tasks/main.yml @@ -0,0 +1,231 @@ +--- +# ── 0. Preconditions / safety asserts ───────────────────────────────────────────────────────── +- name: Assert S3 backup-target credentials are supplied (from vaulted group_vars) + ansible.builtin.assert: + that: + - s3_backup_target_access_key | default('') | length > 0 + - s3_backup_target_secret_key | default('') | length >= 8 + fail_msg: >- + versitygw credentials are missing. Provide s3_backup_target_access_key/_secret_key via + inventory/group_vars/proxmox_hosts/vault.yml (they MUST match the SOPS-staged scoped key at + platform/backup/minio-credentials.sops.yaml so nothing downstream changes). + quiet: true + +# Determine the host's 192.168.1.0/24 address (the S3 endpoint the cluster reaches). If the +# operator pinned s3_backup_target_bind_address, honour it; else auto-discover and assert. +- name: Discover the host LAN (cluster-reachable) IPv4 address + ansible.builtin.set_fact: + s3_backup_target_bind_address: >- + {{ s3_backup_target_bind_address + if (s3_backup_target_bind_address | length > 0) + else (ansible_facts.all_ipv4_addresses + | select('match', '^' ~ (s3_backup_target_lan_prefix | regex_escape)) + | list | first | default('')) }} + +- name: Assert a cluster-reachable LAN address was found + ansible.builtin.assert: + that: + - s3_backup_target_bind_address | length > 0 + fail_msg: >- + Could not find a {{ s3_backup_target_lan_prefix }}x address on this host + (interfaces: {{ ansible_facts.all_ipv4_addresses | default([]) | join(', ') }}). + The cluster egresses onto that LAN — refusing to bind the S3 endpoint to an unreachable + address. Set s3_backup_target_bind_address explicitly if discovery is wrong. + quiet: true + +# ── 1. Decommission any pre-existing MinIO install this role replaces (idempotent + surgical) ─── +# All steps tolerate absence, so on a fresh host (no MinIO) they are harmless no-ops. The ZFS +# dataset and mount are KEPT — only MinIO's service/binary/env/mc-client/object-layout go. +- name: Check whether the old MinIO systemd unit is present + ansible.builtin.stat: + path: "/etc/systemd/system/{{ s3_backup_target_minio_service }}" + register: s3_backup_target_minio_unit + +- name: Stop + disable the old MinIO service (only if its unit exists) + ansible.builtin.systemd: + name: "{{ s3_backup_target_minio_service }}" + state: stopped + enabled: false + when: s3_backup_target_minio_unit.stat.exists + +- name: Remove the old MinIO systemd unit + ansible.builtin.file: + path: "/etc/systemd/system/{{ s3_backup_target_minio_service }}" + state: absent + register: s3_backup_target_minio_unit_removed + +- name: Reload systemd if the MinIO unit was removed + ansible.builtin.systemd: + daemon_reload: true + when: s3_backup_target_minio_unit_removed.changed + +- name: Remove MinIO server + mc client binaries, env file and IAM policy file + ansible.builtin.file: + path: "{{ item }}" + state: absent + loop: + - "{{ s3_backup_target_minio_bin }}" + - "{{ s3_backup_target_minio_mc_bin }}" + - "{{ s3_backup_target_minio_env_file }}" + - "{{ s3_backup_target_minio_policy_file }}" + +- name: Remove the mc client config left in root's home + ansible.builtin.file: + path: "{{ item }}" + state: absent + loop: + - /root/.mc + - /root/.mcli + +- name: Clear MinIO's on-disk object layout from the gateway-root dir (dataset itself KEPT) + ansible.builtin.file: + path: "{{ item }}" + state: absent + loop: "{{ s3_backup_target_minio_leftovers }}" + +- name: Remove the old minio system user (its files are cleared / re-owned below) + ansible.builtin.user: + name: "{{ s3_backup_target_minio_user }}" + state: absent + remove: false + failed_when: false + +- name: Remove the old minio system group + ansible.builtin.group: + name: "{{ s3_backup_target_minio_group }}" + state: absent + failed_when: false + +# ── 2. ZFS dataset (backup drive) — idempotent, never touches other datasets ─────────────────── +# xattr=sa is required by versitygw's posix backend (object metadata is stored in extended attrs). +- name: Create/ensure the k3s-backups ZFS dataset (compression, quota, xattr=sa, mountpoint; NO sharenfs) + community.general.zfs: + name: "{{ s3_backup_target_zfs_dataset }}" + state: present + extra_zfs_properties: + compression: "{{ s3_backup_target_zfs_compression }}" + quota: "{{ s3_backup_target_zfs_quota }}" + xattr: "{{ s3_backup_target_zfs_xattr }}" + mountpoint: "{{ s3_backup_target_mountpoint }}" + +# ── 3. Service identity ─────────────────────────────────────────────────────────────────────── +- name: Create the versitygw system group + ansible.builtin.group: + name: "{{ s3_backup_target_group }}" + system: true + +- name: Create the versitygw system user (no login shell, home = gateway root) + ansible.builtin.user: + name: "{{ s3_backup_target_user }}" + group: "{{ s3_backup_target_group }}" + system: true + shell: /usr/sbin/nologin + home: "{{ s3_backup_target_gateway_root }}" + create_home: false + +- name: Ensure the gateway root dir exists (owned by versitygw, inside the dataset) + ansible.builtin.file: + path: "{{ s3_backup_target_gateway_root }}" + state: directory + owner: "{{ s3_backup_target_user }}" + group: "{{ s3_backup_target_group }}" + mode: "0750" + +- name: Pre-create the S3 buckets (a bucket == a top-level subdir of the gateway root) + ansible.builtin.file: + path: "{{ s3_backup_target_gateway_root }}/{{ item }}" + state: directory + owner: "{{ s3_backup_target_user }}" + group: "{{ s3_backup_target_group }}" + mode: "0750" + loop: "{{ s3_backup_target_buckets }}" + +# ── 4. Binary — fetched + extracted on the CONTROLLER (host has no internet), copied to the host ─ +- name: Ensure controller-side download cache exists + ansible.builtin.file: + path: "{{ s3_backup_target_controller_cache }}" + state: directory + mode: "0700" + delegate_to: localhost + become: false + run_once: true + +- name: Fetch the versitygw release tarball onto the controller (checksummed) + ansible.builtin.get_url: + url: "{{ s3_backup_target_versitygw_url }}" + dest: "{{ s3_backup_target_controller_cache }}/versitygw_{{ s3_backup_target_versitygw_version }}.tar.gz" + checksum: "sha256:{{ s3_backup_target_versitygw_sha256 }}" + mode: "0600" + delegate_to: localhost + become: false + run_once: true + +- name: Extract the versitygw binary on the controller + ansible.builtin.unarchive: + src: "{{ s3_backup_target_controller_cache }}/versitygw_{{ s3_backup_target_versitygw_version }}.tar.gz" + dest: "{{ s3_backup_target_controller_cache }}" + remote_src: true + creates: "{{ s3_backup_target_controller_cache }}/{{ s3_backup_target_versitygw_archive_member }}" + delegate_to: localhost + become: false + run_once: true + +# Integrity is already guaranteed by the SHA256 get_url performed on the controller; copy transfers +# that verified artifact faithfully. (The copy module's own `checksum` arg is SHA1 — not used here.) +- name: Install the versitygw binary on the host + ansible.builtin.copy: + src: "{{ s3_backup_target_controller_cache }}/{{ s3_backup_target_versitygw_archive_member }}" + dest: "{{ s3_backup_target_bin }}" + owner: root + group: root + mode: "0755" + notify: Restart versitygw + +# ── 5. systemd service ──────────────────────────────────────────────────────────────────────── +- name: Write the versitygw environment file (root access key/secret) + ansible.builtin.template: + src: versitygw.env.j2 + dest: "{{ s3_backup_target_env_file }}" + owner: root + group: "{{ s3_backup_target_group }}" + mode: "0640" + no_log: true + notify: Restart versitygw + +- name: Install the versitygw systemd unit + ansible.builtin.template: + src: versitygw.service.j2 + dest: /etc/systemd/system/versitygw.service + owner: root + group: root + mode: "0644" + notify: Restart versitygw + +- name: Enable + start versitygw (apply any pending unit/env changes now) + ansible.builtin.systemd: + name: versitygw + enabled: true + state: started + daemon_reload: true + +- name: Flush handlers so versitygw is (re)started before we verify it + ansible.builtin.meta: flush_handlers + +# ── 6. Readiness gate ───────────────────────────────────────────────────────────────────────── +# versitygw has no health endpoint; an UNAUTHENTICATED GET / returns 403 (AccessDenied) once the +# S3 server is up and enforcing SigV4 — a reliable "listening + serving" signal. +- name: Wait for the versitygw S3 port to accept connections + ansible.builtin.wait_for: + host: "{{ s3_backup_target_bind_address }}" + port: "{{ s3_backup_target_api_port }}" + timeout: 60 + +- name: Confirm versitygw is serving S3 (unauthenticated GET / ⇒ 403 AccessDenied) + ansible.builtin.uri: + url: "http://{{ s3_backup_target_bind_address }}:{{ s3_backup_target_api_port }}/" + method: GET + status_code: 403 + register: s3_backup_target_probe + retries: 15 + delay: 2 + until: s3_backup_target_probe.status == 403 diff --git a/infra/ansible/roles/s3_backup_target/templates/versitygw.env.j2 b/infra/ansible/roles/s3_backup_target/templates/versitygw.env.j2 new file mode 100644 index 0000000..46ab104 --- /dev/null +++ b/infra/ansible/roles/s3_backup_target/templates/versitygw.env.j2 @@ -0,0 +1,10 @@ +# {{ ansible_managed }} +# versitygw root credentials (single-node posix backup target for k3s Velero + CNPG). +# Managed by roles/s3_backup_target — do not edit by hand. +# +# versitygw authenticates every S3 request against this ROOT key. It is set to the SAME scoped +# access key/secret the cluster already uses (mirrored via SOPS in +# platform/backup/minio-credentials.sops.yaml) so the endpoint + key + secret stay byte-identical +# to the previous MinIO target — nothing downstream changes. +ROOT_ACCESS_KEY={{ s3_backup_target_access_key }} +ROOT_SECRET_KEY={{ s3_backup_target_secret_key }} diff --git a/infra/ansible/roles/s3_backup_target/templates/versitygw.service.j2 b/infra/ansible/roles/s3_backup_target/templates/versitygw.service.j2 new file mode 100644 index 0000000..1fc9d9d --- /dev/null +++ b/infra/ansible/roles/s3_backup_target/templates/versitygw.service.j2 @@ -0,0 +1,35 @@ +{# {{ ansible_managed }} — versitygw systemd unit (roles/s3_backup_target) #} +[Unit] +Description=versitygw S3 backup target (k3s Velero + CNPG) +Documentation=https://github.com/versity/versitygw +After=network-online.target +Wants=network-online.target +AssertFileIsExecutable={{ s3_backup_target_bin }} +AssertPathIsDirectory={{ s3_backup_target_gateway_root }} + +[Service] +Type=simple +User={{ s3_backup_target_user }} +Group={{ s3_backup_target_group }} +# ROOT_ACCESS_KEY / ROOT_SECRET_KEY are read from here (kept out of the process args). +EnvironmentFile={{ s3_backup_target_env_file }} +# Global options (--port/--region) come BEFORE the `posix` backend subcommand; posix's single +# positional arg is the gateway root, whose top-level subdirs are the S3 buckets. TLS off (no +# --cert/--key) ⇒ plain HTTP, path-style. +ExecStart={{ s3_backup_target_bin }} \ + --port {{ s3_backup_target_bind_address }}:{{ s3_backup_target_api_port }} \ + --region {{ s3_backup_target_region }} \ + posix {{ s3_backup_target_gateway_root }} +Restart=always +RestartSec=5s +LimitNOFILE=65536 +TimeoutStopSec=30s +# Hardening (backup target holds no other roles on this host). +NoNewPrivileges=true +ProtectSystem=strict +ProtectHome=true +PrivateTmp=true +ReadWritePaths={{ s3_backup_target_mountpoint }} + +[Install] +WantedBy=multi-user.target