Replace MinIO backup target with versitygw (s3_backup_target role)
MinIO's open-source edition is de-prioritised (console removed, commercial
pivot), so swap the off-cluster S3 backup target on the Proxmox host to
versitygw v1.7.0 (github.com/versity/versitygw) — a maintained Go S3 gateway
serving a POSIX filesystem as S3.
Rename role minio_backup_target -> s3_backup_target and playbook backup-minio
-> backup-s3. The role now:
- decommissions any pre-existing MinIO (service/binary/env/mc-client/user/
object-layout), KEEPING the ZFS dataset Storage/k3s-backups and its mount
- installs the pinned, checksummed versitygw binary (fetched+extracted on the
controller, host has no internet), a dedicated versitygw system user, and a
Type=simple hardened systemd unit running the posix backend rooted at
/mnt/Storage/k3s-backups/data, bound to 192.168.1.9:9000 (same endpoint),
plain HTTP / path-style
- ensures the dataset has xattr=sa (versitygw stores object metadata in xattrs)
- pre-creates the velero + cnpg buckets (bucket == top-level subdir)
Credentials are REUSED: versitygw's ROOT key is set to the same scoped
access_key/secret the cluster already uses, so endpoint + key + secret + bucket
names are byte-identical to the old MinIO target. The GitOps SOPS secret
(platform/backup/minio-credentials.sops.yaml) and all cluster-side Velero/CNPG
config need ZERO change. The vaulted group_vars now expose
s3_backup_target_access_key/_secret_key (same values, MinIO root creds retired).
Verified live on the Proxmox host: versitygw active+enabled on 192.168.1.9:9000,
MinIO fully removed, dataset intact; SigV4 ListBuckets with the reused key
returns velero+cnpg; a throwaway in-cluster pod reaches http://192.168.1.9:9000
(versitygw S3 response). Re-run is idempotent (no host changes).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed
1 parent
1bee2554f8
commit
6739339f53
17 files changed
+499
-491
No files matched your search
@@ -1,29 +1,27 @@
|
||||
# group_vars for the `proxmox_hosts` group (the Proxmox hypervisor host)
|
||||
|
||||
Target of `playbooks/backup-minio.yml` (role `minio_backup_target`). The role needs the MinIO
|
||||
credentials as **ansible-vault-encrypted** vars here — encrypted, so safe to commit; the passphrase
|
||||
that decrypts them stays out of the repo (`ansible.cfg` → `vault_password_file`).
|
||||
Target of `playbooks/backup-s3.yml` (role `s3_backup_target`, versitygw). The role needs the S3
|
||||
backup-target credentials as **ansible-vault-encrypted** vars here — encrypted, so safe to commit;
|
||||
the passphrase that decrypts them stays out of the repo (`ansible.cfg` → `vault_password_file`).
|
||||
|
||||
## `vault.yml` (committed, encrypted)
|
||||
|
||||
```bash
|
||||
cd infra/ansible
|
||||
ansible-vault create inventory/group_vars/proxmox_hosts/vault.yml
|
||||
ansible-vault edit inventory/group_vars/proxmox_hosts/vault.yml
|
||||
# contents:
|
||||
# minio_backup_target_root_user: "…" # MinIO root (host admin only)
|
||||
# minio_backup_target_root_password: "…"
|
||||
# minio_backup_target_access_key: "k3s-backup" # scoped key handed to Velero + CNPG
|
||||
# minio_backup_target_secret_key: "…"
|
||||
# s3_backup_target_access_key: "k3s-backup" # versitygw ROOT key = key handed to Velero + CNPG
|
||||
# s3_backup_target_secret_key: "…"
|
||||
```
|
||||
|
||||
The **scoped** access key/secret is mirrored (SOPS-encrypted) for the GitOps side at
|
||||
`platform/backup/minio-credentials.sops.yaml` — the two MUST stay in sync (same key/secret), since
|
||||
the cluster authenticates to this MinIO with them. Rotating means updating BOTH and re-running the
|
||||
playbook (recreates the MinIO user) + reconciling Flux.
|
||||
versitygw authenticates every S3 request against this single **ROOT** key. It is mirrored
|
||||
(SOPS-encrypted) for the GitOps side at `platform/backup/minio-credentials.sops.yaml` — the two MUST
|
||||
stay in sync (same key/secret), since the cluster authenticates to this endpoint with them. Rotating
|
||||
means updating BOTH and re-running the playbook (rewrites the env file + restarts versitygw) +
|
||||
reconciling Flux.
|
||||
|
||||
## Notes
|
||||
- This is the **Ansible-bootstrap-layer** secret (host credential), distinct from the platform's
|
||||
in-cluster **SOPS + age** secret (ADR-0007) — different layer, different tool, same values for the
|
||||
scoped key.
|
||||
in-cluster **SOPS + age** secret (ADR-0007) — different layer, different tool, same values.
|
||||
- `vault.yml` is a vault blob (not parseable YAML) — excluded from the yamllint gate like the other
|
||||
`**/vault.yml`.
|
||||
@@ -1,29 +1,25 @@
|
||||
$ANSIBLE_VAULT;1.1;AES256
|
||||
30633837396565613861623931383238383836356238326436633738613935326461346138373432
|
||||
6234643535303037313337363463633765326239376238380a333331333136396239313733366163
|
||||
33353036366461393530633139636430313736366338313331353163616135303930323038636134
|
||||
6366353362313537330a373863623861663537353539376664306338643539346139396130396632
|
||||
30336231363731616334383863326638636238323565316364303530396636373163393765376438
|
||||
63373263623063656662353434623339316238663535386465653164326236616265616638656639
|
||||
30613466656439396534316339646139343334656165323965323965313361633933633038376663
|
||||
31653539653763623132313533396333643964373032313333613432383433626338376431323731
|
||||
63316163363361303439313266356361613737333066643931656564633038306236363439666435
|
||||
39343536656336343834663464356566373564636531343534343365376332343832343235663339
|
||||
66383561306533323938613738376632626263306337346532643063373166333838333033346365
|
||||
32383136363636643163613334366631653238613166303962656463363663643830343531363533
|
||||
39333630323765643963396562363537396564666462306163336366643739313035636633646661
|
||||
61653637376535393034383631303563363539633265333161363334616239323866353162636632
|
||||
65663830323465393138306361333662383934346438663562316235353035633064643861353334
|
||||
63646435653639316439366139316539653833656561646439323766633139306565383565656261
|
||||
61323230306635343236646438336165643737626132613962633930653461393131303731623438
|
||||
63313265653166643334396461353231343235386434363438623034323762653433303932326630
|
||||
62633464316139353433366338313837646162323231363135353964623035366132643332366662
|
||||
38393465326363616139376465383738626636393639363338653437323832613439313138353033
|
||||
39393765613735666331636230363463306462373238623138316263326339366330333332643833
|
||||
33363764303939326662343665643963333463356638613231363834386132666339393132356538
|
||||
38303362643337366430303165633432666632343536383366653836663135383336316232393936
|
||||
66626632386537333739313432313564633866383132353862626433626366633562346464613066
|
||||
35376332303132326438313132303264313763376163396137356132306131373335346266646532
|
||||
31666230616461386339366461653661323138353662373866316639626134326339643231613337
|
||||
30343361643166353162363866653061313966663435373332353562656164663962326562333038
|
||||
35363834656361393930
|
||||
33623463393666373861613531626336396164356361643939326332626634663234653164313437
|
||||
3963326465326364643639633066636564656565333461360a383438323530663932326462376661
|
||||
64343766366132373436366336353338383365396563636138333333306330643337636532383662
|
||||
6561323836656637380a656436656538613263646461386635303762376163363161336436386263
|
||||
34653437386364363161653132343434333230393732373364383564336162643136636437376231
|
||||
39376463336666396462363431636161306632326134663038343364626232333838333039393338
|
||||
64313762343363666139393864366561316362633465323361633132373533356531353534396433
|
||||
62333131356630343764303438663766623737333137383264623739646539373535306136303530
|
||||
34663063383032643238373537663633343735613265653964356638306566636538363063656435
|
||||
65643666366665393238363037313337633366333338633163316533656134336463363630353961
|
||||
37656637343063373266643237663633613638303165656262313133313535313665336335333039
|
||||
30626537373530663430343637386263353765353538376263333130623536306635353966393561
|
||||
62313362373061303035376635646335303133613365326163643566333031356436326464616430
|
||||
34373433343733386666626234623064383063393330343034363036663632373865643636623961
|
||||
31666131636538303131376339623733353964653266356339313466616363646336663433666631
|
||||
61663564633931626639323235633332366338333837316535376330633534626537666166643262
|
||||
63646239656165303366626163613238393266336566316164356330393836323332336330633635
|
||||
64636636613164366338396666316137353837313639323630353234373936393335366364666361
|
||||
35356163363636623031333733653432636539396235633436313334333663343532303239373238
|
||||
37646230326164396339386339356663396661306165343630316261306331383632626662323234
|
||||
31396539313766356633333736383036623839626663383763626332626237373338653832646463
|
||||
34366434386139306336633964313939643565363838346163656236656561363865386136346530
|
||||
36316661323262323366646332316535306637316663656264303137653831656136633432643765
|
||||
61646433653530646339
|
||||
@@ -38,8 +38,8 @@ localhost ansible_connection=local
|
||||
[vpn_egress_gateway]
|
||||
k3s-dev-3
|
||||
|
||||
# The Proxmox HYPERVISOR host itself (NOT a k3s node) — target of playbooks/backup-minio.yml,
|
||||
# which stands up the off-cluster MinIO S3 backup target for Velero + CNPG. Reached DIRECTLY
|
||||
# The Proxmox HYPERVISOR host itself (NOT a k3s node) — target of playbooks/backup-s3.yml,
|
||||
# which stands up the off-cluster versitygw S3 backup target for Velero + CNPG. Reached DIRECTLY
|
||||
# over SSH as root (its own 192.168.2.102:22, the pfSense NAT to the host's 192.168.1.9), so
|
||||
# it overrides the cluster-wide `ansible_user: k3s` from group_vars/all.yml. Distinct from the
|
||||
# `hypervisor` group above (that is `localhost` for host-prereqs.yml, a different machine).
|
||||
|
||||
@@ -1,20 +0,0 @@
|
||||
---
|
||||
# Stand up the off-cluster S3 backup target (MinIO) on the Proxmox host — the store Velero
|
||||
# (R-STATE-03) and CloudNativePG (R-IDP-05) back up into. Host infra, additive + idempotent:
|
||||
# ONE new ZFS dataset (Storage/k3s-backups), the minio user/service/binaries, and the
|
||||
# velero + cnpg buckets with a scoped non-root access key. It never touches existing
|
||||
# datasets, sharenfs, pvesm storage, VMs, or firewall.
|
||||
#
|
||||
# Credentials come from Ansible vault (inventory/group_vars/proxmox_hosts/vault.yml):
|
||||
# minio_backup_target_root_user/_root_password, minio_backup_target_access_key/_secret_key
|
||||
# The scoped key is ALSO staged for the GitOps side (SOPS) at
|
||||
# platform/backup/minio-credentials.sops.yaml
|
||||
#
|
||||
# Prereqs: ansible-galaxy collection install -r collections/requirements.yml (community.general.zfs)
|
||||
# Run: ansible-playbook playbooks/backup-minio.yml
|
||||
- name: Provision the MinIO S3 backup target on the Proxmox host
|
||||
hosts: proxmox_hosts
|
||||
become: true
|
||||
gather_facts: true # need ansible_facts.all_ipv4_addresses to discover the LAN bind addr
|
||||
roles:
|
||||
- role: minio_backup_target
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
# Stand up the off-cluster S3 backup target on the Proxmox host — the store Velero (R-STATE-03)
|
||||
# and CloudNativePG (R-IDP-05) back up into. Implemented with versitygw (Versity Gateway), which
|
||||
# REPLACES the earlier MinIO target; the role also decommissions any pre-existing MinIO install so
|
||||
# a re-run is a clean swap. Host infra, surgical + idempotent: the ONE ZFS dataset it owns
|
||||
# (Storage/k3s-backups), the versitygw user/service/binary, and the velero + cnpg buckets. It never
|
||||
# touches other datasets, sharenfs, pvesm storage, VMs or firewall.
|
||||
#
|
||||
# Credentials come from Ansible vault (inventory/group_vars/proxmox_hosts/vault.yml):
|
||||
# s3_backup_target_access_key / _secret_key (versitygw ROOT key)
|
||||
# The SAME key is staged for the GitOps side (SOPS) at
|
||||
# platform/backup/minio-credentials.sops.yaml
|
||||
# Endpoint + key + secret + bucket names are byte-identical to the old MinIO target, so the
|
||||
# cluster-side Velero/CNPG config + the SOPS secret need ZERO change.
|
||||
#
|
||||
# Prereqs: ansible-galaxy collection install -r collections/requirements.yml (community.general.zfs)
|
||||
# Run: ansible-playbook playbooks/backup-s3.yml
|
||||
- name: Provision the versitygw S3 backup target on the Proxmox host
|
||||
hosts: proxmox_hosts
|
||||
become: true
|
||||
gather_facts: true # need ansible_facts.all_ipv4_addresses to discover the LAN bind addr
|
||||
roles:
|
||||
- role: s3_backup_target
|
||||
@@ -1,46 +0,0 @@
|
||||
# roles/minio_backup_target
|
||||
|
||||
Stands up a **single-node, single-drive MinIO** S3 endpoint on the **Proxmox host** to serve as the
|
||||
off-cluster **backup target** for the k3s platform's **Velero** (volume/resource DR, R-STATE-03) and
|
||||
**CloudNativePG** (Keycloak DB Barman object store, R-IDP-05). A backup target — not primary storage —
|
||||
so a single drive is intentional.
|
||||
|
||||
## What it does (all additive + idempotent)
|
||||
|
||||
1. **ZFS dataset** `Storage/k3s-backups` — `compression=zstd`, `quota=100G`,
|
||||
`mountpoint=/mnt/Storage/k3s-backups`, **no sharenfs**. Uses `community.general.zfs` so re-runs are
|
||||
no-ops and existing datasets are never touched.
|
||||
2. **`minio` system user/group** + a data dir (`…/k3s-backups/data`) it owns.
|
||||
3. **MinIO + `mc` binaries** (pinned, checksummed). The Proxmox host has **no outbound internet**, so
|
||||
the binaries are downloaded on the **Ansible controller** and copied to the host.
|
||||
4. **systemd unit** `minio.service` (Type=notify, hardened), bound to the host's
|
||||
**cluster-reachable `192.168.1.x` address** on port **9000** (console 9001) — auto-discovered from
|
||||
facts, assert-guarded.
|
||||
5. **Buckets** `velero` + `cnpg`, a **scoped non-root access key** limited by an IAM policy to just
|
||||
those two buckets, plus the retained root creds for host-side admin.
|
||||
|
||||
## Credentials
|
||||
|
||||
Supplied via the vaulted `inventory/group_vars/proxmox_hosts/vault.yml` (ansible-vault). Nothing is
|
||||
defaulted — a missing vault fails the play loudly:
|
||||
|
||||
| var | purpose |
|
||||
|---|---|
|
||||
| `minio_backup_target_root_user` / `_root_password` | MinIO root (host admin only) |
|
||||
| `minio_backup_target_access_key` / `_secret_key` | scoped key handed to Velero + CNPG |
|
||||
|
||||
The **same scoped key** is staged for the GitOps side, SOPS-encrypted, at
|
||||
`platform/backup/minio-credentials.sops.yaml`.
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
cd infra/ansible
|
||||
ansible-galaxy collection install -r collections/requirements.yml # community.general (zfs)
|
||||
ansible-playbook playbooks/backup-minio.yml
|
||||
```
|
||||
|
||||
## Endpoint
|
||||
|
||||
`http://<host-192.168.1.x>:9000` — the S3 URL the cluster uses (path-style). Health:
|
||||
`GET /minio/health/ready` ⇒ 200. Verified reachable from an in-cluster pod at bring-up.
|
||||
@@ -1,62 +0,0 @@
|
||||
---
|
||||
# roles/minio_backup_target — a single-node, single-drive MinIO S3 target on the Proxmox
|
||||
# host, used ONLY as an off-cluster backup bucket store for Velero + CloudNativePG (ADR-0003,
|
||||
# R-STATE-03 / R-IDP-05). This is a BACKUP TARGET, not primary storage — single drive is fine.
|
||||
#
|
||||
# SAFETY: every task in this role is additive and idempotent. It creates ONE new ZFS dataset,
|
||||
# ONE system user, ONE systemd service and two buckets. It never touches existing datasets,
|
||||
# sharenfs, pvesm storage, VMs or firewall. See tasks/main.yml for the guards.
|
||||
|
||||
# ── MinIO / mc release pins (Renovate-friendly; both are AGPL community builds) ───────────────
|
||||
# The Proxmox host has NO outbound internet (PiHole-only DNS, no default route to WAN), so the
|
||||
# binaries are fetched on the ANSIBLE CONTROLLER and copied to the host (tasks/main.yml).
|
||||
minio_backup_target_minio_version: "RELEASE.2025-09-07T16-13-09Z"
|
||||
minio_backup_target_minio_sha256: "7c5bd8512c6e966455b1d198209358b2d191c77a83ab377c4073281065fb855f"
|
||||
minio_backup_target_minio_url: "https://dl.min.io/server/minio/release/linux-amd64/archive/minio.{{ minio_backup_target_minio_version }}"
|
||||
|
||||
minio_backup_target_mc_version: "RELEASE.2025-08-13T08-35-41Z"
|
||||
minio_backup_target_mc_sha256: "01f866e9c5f9b87c2b09116fa5d7c06695b106242d829a8bb32990c00312e891"
|
||||
minio_backup_target_mc_url: "https://dl.min.io/client/mc/release/linux-amd64/archive/mc.{{ minio_backup_target_mc_version }}"
|
||||
|
||||
# Controller-side cache the binaries land in before being copied to the host (delegate_to localhost).
|
||||
minio_backup_target_controller_cache: "/tmp/minio-backup-target-cache"
|
||||
|
||||
# ── ZFS dataset (backup drive) ────────────────────────────────────────────────────────────────
|
||||
# Host convention is Storage/<purpose> mounted at /mnt/Storage/<purpose>. NO sharenfs.
|
||||
minio_backup_target_zfs_dataset: "Storage/k3s-backups"
|
||||
minio_backup_target_mountpoint: "/mnt/Storage/k3s-backups"
|
||||
minio_backup_target_zfs_compression: "zstd"
|
||||
minio_backup_target_zfs_quota: "100G"
|
||||
# MinIO writes its drive (incl. .minio.sys) into a subdir of the dataset, keeping the dataset
|
||||
# root clean and owned by root.
|
||||
minio_backup_target_data_dir: "{{ minio_backup_target_mountpoint }}/data"
|
||||
|
||||
# ── Service identity ────────────────────────────────────────────────────────────────────────
|
||||
minio_backup_target_user: "minio"
|
||||
minio_backup_target_group: "minio"
|
||||
minio_backup_target_minio_bin: "/usr/local/bin/minio"
|
||||
minio_backup_target_mc_bin: "/usr/local/bin/mc"
|
||||
minio_backup_target_env_file: "/etc/default/minio"
|
||||
|
||||
# ── Network binding ──────────────────────────────────────────────────────────────────────────
|
||||
# MinIO's S3 API must bind the host address the k3s pods can reach (the 192.168.1.0/24 LAN;
|
||||
# nodes are .50/.51/.52). tasks/main.yml auto-discovers the host's address in this CIDR prefix
|
||||
# and asserts it exists; override minio_backup_target_bind_address to pin it explicitly.
|
||||
minio_backup_target_lan_prefix: "192.168.1."
|
||||
minio_backup_target_bind_address: "" # "" ⇒ auto-discover from facts (see tasks)
|
||||
minio_backup_target_api_port: 9000
|
||||
minio_backup_target_console_port: 9001
|
||||
|
||||
# ── Buckets + scoped (non-root) access key ────────────────────────────────────────────────────
|
||||
# Velero and CNPG each get a bucket; a single non-root user scoped to ONLY these two buckets is
|
||||
# handed to the cluster (SOPS). Root creds stay on the host for admin.
|
||||
minio_backup_target_buckets:
|
||||
- velero
|
||||
- cnpg
|
||||
minio_backup_target_policy_name: "k3s-backup-rw"
|
||||
|
||||
# Credentials come from the vaulted group_vars (inventory/group_vars/proxmox_hosts/vault.yml):
|
||||
# minio_backup_target_root_user, minio_backup_target_root_password,
|
||||
# minio_backup_target_access_key, minio_backup_target_secret_key
|
||||
# They are intentionally NOT defaulted here so a missing vault fails loudly rather than
|
||||
# silently provisioning guessable creds.
|
||||
@@ -1,263 +0,0 @@
|
||||
---
|
||||
# ── 0. Preconditions / safety asserts ─────────────────────────────────────────────────────────
|
||||
- name: Assert MinIO credentials are supplied (from vaulted group_vars)
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- minio_backup_target_root_user | default('') | length > 0
|
||||
- minio_backup_target_root_password | default('') | length >= 8
|
||||
- minio_backup_target_access_key | default('') | length > 0
|
||||
- minio_backup_target_secret_key | default('') | length >= 8
|
||||
fail_msg: >-
|
||||
MinIO credentials are missing. Provide minio_backup_target_root_user/_password and
|
||||
minio_backup_target_access_key/_secret_key via inventory/group_vars/proxmox_hosts/vault.yml.
|
||||
quiet: true
|
||||
|
||||
# Determine the host's 192.168.1.0/24 address (the S3 endpoint the cluster reaches). If the
|
||||
# operator pinned minio_backup_target_bind_address, honour it; else auto-discover and assert.
|
||||
- name: Discover the host LAN (cluster-reachable) IPv4 address
|
||||
ansible.builtin.set_fact:
|
||||
minio_backup_target_bind_address: >-
|
||||
{{ minio_backup_target_bind_address
|
||||
if (minio_backup_target_bind_address | length > 0)
|
||||
else (ansible_facts.all_ipv4_addresses
|
||||
| select('match', '^' ~ (minio_backup_target_lan_prefix | regex_escape))
|
||||
| list | first | default('')) }}
|
||||
|
||||
- name: Assert a cluster-reachable LAN address was found
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- minio_backup_target_bind_address | length > 0
|
||||
fail_msg: >-
|
||||
Could not find a {{ minio_backup_target_lan_prefix }}x address on this host
|
||||
(interfaces: {{ ansible_facts.all_ipv4_addresses | default([]) | join(', ') }}).
|
||||
The cluster egresses onto that LAN — refusing to bind MinIO to an unreachable address.
|
||||
Set minio_backup_target_bind_address explicitly if discovery is wrong.
|
||||
quiet: true
|
||||
|
||||
# ── 1. ZFS dataset (backup drive) — idempotent, never touches other datasets ───────────────────
|
||||
- name: Create/ensure the k3s-backups ZFS dataset (compression, quota, mountpoint; NO sharenfs)
|
||||
community.general.zfs:
|
||||
name: "{{ minio_backup_target_zfs_dataset }}"
|
||||
state: present
|
||||
extra_zfs_properties:
|
||||
compression: "{{ minio_backup_target_zfs_compression }}"
|
||||
quota: "{{ minio_backup_target_zfs_quota }}"
|
||||
mountpoint: "{{ minio_backup_target_mountpoint }}"
|
||||
|
||||
# ── 2. Service identity ───────────────────────────────────────────────────────────────────────
|
||||
- name: Create the minio system group
|
||||
ansible.builtin.group:
|
||||
name: "{{ minio_backup_target_group }}"
|
||||
system: true
|
||||
|
||||
- name: Create the minio system user (no login shell, home = data dir)
|
||||
ansible.builtin.user:
|
||||
name: "{{ minio_backup_target_user }}"
|
||||
group: "{{ minio_backup_target_group }}"
|
||||
system: true
|
||||
shell: /usr/sbin/nologin
|
||||
home: "{{ minio_backup_target_data_dir }}"
|
||||
create_home: false
|
||||
|
||||
- name: Ensure the MinIO data directory exists (owned by minio, inside the dataset)
|
||||
ansible.builtin.file:
|
||||
path: "{{ minio_backup_target_data_dir }}"
|
||||
state: directory
|
||||
owner: "{{ minio_backup_target_user }}"
|
||||
group: "{{ minio_backup_target_group }}"
|
||||
mode: "0750"
|
||||
|
||||
# ── 3. Binaries — fetched on the CONTROLLER (host has no internet), copied to the host ──────────
|
||||
- name: Ensure controller-side download cache exists
|
||||
ansible.builtin.file:
|
||||
path: "{{ minio_backup_target_controller_cache }}"
|
||||
state: directory
|
||||
mode: "0700"
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
run_once: true
|
||||
|
||||
- name: Fetch MinIO server binary onto the controller (checksummed)
|
||||
ansible.builtin.get_url:
|
||||
url: "{{ minio_backup_target_minio_url }}"
|
||||
dest: "{{ minio_backup_target_controller_cache }}/minio.{{ minio_backup_target_minio_version }}"
|
||||
checksum: "sha256:{{ minio_backup_target_minio_sha256 }}"
|
||||
mode: "0755"
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
run_once: true
|
||||
|
||||
- name: Fetch mc client binary onto the controller (checksummed)
|
||||
ansible.builtin.get_url:
|
||||
url: "{{ minio_backup_target_mc_url }}"
|
||||
dest: "{{ minio_backup_target_controller_cache }}/mc.{{ minio_backup_target_mc_version }}"
|
||||
checksum: "sha256:{{ minio_backup_target_mc_sha256 }}"
|
||||
mode: "0755"
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
run_once: true
|
||||
|
||||
# Integrity is already guaranteed by the SHA256 verification get_url performed on the controller
|
||||
# above; copy transfers that verified artifact faithfully. (The copy module's own `checksum` arg is
|
||||
# SHA1, not SHA256 — deliberately not used here to avoid a false mismatch.)
|
||||
- name: Install MinIO server binary on the host
|
||||
ansible.builtin.copy:
|
||||
src: "{{ minio_backup_target_controller_cache }}/minio.{{ minio_backup_target_minio_version }}"
|
||||
dest: "{{ minio_backup_target_minio_bin }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
notify: Restart minio
|
||||
|
||||
- name: Install mc client binary on the host
|
||||
ansible.builtin.copy:
|
||||
src: "{{ minio_backup_target_controller_cache }}/mc.{{ minio_backup_target_mc_version }}"
|
||||
dest: "{{ minio_backup_target_mc_bin }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
|
||||
# ── 4. systemd service ────────────────────────────────────────────────────────────────────────
|
||||
- name: Write the MinIO environment file (root creds, bind address, data dir)
|
||||
ansible.builtin.template:
|
||||
src: minio.env.j2
|
||||
dest: "{{ minio_backup_target_env_file }}"
|
||||
owner: root
|
||||
group: "{{ minio_backup_target_group }}"
|
||||
mode: "0640"
|
||||
no_log: true
|
||||
notify: Restart minio
|
||||
|
||||
- name: Install the MinIO systemd unit
|
||||
ansible.builtin.template:
|
||||
src: minio.service.j2
|
||||
dest: /etc/systemd/system/minio.service
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
notify: Restart minio
|
||||
|
||||
- name: Enable + start MinIO (apply any pending unit/env changes now)
|
||||
ansible.builtin.systemd:
|
||||
name: minio
|
||||
enabled: true
|
||||
state: started
|
||||
daemon_reload: true
|
||||
|
||||
- name: Flush handlers so MinIO is (re)started before we configure buckets/users
|
||||
ansible.builtin.meta: flush_handlers
|
||||
|
||||
# ── 5. Readiness gate ─────────────────────────────────────────────────────────────────────────
|
||||
- name: Wait for the MinIO S3 health endpoint to report ready
|
||||
ansible.builtin.uri:
|
||||
url: "http://{{ minio_backup_target_bind_address }}:{{ minio_backup_target_api_port }}/minio/health/ready"
|
||||
status_code: 200
|
||||
register: minio_backup_target_health
|
||||
retries: 30
|
||||
delay: 2
|
||||
until: minio_backup_target_health.status == 200
|
||||
|
||||
# ── 6. Buckets + scoped non-root user (via mc, idempotent) ─────────────────────────────────────
|
||||
- name: Configure the local mc alias against the running MinIO
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- "{{ minio_backup_target_mc_bin }}"
|
||||
- alias
|
||||
- set
|
||||
- k3sbackup
|
||||
- "http://{{ minio_backup_target_bind_address }}:{{ minio_backup_target_api_port }}"
|
||||
- "{{ minio_backup_target_root_user }}"
|
||||
- "{{ minio_backup_target_root_password }}"
|
||||
no_log: true
|
||||
changed_when: false
|
||||
|
||||
- name: Create the Velero + CNPG buckets (idempotent)
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- "{{ minio_backup_target_mc_bin }}"
|
||||
- mb
|
||||
- --ignore-existing
|
||||
- "k3sbackup/{{ item }}"
|
||||
loop: "{{ minio_backup_target_buckets }}"
|
||||
register: minio_backup_target_mb
|
||||
changed_when: "'created successfully' in (minio_backup_target_mb.stdout | default(''))"
|
||||
|
||||
- name: Render the scoped bucket RW policy to the host
|
||||
ansible.builtin.template:
|
||||
src: bucket-policy.json.j2
|
||||
dest: "/etc/minio-{{ minio_backup_target_policy_name }}.json"
|
||||
owner: root
|
||||
group: "{{ minio_backup_target_group }}"
|
||||
mode: "0640"
|
||||
|
||||
- name: Check whether the scoped policy already exists
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- "{{ minio_backup_target_mc_bin }}"
|
||||
- admin
|
||||
- policy
|
||||
- info
|
||||
- k3sbackup
|
||||
- "{{ minio_backup_target_policy_name }}"
|
||||
register: minio_backup_target_policy_info
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
|
||||
- name: Create the scoped bucket RW policy (velero + cnpg only)
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- "{{ minio_backup_target_mc_bin }}"
|
||||
- admin
|
||||
- policy
|
||||
- create
|
||||
- k3sbackup
|
||||
- "{{ minio_backup_target_policy_name }}"
|
||||
- "/etc/minio-{{ minio_backup_target_policy_name }}.json"
|
||||
when: minio_backup_target_policy_info.rc != 0
|
||||
changed_when: true
|
||||
|
||||
- name: Check whether the scoped access key already exists
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- "{{ minio_backup_target_mc_bin }}"
|
||||
- admin
|
||||
- user
|
||||
- info
|
||||
- k3sbackup
|
||||
- "{{ minio_backup_target_access_key }}"
|
||||
register: minio_backup_target_user_info
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
|
||||
- name: Create the scoped non-root access key
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- "{{ minio_backup_target_mc_bin }}"
|
||||
- admin
|
||||
- user
|
||||
- add
|
||||
- k3sbackup
|
||||
- "{{ minio_backup_target_access_key }}"
|
||||
- "{{ minio_backup_target_secret_key }}"
|
||||
when: minio_backup_target_user_info.rc != 0
|
||||
no_log: true
|
||||
changed_when: true
|
||||
|
||||
- name: Attach the scoped policy to the access key (idempotent)
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- "{{ minio_backup_target_mc_bin }}"
|
||||
- admin
|
||||
- policy
|
||||
- attach
|
||||
- k3sbackup
|
||||
- "{{ minio_backup_target_policy_name }}"
|
||||
- --user
|
||||
- "{{ minio_backup_target_access_key }}"
|
||||
register: minio_backup_target_attach
|
||||
changed_when:
|
||||
- minio_backup_target_attach.rc == 0
|
||||
- "'already in effect' not in (minio_backup_target_attach.stderr | default(''))"
|
||||
failed_when:
|
||||
- minio_backup_target_attach.rc != 0
|
||||
- "'already in effect' not in (minio_backup_target_attach.stderr | default(''))"
|
||||
@@ -1,15 +0,0 @@
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["s3:*"],
|
||||
"Resource": [
|
||||
{% for b in minio_backup_target_buckets %}
|
||||
"arn:aws:s3:::{{ b }}",
|
||||
"arn:aws:s3:::{{ b }}/*"{{ "," if not loop.last else "" }}
|
||||
{% endfor %}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,11 +0,0 @@
|
||||
# {{ ansible_managed }}
|
||||
# MinIO server environment (single-node, single-drive backup target for k3s Velero + CNPG).
|
||||
# Managed by roles/minio_backup_target — do not edit by hand.
|
||||
MINIO_ROOT_USER={{ minio_backup_target_root_user }}
|
||||
MINIO_ROOT_PASSWORD={{ minio_backup_target_root_password }}
|
||||
|
||||
# The S3 drive lives on the ZFS dataset {{ minio_backup_target_zfs_dataset }}.
|
||||
MINIO_VOLUMES="{{ minio_backup_target_data_dir }}"
|
||||
|
||||
# Bind the S3 API (and console) to the cluster-reachable LAN address only.
|
||||
MINIO_OPTS="--address {{ minio_backup_target_bind_address }}:{{ minio_backup_target_api_port }} --console-address {{ minio_backup_target_bind_address }}:{{ minio_backup_target_console_port }}"
|
||||
@@ -1,28 +0,0 @@
|
||||
{# {{ ansible_managed }} — MinIO systemd unit (roles/minio_backup_target) #}
|
||||
[Unit]
|
||||
Description=MinIO S3 backup target (k3s Velero + CNPG)
|
||||
Documentation=https://min.io/docs/minio/linux/index.html
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
AssertFileIsExecutable={{ minio_backup_target_minio_bin }}
|
||||
|
||||
[Service]
|
||||
Type=notify
|
||||
User={{ minio_backup_target_user }}
|
||||
Group={{ minio_backup_target_group }}
|
||||
EnvironmentFile={{ minio_backup_target_env_file }}
|
||||
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo 'MINIO_VOLUMES unset in {{ minio_backup_target_env_file }}'; exit 1; fi"
|
||||
ExecStart={{ minio_backup_target_minio_bin }} server $MINIO_OPTS $MINIO_VOLUMES
|
||||
Restart=always
|
||||
RestartSec=5s
|
||||
LimitNOFILE=65536
|
||||
TimeoutStopSec=30s
|
||||
# Hardening (backup target holds no other roles on this host).
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=full
|
||||
ProtectHome=true
|
||||
PrivateTmp=true
|
||||
ReadWritePaths={{ minio_backup_target_mountpoint }}
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,66 @@
|
||||
# roles/s3_backup_target
|
||||
|
||||
Stands up a **single-node S3 endpoint** on the **Proxmox host** to serve as the off-cluster
|
||||
**backup target** for the k3s platform's **Velero** (volume/resource DR, R-STATE-03) and
|
||||
**CloudNativePG** (Keycloak DB Barman object store, R-IDP-05). A backup target — not primary
|
||||
storage — so a single dataset is intentional.
|
||||
|
||||
Implemented with **[versitygw](https://github.com/versity/versitygw)** (Versity Gateway): a
|
||||
maintained Go S3 gateway that serves a POSIX filesystem as S3. It **replaces the earlier MinIO
|
||||
implementation** (MinIO's community edition was de-prioritised — console removed, commercial
|
||||
pivot). The role also **decommissions any pre-existing MinIO** install on the host, so a re-run is
|
||||
a clean, idempotent swap. Endpoint, access key, secret and bucket names are kept **byte-identical**
|
||||
to the previous MinIO target, so nothing downstream (the SOPS secret, Velero, CNPG) changes.
|
||||
|
||||
## What it does (all additive + idempotent, surgical)
|
||||
|
||||
1. **Decommission MinIO** (tolerant no-op on a fresh host): stop/disable/remove `minio.service`,
|
||||
the `minio` + `mc` binaries, `/etc/default/minio`, the IAM policy file, the `mc` client config,
|
||||
the `minio` user/group, and MinIO's on-disk object layout (`.minio`, `.minio.sys`) inside the
|
||||
gateway-root dir. **The ZFS dataset and mount are KEPT.**
|
||||
2. **ZFS dataset** `Storage/k3s-backups` — `compression=zstd`, `quota=100G`, **`xattr=sa`**
|
||||
(required — versitygw stores object metadata in POSIX extended attributes and refuses to start
|
||||
without them), `mountpoint=/mnt/Storage/k3s-backups`, **no sharenfs**. Uses `community.general.zfs`
|
||||
so re-runs are no-ops and existing datasets are never touched.
|
||||
3. **`versitygw` system user/group** + the gateway-root dir (`…/k3s-backups/data`) it owns.
|
||||
4. **versitygw binary** (pinned, checksummed). The Proxmox host has **no outbound internet**, so the
|
||||
release tarball is downloaded + verified + extracted on the **Ansible controller** and only the
|
||||
binary is copied to the host.
|
||||
5. **systemd unit** `versitygw.service` (Type=simple, hardened), `posix` backend rooted at the
|
||||
gateway root, bound to the host's **cluster-reachable `192.168.1.x` address** on port **9000**
|
||||
(same endpoint as the old MinIO) — auto-discovered from facts, assert-guarded. TLS off (plain
|
||||
HTTP, path-style).
|
||||
6. **Buckets** `velero` + `cnpg` — on the posix backend a bucket is simply a top-level subdirectory
|
||||
of the gateway root, so they are pre-created owned by the service user.
|
||||
|
||||
## Credentials
|
||||
|
||||
versitygw authenticates every S3 request against a single **ROOT** access key/secret
|
||||
(`ROOT_ACCESS_KEY` / `ROOT_SECRET_KEY`). This role sets it to the **same** scoped key/secret the
|
||||
cluster already uses, supplied via the vaulted `inventory/group_vars/proxmox_hosts/vault.yml`
|
||||
(ansible-vault). Nothing is defaulted — a missing vault fails the play loudly:
|
||||
|
||||
| var | purpose |
|
||||
|---|---|
|
||||
| `s3_backup_target_access_key` / `_secret_key` | versitygw ROOT key = the key handed to Velero + CNPG |
|
||||
|
||||
The **same** key/secret is staged for the GitOps side, SOPS-encrypted, at
|
||||
`platform/backup/minio-credentials.sops.yaml` — the two MUST stay in sync.
|
||||
|
||||
> Note: unlike MinIO, versitygw's single-binary posix backend has no separate "root admin" vs
|
||||
> "scoped user" distinction without enabling an external IAM backend. For a dedicated backup target
|
||||
> the scoped key **is** the root key — it only ever fronts the velero + cnpg buckets on this host.
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
cd infra/ansible
|
||||
ansible-galaxy collection install -r collections/requirements.yml # community.general (zfs)
|
||||
ansible-playbook playbooks/backup-s3.yml
|
||||
```
|
||||
|
||||
## Endpoint
|
||||
|
||||
`http://<host-192.168.1.x>:9000` — the S3 URL the cluster uses (path-style). An unauthenticated
|
||||
`GET /` returns `403 AccessDenied` once serving; a SigV4 `ListBuckets` with the root key returns
|
||||
`velero` + `cnpg`. Verified reachable from an in-cluster pod at bring-up.
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
# roles/s3_backup_target — a single-node S3 backup endpoint on the Proxmox host, used ONLY as an
|
||||
# off-cluster backup bucket store for Velero + CloudNativePG (ADR-0003, R-STATE-03 / R-IDP-05).
|
||||
# This is a BACKUP TARGET, not primary storage — a single drive/dataset is intentional.
|
||||
#
|
||||
# Implemented with versitygw (Versity Gateway, github.com/versity/versitygw): a maintained Go S3
|
||||
# gateway that serves a POSIX filesystem as S3. It REPLACES the earlier MinIO implementation (MinIO's
|
||||
# community edition was de-prioritised); the role also decommissions any pre-existing MinIO install
|
||||
# on the host so a re-run is a clean swap.
|
||||
#
|
||||
# SAFETY: every task in this role is additive/idempotent and SURGICAL. It touches ONLY the
|
||||
# minio→versitygw swap: the ONE ZFS dataset it owns (Storage/k3s-backups), the versitygw system
|
||||
# user, its systemd service + binary, and the two buckets inside that dataset. It never touches
|
||||
# other datasets, sharenfs, pvesm storage, VMs or firewall. See tasks/main.yml for the guards.
|
||||
|
||||
# ── versitygw release pin (Renovate-friendly) ─────────────────────────────────────────────────
|
||||
# The Proxmox host has NO outbound internet (PiHole-only DNS, no default route to WAN), so the
|
||||
# release tarball is fetched + verified on the ANSIBLE CONTROLLER, the binary extracted there, and
|
||||
# only the binary is copied to the host (tasks/main.yml).
|
||||
# Release assets: https://github.com/versity/versitygw/releases
|
||||
# tarball : versitygw_v{ver}_Linux_x86_64.tar.gz (extracts to versitygw_v{ver}_Linux_x86_64/versitygw)
|
||||
# checksums: checksums.txt (sha256) in the same release
|
||||
s3_backup_target_versitygw_version: "v1.7.0"
|
||||
s3_backup_target_versitygw_sha256: "dacec530fecaa8776feec0536998dd4bd78036e0d4db0da9b9189d96bb790e03"
|
||||
s3_backup_target_versitygw_url: "https://github.com/versity/versitygw/releases/download/{{ s3_backup_target_versitygw_version }}/versitygw_{{ s3_backup_target_versitygw_version }}_Linux_x86_64.tar.gz"
|
||||
# Path of the binary INSIDE the extracted tarball (GoReleaser lays it out under a version dir).
|
||||
s3_backup_target_versitygw_archive_member: "versitygw_{{ s3_backup_target_versitygw_version }}_Linux_x86_64/versitygw"
|
||||
|
||||
# Controller-side cache the tarball is fetched + extracted into (delegate_to localhost).
|
||||
s3_backup_target_controller_cache: "/tmp/s3-backup-target-cache"
|
||||
|
||||
# ── ZFS dataset (backup drive) ────────────────────────────────────────────────────────────────
|
||||
# Host convention is Storage/<purpose> mounted at /mnt/Storage/<purpose>. NO sharenfs.
|
||||
# xattr=sa is REQUIRED: versitygw's posix backend stores object metadata (ETags, etc.) in POSIX
|
||||
# extended attributes and refuses to start if the root FS can't hold them.
|
||||
s3_backup_target_zfs_dataset: "Storage/k3s-backups"
|
||||
s3_backup_target_mountpoint: "/mnt/Storage/k3s-backups"
|
||||
s3_backup_target_zfs_compression: "zstd"
|
||||
s3_backup_target_zfs_quota: "100G"
|
||||
s3_backup_target_zfs_xattr: "sa"
|
||||
# versitygw's gateway root — a subdir of the dataset, keeping the dataset root owned by root and
|
||||
# clean. Its top-level subdirectories ARE the S3 buckets.
|
||||
s3_backup_target_gateway_root: "{{ s3_backup_target_mountpoint }}/data"
|
||||
|
||||
# ── Service identity ────────────────────────────────────────────────────────────────────────
|
||||
s3_backup_target_user: "versitygw"
|
||||
s3_backup_target_group: "versitygw"
|
||||
s3_backup_target_bin: "/usr/local/bin/versitygw"
|
||||
s3_backup_target_env_file: "/etc/default/versitygw"
|
||||
|
||||
# ── Network binding ──────────────────────────────────────────────────────────────────────────
|
||||
# The S3 API must bind the host address the k3s pods can reach (the 192.168.1.0/24 LAN; nodes are
|
||||
# .50/.51/.52). tasks/main.yml auto-discovers the host's address in this CIDR prefix and asserts it
|
||||
# exists; override s3_backup_target_bind_address to pin it explicitly. Port 9000 keeps the endpoint
|
||||
# byte-identical to the previous MinIO install so nothing downstream (SOPS secret, Velero/CNPG)
|
||||
# changes. (versitygw's own default is :7070 — deliberately overridden here.)
|
||||
s3_backup_target_lan_prefix: "192.168.1."
|
||||
s3_backup_target_bind_address: "" # "" ⇒ auto-discover from facts (see tasks)
|
||||
s3_backup_target_api_port: 9000
|
||||
s3_backup_target_region: "us-east-1"
|
||||
|
||||
# ── Buckets ────────────────────────────────────────────────────────────────────────────────────
|
||||
# Velero and CNPG each get a bucket. On the posix backend a bucket == a top-level subdirectory of
|
||||
# the gateway root, so they are simply pre-created (owned by the service user). The single root key
|
||||
# below has full access to all buckets.
|
||||
s3_backup_target_buckets:
|
||||
- velero
|
||||
- cnpg
|
||||
|
||||
# ── Credentials ──────────────────────────────────────────────────────────────────────────────────
|
||||
# versitygw authenticates every S3 request against a single ROOT access key/secret (ROOT_ACCESS_KEY
|
||||
# / ROOT_SECRET_KEY). We set it to the SAME scoped key/secret the cluster already uses, so the
|
||||
# endpoint + access key + secret + buckets stay byte-identical to the previous MinIO target and the
|
||||
# GitOps side (SOPS: platform/backup/minio-credentials.sops.yaml) needs ZERO change.
|
||||
#
|
||||
# Supplied via the vaulted group_vars (inventory/group_vars/proxmox_hosts/vault.yml):
|
||||
# s3_backup_target_access_key, s3_backup_target_secret_key
|
||||
# Intentionally NOT defaulted here so a missing vault fails loudly rather than silently
|
||||
# provisioning guessable creds.
|
||||
|
||||
# ── MinIO decommission (of any pre-existing MinIO install this role is replacing) ──────────────
|
||||
# Paths/identities the OLD minio_backup_target role created, removed by the decommission block.
|
||||
s3_backup_target_minio_service: "minio.service"
|
||||
s3_backup_target_minio_bin: "/usr/local/bin/minio"
|
||||
s3_backup_target_minio_mc_bin: "/usr/local/bin/mc"
|
||||
s3_backup_target_minio_env_file: "/etc/default/minio"
|
||||
s3_backup_target_minio_policy_file: "/etc/minio-k3s-backup-rw.json"
|
||||
s3_backup_target_minio_user: "minio"
|
||||
s3_backup_target_minio_group: "minio"
|
||||
# MinIO's on-disk object layout left inside the gateway-root dir (cleared so versitygw starts
|
||||
# fresh; there are no real backups yet so this is safe). The dataset itself is KEPT.
|
||||
s3_backup_target_minio_leftovers:
|
||||
- "{{ s3_backup_target_gateway_root }}/.minio"
|
||||
- "{{ s3_backup_target_gateway_root }}/.minio.sys"
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
---
|
||||
- name: Restart minio
|
||||
- name: Restart versitygw
|
||||
ansible.builtin.systemd:
|
||||
name: minio
|
||||
name: versitygw
|
||||
state: restarted
|
||||
daemon_reload: true
|
||||
@@ -0,0 +1,231 @@
|
||||
---
|
||||
# ── 0. Preconditions / safety asserts ─────────────────────────────────────────────────────────
|
||||
- name: Assert S3 backup-target credentials are supplied (from vaulted group_vars)
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- s3_backup_target_access_key | default('') | length > 0
|
||||
- s3_backup_target_secret_key | default('') | length >= 8
|
||||
fail_msg: >-
|
||||
versitygw credentials are missing. Provide s3_backup_target_access_key/_secret_key via
|
||||
inventory/group_vars/proxmox_hosts/vault.yml (they MUST match the SOPS-staged scoped key at
|
||||
platform/backup/minio-credentials.sops.yaml so nothing downstream changes).
|
||||
quiet: true
|
||||
|
||||
# Determine the host's 192.168.1.0/24 address (the S3 endpoint the cluster reaches). If the
|
||||
# operator pinned s3_backup_target_bind_address, honour it; else auto-discover and assert.
|
||||
- name: Discover the host LAN (cluster-reachable) IPv4 address
|
||||
ansible.builtin.set_fact:
|
||||
s3_backup_target_bind_address: >-
|
||||
{{ s3_backup_target_bind_address
|
||||
if (s3_backup_target_bind_address | length > 0)
|
||||
else (ansible_facts.all_ipv4_addresses
|
||||
| select('match', '^' ~ (s3_backup_target_lan_prefix | regex_escape))
|
||||
| list | first | default('')) }}
|
||||
|
||||
- name: Assert a cluster-reachable LAN address was found
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- s3_backup_target_bind_address | length > 0
|
||||
fail_msg: >-
|
||||
Could not find a {{ s3_backup_target_lan_prefix }}x address on this host
|
||||
(interfaces: {{ ansible_facts.all_ipv4_addresses | default([]) | join(', ') }}).
|
||||
The cluster egresses onto that LAN — refusing to bind the S3 endpoint to an unreachable
|
||||
address. Set s3_backup_target_bind_address explicitly if discovery is wrong.
|
||||
quiet: true
|
||||
|
||||
# ── 1. Decommission any pre-existing MinIO install this role replaces (idempotent + surgical) ───
|
||||
# All steps tolerate absence, so on a fresh host (no MinIO) they are harmless no-ops. The ZFS
|
||||
# dataset and mount are KEPT — only MinIO's service/binary/env/mc-client/object-layout go.
|
||||
- name: Check whether the old MinIO systemd unit is present
|
||||
ansible.builtin.stat:
|
||||
path: "/etc/systemd/system/{{ s3_backup_target_minio_service }}"
|
||||
register: s3_backup_target_minio_unit
|
||||
|
||||
- name: Stop + disable the old MinIO service (only if its unit exists)
|
||||
ansible.builtin.systemd:
|
||||
name: "{{ s3_backup_target_minio_service }}"
|
||||
state: stopped
|
||||
enabled: false
|
||||
when: s3_backup_target_minio_unit.stat.exists
|
||||
|
||||
- name: Remove the old MinIO systemd unit
|
||||
ansible.builtin.file:
|
||||
path: "/etc/systemd/system/{{ s3_backup_target_minio_service }}"
|
||||
state: absent
|
||||
register: s3_backup_target_minio_unit_removed
|
||||
|
||||
- name: Reload systemd if the MinIO unit was removed
|
||||
ansible.builtin.systemd:
|
||||
daemon_reload: true
|
||||
when: s3_backup_target_minio_unit_removed.changed
|
||||
|
||||
- name: Remove MinIO server + mc client binaries, env file and IAM policy file
|
||||
ansible.builtin.file:
|
||||
path: "{{ item }}"
|
||||
state: absent
|
||||
loop:
|
||||
- "{{ s3_backup_target_minio_bin }}"
|
||||
- "{{ s3_backup_target_minio_mc_bin }}"
|
||||
- "{{ s3_backup_target_minio_env_file }}"
|
||||
- "{{ s3_backup_target_minio_policy_file }}"
|
||||
|
||||
- name: Remove the mc client config left in root's home
|
||||
ansible.builtin.file:
|
||||
path: "{{ item }}"
|
||||
state: absent
|
||||
loop:
|
||||
- /root/.mc
|
||||
- /root/.mcli
|
||||
|
||||
- name: Clear MinIO's on-disk object layout from the gateway-root dir (dataset itself KEPT)
|
||||
ansible.builtin.file:
|
||||
path: "{{ item }}"
|
||||
state: absent
|
||||
loop: "{{ s3_backup_target_minio_leftovers }}"
|
||||
|
||||
- name: Remove the old minio system user (its files are cleared / re-owned below)
|
||||
ansible.builtin.user:
|
||||
name: "{{ s3_backup_target_minio_user }}"
|
||||
state: absent
|
||||
remove: false
|
||||
failed_when: false
|
||||
|
||||
- name: Remove the old minio system group
|
||||
ansible.builtin.group:
|
||||
name: "{{ s3_backup_target_minio_group }}"
|
||||
state: absent
|
||||
failed_when: false
|
||||
|
||||
# ── 2. ZFS dataset (backup drive) — idempotent, never touches other datasets ───────────────────
|
||||
# xattr=sa is required by versitygw's posix backend (object metadata is stored in extended attrs).
|
||||
- name: Create/ensure the k3s-backups ZFS dataset (compression, quota, xattr=sa, mountpoint; NO sharenfs)
|
||||
community.general.zfs:
|
||||
name: "{{ s3_backup_target_zfs_dataset }}"
|
||||
state: present
|
||||
extra_zfs_properties:
|
||||
compression: "{{ s3_backup_target_zfs_compression }}"
|
||||
quota: "{{ s3_backup_target_zfs_quota }}"
|
||||
xattr: "{{ s3_backup_target_zfs_xattr }}"
|
||||
mountpoint: "{{ s3_backup_target_mountpoint }}"
|
||||
|
||||
# ── 3. Service identity ───────────────────────────────────────────────────────────────────────
|
||||
- name: Create the versitygw system group
|
||||
ansible.builtin.group:
|
||||
name: "{{ s3_backup_target_group }}"
|
||||
system: true
|
||||
|
||||
- name: Create the versitygw system user (no login shell, home = gateway root)
|
||||
ansible.builtin.user:
|
||||
name: "{{ s3_backup_target_user }}"
|
||||
group: "{{ s3_backup_target_group }}"
|
||||
system: true
|
||||
shell: /usr/sbin/nologin
|
||||
home: "{{ s3_backup_target_gateway_root }}"
|
||||
create_home: false
|
||||
|
||||
- name: Ensure the gateway root dir exists (owned by versitygw, inside the dataset)
|
||||
ansible.builtin.file:
|
||||
path: "{{ s3_backup_target_gateway_root }}"
|
||||
state: directory
|
||||
owner: "{{ s3_backup_target_user }}"
|
||||
group: "{{ s3_backup_target_group }}"
|
||||
mode: "0750"
|
||||
|
||||
- name: Pre-create the S3 buckets (a bucket == a top-level subdir of the gateway root)
|
||||
ansible.builtin.file:
|
||||
path: "{{ s3_backup_target_gateway_root }}/{{ item }}"
|
||||
state: directory
|
||||
owner: "{{ s3_backup_target_user }}"
|
||||
group: "{{ s3_backup_target_group }}"
|
||||
mode: "0750"
|
||||
loop: "{{ s3_backup_target_buckets }}"
|
||||
|
||||
# ── 4. Binary — fetched + extracted on the CONTROLLER (host has no internet), copied to the host ─
|
||||
- name: Ensure controller-side download cache exists
|
||||
ansible.builtin.file:
|
||||
path: "{{ s3_backup_target_controller_cache }}"
|
||||
state: directory
|
||||
mode: "0700"
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
run_once: true
|
||||
|
||||
- name: Fetch the versitygw release tarball onto the controller (checksummed)
|
||||
ansible.builtin.get_url:
|
||||
url: "{{ s3_backup_target_versitygw_url }}"
|
||||
dest: "{{ s3_backup_target_controller_cache }}/versitygw_{{ s3_backup_target_versitygw_version }}.tar.gz"
|
||||
checksum: "sha256:{{ s3_backup_target_versitygw_sha256 }}"
|
||||
mode: "0600"
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
run_once: true
|
||||
|
||||
- name: Extract the versitygw binary on the controller
|
||||
ansible.builtin.unarchive:
|
||||
src: "{{ s3_backup_target_controller_cache }}/versitygw_{{ s3_backup_target_versitygw_version }}.tar.gz"
|
||||
dest: "{{ s3_backup_target_controller_cache }}"
|
||||
remote_src: true
|
||||
creates: "{{ s3_backup_target_controller_cache }}/{{ s3_backup_target_versitygw_archive_member }}"
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
run_once: true
|
||||
|
||||
# Integrity is already guaranteed by the SHA256 get_url performed on the controller; copy transfers
|
||||
# that verified artifact faithfully. (The copy module's own `checksum` arg is SHA1 — not used here.)
|
||||
- name: Install the versitygw binary on the host
|
||||
ansible.builtin.copy:
|
||||
src: "{{ s3_backup_target_controller_cache }}/{{ s3_backup_target_versitygw_archive_member }}"
|
||||
dest: "{{ s3_backup_target_bin }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
notify: Restart versitygw
|
||||
|
||||
# ── 5. systemd service ────────────────────────────────────────────────────────────────────────
|
||||
- name: Write the versitygw environment file (root access key/secret)
|
||||
ansible.builtin.template:
|
||||
src: versitygw.env.j2
|
||||
dest: "{{ s3_backup_target_env_file }}"
|
||||
owner: root
|
||||
group: "{{ s3_backup_target_group }}"
|
||||
mode: "0640"
|
||||
no_log: true
|
||||
notify: Restart versitygw
|
||||
|
||||
- name: Install the versitygw systemd unit
|
||||
ansible.builtin.template:
|
||||
src: versitygw.service.j2
|
||||
dest: /etc/systemd/system/versitygw.service
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
notify: Restart versitygw
|
||||
|
||||
- name: Enable + start versitygw (apply any pending unit/env changes now)
|
||||
ansible.builtin.systemd:
|
||||
name: versitygw
|
||||
enabled: true
|
||||
state: started
|
||||
daemon_reload: true
|
||||
|
||||
- name: Flush handlers so versitygw is (re)started before we verify it
|
||||
ansible.builtin.meta: flush_handlers
|
||||
|
||||
# ── 6. Readiness gate ─────────────────────────────────────────────────────────────────────────
|
||||
# versitygw has no health endpoint; an UNAUTHENTICATED GET / returns 403 (AccessDenied) once the
|
||||
# S3 server is up and enforcing SigV4 — a reliable "listening + serving" signal.
|
||||
- name: Wait for the versitygw S3 port to accept connections
|
||||
ansible.builtin.wait_for:
|
||||
host: "{{ s3_backup_target_bind_address }}"
|
||||
port: "{{ s3_backup_target_api_port }}"
|
||||
timeout: 60
|
||||
|
||||
- name: Confirm versitygw is serving S3 (unauthenticated GET / ⇒ 403 AccessDenied)
|
||||
ansible.builtin.uri:
|
||||
url: "http://{{ s3_backup_target_bind_address }}:{{ s3_backup_target_api_port }}/"
|
||||
method: GET
|
||||
status_code: 403
|
||||
register: s3_backup_target_probe
|
||||
retries: 15
|
||||
delay: 2
|
||||
until: s3_backup_target_probe.status == 403
|
||||
@@ -0,0 +1,10 @@
|
||||
# {{ ansible_managed }}
|
||||
# versitygw root credentials (single-node posix backup target for k3s Velero + CNPG).
|
||||
# Managed by roles/s3_backup_target — do not edit by hand.
|
||||
#
|
||||
# versitygw authenticates every S3 request against this ROOT key. It is set to the SAME scoped
|
||||
# access key/secret the cluster already uses (mirrored via SOPS in
|
||||
# platform/backup/minio-credentials.sops.yaml) so the endpoint + key + secret stay byte-identical
|
||||
# to the previous MinIO target — nothing downstream changes.
|
||||
ROOT_ACCESS_KEY={{ s3_backup_target_access_key }}
|
||||
ROOT_SECRET_KEY={{ s3_backup_target_secret_key }}
|
||||
@@ -0,0 +1,35 @@
|
||||
{# {{ ansible_managed }} — versitygw systemd unit (roles/s3_backup_target) #}
|
||||
[Unit]
|
||||
Description=versitygw S3 backup target (k3s Velero + CNPG)
|
||||
Documentation=https://github.com/versity/versitygw
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
AssertFileIsExecutable={{ s3_backup_target_bin }}
|
||||
AssertPathIsDirectory={{ s3_backup_target_gateway_root }}
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User={{ s3_backup_target_user }}
|
||||
Group={{ s3_backup_target_group }}
|
||||
# ROOT_ACCESS_KEY / ROOT_SECRET_KEY are read from here (kept out of the process args).
|
||||
EnvironmentFile={{ s3_backup_target_env_file }}
|
||||
# Global options (--port/--region) come BEFORE the `posix` backend subcommand; posix's single
|
||||
# positional arg is the gateway root, whose top-level subdirs are the S3 buckets. TLS off (no
|
||||
# --cert/--key) ⇒ plain HTTP, path-style.
|
||||
ExecStart={{ s3_backup_target_bin }} \
|
||||
--port {{ s3_backup_target_bind_address }}:{{ s3_backup_target_api_port }} \
|
||||
--region {{ s3_backup_target_region }} \
|
||||
posix {{ s3_backup_target_gateway_root }}
|
||||
Restart=always
|
||||
RestartSec=5s
|
||||
LimitNOFILE=65536
|
||||
TimeoutStopSec=30s
|
||||
# Hardening (backup target holds no other roles on this host).
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
PrivateTmp=true
|
||||
ReadWritePaths={{ s3_backup_target_mountpoint }}
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Reference in new issue
Block a user