Replace MinIO backup target with versitygw (s3_backup_target role)

MinIO's open-source edition is de-prioritised (console removed, commercial
pivot), so swap the off-cluster S3 backup target on the Proxmox host to
versitygw v1.7.0 (github.com/versity/versitygw) — a maintained Go S3 gateway
serving a POSIX filesystem as S3.

Rename role minio_backup_target -> s3_backup_target and playbook backup-minio
-> backup-s3. The role now:
  - decommissions any pre-existing MinIO (service/binary/env/mc-client/user/
    object-layout), KEEPING the ZFS dataset Storage/k3s-backups and its mount
  - installs the pinned, checksummed versitygw binary (fetched+extracted on the
    controller, host has no internet), a dedicated versitygw system user, and a
    Type=simple hardened systemd unit running the posix backend rooted at
    /mnt/Storage/k3s-backups/data, bound to 192.168.1.9:9000 (same endpoint),
    plain HTTP / path-style
  - ensures the dataset has xattr=sa (versitygw stores object metadata in xattrs)
  - pre-creates the velero + cnpg buckets (bucket == top-level subdir)

Credentials are REUSED: versitygw's ROOT key is set to the same scoped
access_key/secret the cluster already uses, so endpoint + key + secret + bucket
names are byte-identical to the old MinIO target. The GitOps SOPS secret
(platform/backup/minio-credentials.sops.yaml) and all cluster-side Velero/CNPG
config need ZERO change. The vaulted group_vars now expose
s3_backup_target_access_key/_secret_key (same values, MinIO root creds retired).

Verified live on the Proxmox host: versitygw active+enabled on 192.168.1.9:9000,
MinIO fully removed, dataset intact; SigV4 ListBuckets with the reused key
returns velero+cnpg; a throwaway in-cluster pod reaches http://192.168.1.9:9000
(versitygw S3 response). Re-run is idempotent (no host changes).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed 2026-08-28 22:48:38 -05:00
1 parent 1bee2554f8
commit 6739339f53
17 files changed
+499 -491

No files matched your search

@@ -1,29 +1,27 @@
# group_vars for the `proxmox_hosts` group (the Proxmox hypervisor host)
Target of `playbooks/backup-minio.yml` (role `minio_backup_target`). The role needs the MinIO
credentials as **ansible-vault-encrypted** vars here — encrypted, so safe to commit; the passphrase
that decrypts them stays out of the repo (`ansible.cfg` → `vault_password_file`).
Target of `playbooks/backup-s3.yml` (role `s3_backup_target`, versitygw). The role needs the S3
backup-target credentials as **ansible-vault-encrypted** vars here — encrypted, so safe to commit;
the passphrase that decrypts them stays out of the repo (`ansible.cfg` → `vault_password_file`).
## `vault.yml` (committed, encrypted)
```bash
cd infra/ansible
ansible-vault create inventory/group_vars/proxmox_hosts/vault.yml
ansible-vault edit inventory/group_vars/proxmox_hosts/vault.yml
# contents:
# minio_backup_target_root_user: "…" # MinIO root (host admin only)
# minio_backup_target_root_password: "…"
# minio_backup_target_access_key: "k3s-backup" # scoped key handed to Velero + CNPG
# minio_backup_target_secret_key: "…"
# s3_backup_target_access_key: "k3s-backup" # versitygw ROOT key = key handed to Velero + CNPG
# s3_backup_target_secret_key: "…"
```
The **scoped** access key/secret is mirrored (SOPS-encrypted) for the GitOps side at
`platform/backup/minio-credentials.sops.yaml` — the two MUST stay in sync (same key/secret), since
the cluster authenticates to this MinIO with them. Rotating means updating BOTH and re-running the
playbook (recreates the MinIO user) + reconciling Flux.
versitygw authenticates every S3 request against this single **ROOT** key. It is mirrored
(SOPS-encrypted) for the GitOps side at `platform/backup/minio-credentials.sops.yaml` — the two MUST
stay in sync (same key/secret), since the cluster authenticates to this endpoint with them. Rotating
means updating BOTH and re-running the playbook (rewrites the env file + restarts versitygw) +
reconciling Flux.
## Notes
- This is the **Ansible-bootstrap-layer** secret (host credential), distinct from the platform's
in-cluster **SOPS + age** secret (ADR-0007) — different layer, different tool, same values for the
scoped key.
in-cluster **SOPS + age** secret (ADR-0007) — different layer, different tool, same values.
- `vault.yml` is a vault blob (not parseable YAML) — excluded from the yamllint gate like the other
`**/vault.yml`.
@@ -1,29 +1,25 @@
$ANSIBLE_VAULT;1.1;AES256
30633837396565613861623931383238383836356238326436633738613935326461346138373432
6234643535303037313337363463633765326239376238380a333331333136396239313733366163
33353036366461393530633139636430313736366338313331353163616135303930323038636134
6366353362313537330a373863623861663537353539376664306338643539346139396130396632
30336231363731616334383863326638636238323565316364303530396636373163393765376438
63373263623063656662353434623339316238663535386465653164326236616265616638656639
30613466656439396534316339646139343334656165323965323965313361633933633038376663
31653539653763623132313533396333643964373032313333613432383433626338376431323731
63316163363361303439313266356361613737333066643931656564633038306236363439666435
39343536656336343834663464356566373564636531343534343365376332343832343235663339
66383561306533323938613738376632626263306337346532643063373166333838333033346365
32383136363636643163613334366631653238613166303962656463363663643830343531363533
39333630323765643963396562363537396564666462306163336366643739313035636633646661
61653637376535393034383631303563363539633265333161363334616239323866353162636632
65663830323465393138306361333662383934346438663562316235353035633064643861353334
63646435653639316439366139316539653833656561646439323766633139306565383565656261
61323230306635343236646438336165643737626132613962633930653461393131303731623438
63313265653166643334396461353231343235386434363438623034323762653433303932326630
62633464316139353433366338313837646162323231363135353964623035366132643332366662
38393465326363616139376465383738626636393639363338653437323832613439313138353033
39393765613735666331636230363463306462373238623138316263326339366330333332643833
33363764303939326662343665643963333463356638613231363834386132666339393132356538
38303362643337366430303165633432666632343536383366653836663135383336316232393936
66626632386537333739313432313564633866383132353862626433626366633562346464613066
35376332303132326438313132303264313763376163396137356132306131373335346266646532
31666230616461386339366461653661323138353662373866316639626134326339643231613337
30343361643166353162363866653061313966663435373332353562656164663962326562333038
35363834656361393930
33623463393666373861613531626336396164356361643939326332626634663234653164313437
3963326465326364643639633066636564656565333461360a383438323530663932326462376661
64343766366132373436366336353338383365396563636138333333306330643337636532383662
6561323836656637380a656436656538613263646461386635303762376163363161336436386263
34653437386364363161653132343434333230393732373364383564336162643136636437376231
39376463336666396462363431636161306632326134663038343364626232333838333039393338
64313762343363666139393864366561316362633465323361633132373533356531353534396433
62333131356630343764303438663766623737333137383264623739646539373535306136303530
34663063383032643238373537663633343735613265653964356638306566636538363063656435
65643666366665393238363037313337633366333338633163316533656134336463363630353961
37656637343063373266643237663633613638303165656262313133313535313665336335333039
30626537373530663430343637386263353765353538376263333130623536306635353966393561
62313362373061303035376635646335303133613365326163643566333031356436326464616430
34373433343733386666626234623064383063393330343034363036663632373865643636623961
31666131636538303131376339623733353964653266356339313466616363646336663433666631
61663564633931626639323235633332366338333837316535376330633534626537666166643262
63646239656165303366626163613238393266336566316164356330393836323332336330633635
64636636613164366338396666316137353837313639323630353234373936393335366364666361
35356163363636623031333733653432636539396235633436313334333663343532303239373238
37646230326164396339386339356663396661306165343630316261306331383632626662323234
31396539313766356633333736383036623839626663383763626332626237373338653832646463
34366434386139306336633964313939643565363838346163656236656561363865386136346530
36316661323262323366646332316535306637316663656264303137653831656136633432643765
61646433653530646339
+2 -2
View File
@@ -38,8 +38,8 @@ localhost ansible_connection=local
[vpn_egress_gateway]
k3s-dev-3
# The Proxmox HYPERVISOR host itself (NOT a k3s node) — target of playbooks/backup-minio.yml,
# which stands up the off-cluster MinIO S3 backup target for Velero + CNPG. Reached DIRECTLY
# The Proxmox HYPERVISOR host itself (NOT a k3s node) — target of playbooks/backup-s3.yml,
# which stands up the off-cluster versitygw S3 backup target for Velero + CNPG. Reached DIRECTLY
# over SSH as root (its own 192.168.2.102:22, the pfSense NAT to the host's 192.168.1.9), so
# it overrides the cluster-wide `ansible_user: k3s` from group_vars/all.yml. Distinct from the
# `hypervisor` group above (that is `localhost` for host-prereqs.yml, a different machine).