jakman85andClaude Opus 4.8
485b9e3888
Local-only tfstate meant losing the workstation would strand a rebuild (apply would try to recreate the 3 live nodes). Move state to Cloudflare R2, which sits OUTSIDE the Proxmox->VM->cluster->Gitea chain, so a full cluster rebuild can still read it; encrypted at rest via OpenTofu native AES-GCM. Adds a partial s3 backend (R2 skip-flags), gitignored backend.hcl/.r2.env for the account-specific + secret bits, their .example templates, and a migration+DR runbook. Locking (use_lockfile) deferred to an OpenTofu >=1.10 bump (1.9.0 lacks it). Migration (tofu init -migrate-state) runs once the operator provisions the R2 bucket. Co-Authored-By: Claude Opus 4.8 <[email protected]>
62 lines
3.4 KiB
Markdown
62 lines
3.4 KiB
Markdown
# Tofu remote state — Cloudflare R2 (off-cluster, encrypted)
|
|
|
|
## Why
|
|
State was **local** (`terraform.tfstate` on one workstation, git-ignored). Losing that box meant a
|
|
`tofu apply` from anywhere else would try to **recreate the three live k3s nodes**. State must live
|
|
**off the cluster it manages** — the chain is `Proxmox host → k3s VMs → k3s cluster → Gitea/apps`, so
|
|
an in-cluster backend (Gitea) is unreadable in the one scenario it exists for (cluster gone, rebuild).
|
|
|
|
**Cloudflare R2** sits outside that chain entirely: a full cluster rebuild can still read state, it
|
|
survives total-site loss (off-site), and it works from any machine. State bytes are **encrypted at
|
|
rest** (OpenTofu native AES-GCM) so R2 only ever holds ciphertext.
|
|
|
|
## One-time setup (operator)
|
|
|
|
### 1. Create the R2 bucket + token (Cloudflare dashboard)
|
|
1. **R2 → Create bucket** → name `homelab-tofu-state` (any name; match `backend.hcl`).
|
|
2. **R2 → Manage API Tokens → Create API Token** → permission **Object Read & Write**, scoped to
|
|
that bucket. Copy the **Access Key ID** and **Secret Access Key** (shown once).
|
|
3. Note your **Account ID** (32 hex chars, in the dashboard URL / the R2 S3 endpoint).
|
|
|
|
### 2. Fill the two gitignored files (secrets stay out of git and out of chat)
|
|
```bash
|
|
cd infra/tofu
|
|
cp backend.hcl.example backend.hcl # set bucket/key + endpoints.s3 = https://<ACCOUNT_ID>.r2.cloudflarestorage.com
|
|
cp .r2.env.example .r2.env # set AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY + a strong TF_ENCRYPTION passphrase
|
|
```
|
|
**Store the encryption passphrase in your password manager.** It is required to read state on any
|
|
machine and after any rebuild. Also record the R2 keys there. (Both files are in `.gitignore`.)
|
|
|
|
### 3. Migrate the existing local state -> R2 (one shot, moves state only — touches NO infra)
|
|
```bash
|
|
cd infra/tofu
|
|
cp terraform.tfstate terraform.tfstate.premigrate.bak # extra belt-and-suspenders backup
|
|
source .r2.env
|
|
tofu init -backend-config=backend.hcl -migrate-state # answer "yes" to copy local -> R2
|
|
tofu plan # MUST report "No changes." — proves parity
|
|
```
|
|
If `plan` shows no changes, the migration is correct. If it wants to create/destroy nodes, STOP —
|
|
the state did not carry over; re-check backend.hcl/creds and do not apply.
|
|
|
|
## Day-2 use (any machine)
|
|
```bash
|
|
cd infra/tofu
|
|
source .r2.env # R2 creds + TF_ENCRYPTION passphrase
|
|
tofu init -backend-config=backend.hcl
|
|
tofu plan
|
|
```
|
|
That's the whole "work from a different computer" story: clone the repo, create the two local files
|
|
from the templates (creds + passphrase from your password manager), `init`, done.
|
|
|
|
## DR: rebuild from nothing
|
|
Because state is in R2 (not the cluster), a destroyed cluster is recoverable: on a fresh box, install
|
|
OpenTofu, create `backend.hcl` + `.r2.env`, `tofu init -backend-config=backend.hcl`, then `tofu apply`
|
|
to rebuild the VMs against the preserved state. R2 itself should ALSO be covered by the off-cluster
|
|
backup target (OQ-BKP) so state survives even an R2/account loss.
|
|
|
|
## Follow-up (tracked)
|
|
- **Locking.** OpenTofu 1.9.0 has no `use_lockfile`; state locking is currently OFF (safe for a single
|
|
operator). After bumping the toolchain to **OpenTofu ≥ 1.10** (`infra/devcontainer/versions.env`
|
|
+ `versions.tf` `required_version`), set `use_lockfile = true` in `backend.tf` for native R2
|
|
conditional-write locking (no DynamoDB) — do this before regular two-machine use.
|