jakman85andClaude Opus 4.8
df947b33d0
backend.tf began with a stray YAML `---` doc separator (invalid HCL) that broke `tofu` parsing. Remove it. Also document the one-time `unencrypted` fallback method needed to adopt encryption on pre-existing plaintext state (add for the migrate, remove after), so the record is complete. State migration to R2 verified: 4/4 resources read back through the encrypt-only method (proves ciphertext at rest). Co-Authored-By: Claude Opus 4.8 <[email protected]>
71 lines
4.2 KiB
Markdown
71 lines
4.2 KiB
Markdown
# Tofu remote state — Cloudflare R2 (off-cluster, encrypted)
|
|
|
|
## Why
|
|
State was **local** (`terraform.tfstate` on one workstation, git-ignored). Losing that box meant a
|
|
`tofu apply` from anywhere else would try to **recreate the three live k3s nodes**. State must live
|
|
**off the cluster it manages** — the chain is `Proxmox host → k3s VMs → k3s cluster → Gitea/apps`, so
|
|
an in-cluster backend (Gitea) is unreadable in the one scenario it exists for (cluster gone, rebuild).
|
|
|
|
**Cloudflare R2** sits outside that chain entirely: a full cluster rebuild can still read state, it
|
|
survives total-site loss (off-site), and it works from any machine. State bytes are **encrypted at
|
|
rest** (OpenTofu native AES-GCM) so R2 only ever holds ciphertext.
|
|
|
|
## One-time setup (operator)
|
|
|
|
### 1. Create the R2 bucket + token (Cloudflare dashboard)
|
|
1. **R2 → Create bucket** → name `homelab-tofu-state` (any name; match `backend.hcl`).
|
|
2. **R2 → Manage API Tokens → Create API Token** → permission **Object Read & Write**, scoped to
|
|
that bucket. Copy the **Access Key ID** and **Secret Access Key** (shown once).
|
|
3. Note your **Account ID** (32 hex chars, in the dashboard URL / the R2 S3 endpoint).
|
|
|
|
### 2. Fill the two gitignored files (secrets stay out of git and out of chat)
|
|
```bash
|
|
cd infra/tofu
|
|
cp backend.hcl.example backend.hcl # set bucket/key + endpoints.s3 = https://<ACCOUNT_ID>.r2.cloudflarestorage.com
|
|
cp .r2.env.example .r2.env # set AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY + a strong TF_ENCRYPTION passphrase
|
|
```
|
|
**Store the encryption passphrase in your password manager.** It is required to read state on any
|
|
machine and after any rebuild. Also record the R2 keys there. (Both files are in `.gitignore`.)
|
|
|
|
### 3. Migrate the existing local state -> R2 (one shot, moves state only — touches NO infra)
|
|
```bash
|
|
cd infra/tofu
|
|
cp terraform.tfstate terraform.tfstate.premigrate.bak # extra belt-and-suspenders backup
|
|
source .r2.env
|
|
tofu init -backend-config=backend.hcl -migrate-state # answer "yes" to copy local -> R2
|
|
tofu plan # MUST report "No changes." — proves parity
|
|
```
|
|
If `plan` shows no changes, the migration is correct. If it wants to create/destroy nodes, STOP —
|
|
the state did not carry over; re-check backend.hcl/creds and do not apply.
|
|
|
|
> **Adopting encryption on pre-existing *unencrypted* state (one-time — already done on first
|
|
> migration).** OpenTofu refuses to read a plaintext state while encryption is configured
|
|
> (`encountered unencrypted payload without unencrypted method configured`). For that ONE migration
|
|
> only, `TF_ENCRYPTION` must include a fallback so tofu can read plaintext and rewrite it encrypted:
|
|
> add `method "unencrypted" "migrate" {}` and give the `state` block a
|
|
> `fallback { method = method.unencrypted.migrate }`. Run the migrate, then **remove both** and
|
|
> `tofu init -reconfigure` so encryption is enforced. A fresh machine adopting the ALREADY-encrypted
|
|
> R2 state does NOT need this — the encrypt-only `.r2.env.example` is correct as-is.
|
|
|
|
## Day-2 use (any machine)
|
|
```bash
|
|
cd infra/tofu
|
|
source .r2.env # R2 creds + TF_ENCRYPTION passphrase
|
|
tofu init -backend-config=backend.hcl
|
|
tofu plan
|
|
```
|
|
That's the whole "work from a different computer" story: clone the repo, create the two local files
|
|
from the templates (creds + passphrase from your password manager), `init`, done.
|
|
|
|
## DR: rebuild from nothing
|
|
Because state is in R2 (not the cluster), a destroyed cluster is recoverable: on a fresh box, install
|
|
OpenTofu, create `backend.hcl` + `.r2.env`, `tofu init -backend-config=backend.hcl`, then `tofu apply`
|
|
to rebuild the VMs against the preserved state. R2 itself should ALSO be covered by the off-cluster
|
|
backup target (OQ-BKP) so state survives even an R2/account loss.
|
|
|
|
## Follow-up (tracked)
|
|
- **Locking.** OpenTofu 1.9.0 has no `use_lockfile`; state locking is currently OFF (safe for a single
|
|
operator). After bumping the toolchain to **OpenTofu ≥ 1.10** (`infra/devcontainer/versions.env`
|
|
+ `versions.tf` `required_version`), set `use_lockfile = true` in `backend.tf` for native R2
|
|
conditional-write locking (no DynamoDB) — do this before regular two-machine use.
|