home-network-k3s-impl/infra/tofu/REMOTE-STATE.md

Tofu remote state — Cloudflare R2 (off-cluster, encrypted)

Why

State was local (terraform.tfstate on one workstation, git-ignored). Losing that box meant a tofu apply from anywhere else would try to recreate the three live k3s nodes. State must live off the cluster it manages — the chain is Proxmox host → k3s VMs → k3s cluster → Gitea/apps, so an in-cluster backend (Gitea) is unreadable in the one scenario it exists for (cluster gone, rebuild).

Cloudflare R2 sits outside that chain entirely: a full cluster rebuild can still read state, it survives total-site loss (off-site), and it works from any machine. State bytes are encrypted at rest (OpenTofu native AES-GCM) so R2 only ever holds ciphertext.

One-time setup (operator)

1. Create the R2 bucket + token (Cloudflare dashboard)

1. R2 → Create bucket → name homelab-tofu-state (any name; match backend.hcl).
2. R2 → Manage API Tokens → Create API Token → permission Object Read & Write, scoped to that bucket. Copy the Access Key ID and Secret Access Key (shown once).
3. Note your Account ID (32 hex chars, in the dashboard URL / the R2 S3 endpoint).

2. Fill the two gitignored files (secrets stay out of git and out of chat)

cd infra/tofu
cp backend.hcl.example backend.hcl     # set bucket/key + endpoints.s3 = https://<ACCOUNT_ID>.r2.cloudflarestorage.com
cp .r2.env.example     .r2.env         # set AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY + a strong TF_ENCRYPTION passphrase

Store the encryption passphrase in your password manager. It is required to read state on any machine and after any rebuild. Also record the R2 keys there. (Both files are in .gitignore.)

3. Migrate the existing local state -> R2 (one shot, moves state only — touches NO infra)

cd infra/tofu
cp terraform.tfstate terraform.tfstate.premigrate.bak   # extra belt-and-suspenders backup
source .r2.env
tofu init -backend-config=backend.hcl -migrate-state     # answer "yes" to copy local -> R2
tofu plan                                                # MUST report "No changes." — proves parity

If plan shows no changes, the migration is correct. If it wants to create/destroy nodes, STOP — the state did not carry over; re-check backend.hcl/creds and do not apply.

Adopting encryption on pre-existing unencrypted state (one-time — already done on first migration). OpenTofu refuses to read a plaintext state while encryption is configured (encountered unencrypted payload without unencrypted method configured). For that ONE migration only, TF_ENCRYPTION must include a fallback so tofu can read plaintext and rewrite it encrypted: add method "unencrypted" "migrate" {} and give the state block a fallback { method = method.unencrypted.migrate }. Run the migrate, then remove both and tofu init -reconfigure so encryption is enforced. A fresh machine adopting the ALREADY-encrypted R2 state does NOT need this — the encrypt-only .r2.env.example is correct as-is.

Day-2 use (any machine)

cd infra/tofu
source .r2.env                 # R2 creds + TF_ENCRYPTION passphrase
tofu init -backend-config=backend.hcl
tofu plan

That's the whole "work from a different computer" story: clone the repo, create the two local files from the templates (creds + passphrase from your password manager), init, done.

DR: rebuild from nothing

Because state is in R2 (not the cluster), a destroyed cluster is recoverable: on a fresh box, install OpenTofu, create backend.hcl + .r2.env, tofu init -backend-config=backend.hcl, then tofu apply to rebuild the VMs against the preserved state. R2 itself should ALSO be covered by the off-cluster backup target (OQ-BKP) so state survives even an R2/account loss.

Follow-up (tracked)

• Locking. OpenTofu 1.9.0 has no use_lockfile; state locking is currently OFF (safe for a single operator). After bumping the toolchain to OpenTofu ≥ 1.10 (infra/devcontainer/versions.env
  • versions.tf required_version), set use_lockfile = true in backend.tf for native R2 conditional-write locking (no DynamoDB) — do this before regular two-machine use.