home-network-k3s-impl/infra/tofu/REMOTE-STATE.md

Tofu remote state — Cloudflare R2 (off-cluster, encrypted)

Why

State was local (terraform.tfstate on one workstation, git-ignored). Losing that box meant a tofu apply from anywhere else would try to recreate the three live k3s nodes. State must live off the cluster it manages — the chain is Proxmox host → k3s VMs → k3s cluster → Gitea/apps, so an in-cluster backend (Gitea) is unreadable in the one scenario it exists for (cluster gone, rebuild).

Cloudflare R2 sits outside that chain entirely: a full cluster rebuild can still read state, it survives total-site loss (off-site), and it works from any machine. State bytes are encrypted at rest (OpenTofu native AES-GCM) so R2 only ever holds ciphertext.

One-time setup (operator)

1. Create the R2 bucket + token (Cloudflare dashboard)

1. R2 → Create bucket → name homelab-tofu-state (any name; match backend.hcl).
2. R2 → Manage API Tokens → Create API Token → permission Object Read & Write, scoped to that bucket. Copy the Access Key ID and Secret Access Key (shown once).
3. Note your Account ID (32 hex chars, in the dashboard URL / the R2 S3 endpoint).

2. Fill in the encrypted secrets file (edited in place; ciphertext is committed)

All five secrets live in ONE committed, SOPS/age-encrypted dotenv, infra/tofu/secrets.sops.env. Edit it inside the dev container (it has sops + the age key). sops decrypts to your $EDITOR and re-encrypts on save — plaintext never touches disk:

./infra/devcontainer/dev.sh run sops infra/tofu/secrets.sops.env

Replace every REPLACE-ME:

• PROXMOX_VE_API_TOKEN — user@realm!tokenid=secret (injected into the provider's env by tf.sh; never a tofu variable, so it never lands in state).
• PROXMOX_VE_ENDPOINT / PROXMOX_VE_INSECURE — already set to https://192.168.2.102:8006/ / true.
• AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY — the R2 Object-Read/Write token from step 1.
• TF_STATE_PASSPHRASE — the state-at-rest passphrase (≥ 16 chars). tf.sh builds the multi-line TF_ENCRYPTION blob from this one scalar.

⚠️ The passphrase and R2 keys MUST be the SAME values that already encrypt the live R2 state. The state in R2 is already AES-GCM-encrypted with the current passphrase; a different TF_STATE_PASSPHRASE makes the existing state UNREADABLE (and wrong R2 keys can't reach it at all). Reuse your existing values from your password manager — do NOT invent new ones here.

The non-secret R2 bucket/key/endpoint live in the COMMITTED infra/tofu/backend.r2.hcl (an R2 account id is not a credential). Still keep the passphrase + R2 keys in your password manager as the authoritative backup. The old plaintext .r2.env / backend.hcl are retired — delete any local copies (they stay in .gitignore only as a safety net).

3. Migrate the existing local state -> R2 (one shot, moves state only — touches NO infra)

The R2 state has ALREADY been migrated and encrypted. This step is historical — a fresh machine adopting the existing encrypted state does NOT run it; it just does the Day-2 flow below.

cp infra/tofu/terraform.tfstate infra/tofu/terraform.tfstate.premigrate.bak   # belt-and-suspenders
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh init -migrate-state         # answer "yes"
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan                        # MUST say "No changes."

If plan shows no changes, the migration is correct. If it wants to create/destroy nodes, STOP — the state did not carry over; re-check backend.hcl/creds and do not apply.

Adopting encryption on pre-existing unencrypted state (one-time — already done on first migration). OpenTofu refuses to read a plaintext state while encryption is configured (encountered unencrypted payload without unencrypted method configured). For that ONE migration only, TF_ENCRYPTION must include a fallback so tofu can read plaintext and rewrite it encrypted: add method "unencrypted" "migrate" {} and give the state block a fallback { method = method.unencrypted.migrate }. Run the migrate, then remove both and tofu init -reconfigure so encryption is enforced. A fresh machine adopting the ALREADY-encrypted R2 state does NOT need this — the encrypt-only .r2.env.example is correct as-is.

Day-2 use (any machine)

Everything goes through the tf.sh wrapper inside the dev container. It decrypts secrets.sops.env, exports PROXMOX_VE_* / AWS_*, builds TF_ENCRYPTION from TF_STATE_PASSPHRASE, and passes backend.r2.hcl to init. NO manual secret entry, no plaintext env file:

./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh init
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan     # expect "No changes."
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh apply

To CHANGE a secret, edit the encrypted file in place (re-encrypts on save):

./infra/devcontainer/dev.sh run sops infra/tofu/secrets.sops.env

That's the whole "work from a different computer" story: clone the repo, have the age PRIVATE key in ~/.config/sops/age/keys.txt (the dev container mounts it), then tf.sh init / plan. The secrets ride along in git as ciphertext; only the age key lives outside the repo.

DR: rebuild from nothing

Because state is in R2 (not the cluster) AND the tofu secrets are committed (encrypted), a destroyed cluster is recoverable from just the repo + the age private key. On a fresh box: install the dev container toolchain, drop the age key at ~/.config/sops/age/keys.txt, then ./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh init and ... tf.sh apply to rebuild the VMs against the preserved state. Two things must survive off-cluster and off-repo: the age private key (in flux-system/sops-age in a live cluster; also keep an offline copy) and the R2 state bucket itself (covered by the off-cluster backup target OQ-BKP so state survives an R2/account loss). The TF_STATE_PASSPHRASE and R2 keys are recoverable from the encrypted secrets.sops.env as long as you hold the age key — keep them in your password manager too, belt and suspenders.

Follow-up (tracked)

• Locking. OpenTofu 1.9.0 has no use_lockfile; state locking is currently OFF (safe for a single operator). After bumping the toolchain to OpenTofu ≥ 1.10 (infra/devcontainer/versions.env
  • versions.tf required_version), set use_lockfile = true in backend.tf for native R2 conditional-write locking (no DynamoDB) — do this before regular two-machine use.