# Tofu remote state — Cloudflare R2 (off-cluster, encrypted)

## Why
State was **local** (`terraform.tfstate` on one workstation, git-ignored). Losing that box meant a
`tofu apply` from anywhere else would try to **recreate the three live k3s nodes**. State must live
**off the cluster it manages** — the chain is `Proxmox host → k3s VMs → k3s cluster → Gitea/apps`, so
an in-cluster backend (Gitea) is unreadable in the one scenario it exists for (cluster gone, rebuild).

**Cloudflare R2** sits outside that chain entirely: a full cluster rebuild can still read state, it
survives total-site loss (off-site), and it works from any machine. State bytes are **encrypted at
rest** (OpenTofu native AES-GCM) so R2 only ever holds ciphertext.

## One-time setup (operator)

### 1. Create the R2 bucket + token (Cloudflare dashboard)
1. **R2 → Create bucket** → name `homelab-tofu-state` (any name; match `backend.hcl`).
2. **R2 → Manage API Tokens → Create API Token** → permission **Object Read & Write**, scoped to
   that bucket. Copy the **Access Key ID** and **Secret Access Key** (shown once).
3. Note your **Account ID** (32 hex chars, in the dashboard URL / the R2 S3 endpoint).

### 2. Fill in the encrypted secrets file (edited in place; ciphertext is committed)
All five secrets live in ONE committed, SOPS/age-encrypted dotenv, `infra/tofu/secrets.sops.env`.
Edit it inside the dev container (it has `sops` + the age key). `sops` decrypts to your `$EDITOR`
and re-encrypts on save — plaintext never touches disk:
```bash
./infra/devcontainer/dev.sh run sops infra/tofu/secrets.sops.env
```
Replace every `REPLACE-ME`:
- `PROXMOX_VE_API_TOKEN` — `user@realm!tokenid=secret` (injected into the provider's env by tf.sh;
  never a tofu variable, so it never lands in state).
- `PROXMOX_VE_ENDPOINT` / `PROXMOX_VE_INSECURE` — already set to `https://192.168.2.102:8006/` / `true`.
- `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` — the R2 Object-Read/Write token from step 1.
- `TF_STATE_PASSPHRASE` — the state-at-rest passphrase (≥ 16 chars). **tf.sh builds the multi-line
  `TF_ENCRYPTION` blob from this one scalar.**

> ⚠️ **The passphrase and R2 keys MUST be the SAME values that already encrypt the live R2 state.**
> The state in R2 is already AES-GCM-encrypted with the current passphrase; a *different*
> `TF_STATE_PASSPHRASE` makes the existing state UNREADABLE (and wrong R2 keys can't reach it at
> all). Reuse your existing values from your password manager — do NOT invent new ones here.

The non-secret R2 bucket/key/endpoint live in the COMMITTED `infra/tofu/backend.r2.hcl` (an R2
account id is not a credential). Still keep the passphrase + R2 keys in your password manager as
the authoritative backup. The old plaintext `.r2.env` / `backend.hcl` are retired — delete any
local copies (they stay in `.gitignore` only as a safety net).

### 3. Migrate the existing local state -> R2 (one shot, moves state only — touches NO infra)
> The R2 state has ALREADY been migrated and encrypted. This step is historical — a fresh machine
> adopting the existing encrypted state does NOT run it; it just does the Day-2 flow below.
```bash
cp infra/tofu/terraform.tfstate infra/tofu/terraform.tfstate.premigrate.bak   # belt-and-suspenders
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh init -migrate-state         # answer "yes"
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan                        # MUST say "No changes."
```
If `plan` shows no changes, the migration is correct. If it wants to create/destroy nodes, STOP —
the state did not carry over; re-check backend.hcl/creds and do not apply.

> **Adopting encryption on pre-existing *unencrypted* state (one-time — already done on first
> migration).** OpenTofu refuses to read a plaintext state while encryption is configured
> (`encountered unencrypted payload without unencrypted method configured`). For that ONE migration
> only, `TF_ENCRYPTION` must include a fallback so tofu can read plaintext and rewrite it encrypted:
> add `method "unencrypted" "migrate" {}` and give the `state` block a
> `fallback { method = method.unencrypted.migrate }`. Run the migrate, then **remove both** and
> `tofu init -reconfigure` so encryption is enforced. A fresh machine adopting the ALREADY-encrypted
> R2 state does NOT need this — the encrypt-only `.r2.env.example` is correct as-is.

## Day-2 use (any machine)
Everything goes through the `tf.sh` wrapper inside the dev container. It decrypts
`secrets.sops.env`, exports `PROXMOX_VE_*` / `AWS_*`, builds `TF_ENCRYPTION` from
`TF_STATE_PASSPHRASE`, and passes `backend.r2.hcl` to `init`. NO manual secret entry, no plaintext
env file:
```bash
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh init
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan     # expect "No changes."
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh apply
```
To CHANGE a secret, edit the encrypted file in place (re-encrypts on save):
```bash
./infra/devcontainer/dev.sh run sops infra/tofu/secrets.sops.env
```

That's the whole "work from a different computer" story: clone the repo, have the age PRIVATE key in
`~/.config/sops/age/keys.txt` (the dev container mounts it), then `tf.sh init` / `plan`. The secrets
ride along in git as ciphertext; only the age key lives outside the repo.

## DR: rebuild from nothing
Because state is in R2 (not the cluster) AND the tofu secrets are committed (encrypted), a destroyed
cluster is recoverable from just **the repo + the age private key**. On a fresh box: install the dev
container toolchain, drop the age key at `~/.config/sops/age/keys.txt`, then
`./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh init` and `... tf.sh apply` to rebuild the VMs
against the preserved state. Two things must survive off-cluster and off-repo: the **age private
key** (in `flux-system/sops-age` in a live cluster; also keep an offline copy) and the **R2 state
bucket** itself (covered by the off-cluster backup target OQ-BKP so state survives an R2/account loss).
The `TF_STATE_PASSPHRASE` and R2 keys are recoverable from the encrypted `secrets.sops.env` as long
as you hold the age key — keep them in your password manager too, belt and suspenders.

## Follow-up (tracked)
- **Locking.** OpenTofu 1.9.0 has no `use_lockfile`; state locking is currently OFF (safe for a single
  operator). After bumping the toolchain to **OpenTofu ≥ 1.10** (`infra/devcontainer/versions.env`
  + `versions.tf` `required_version`), set `use_lockfile = true` in `backend.tf` for native R2
  conditional-write locking (no DynamoDB) — do this before regular two-machine use.