jakman85andClaude Opus 4.8
216bea2a16
Replace the gitignored plaintext infra/tofu/.r2.env and backend.hcl with a single committed, SOPS/age-encrypted dotenv (secrets.sops.env) plus a thin wrapper (tf.sh), so tofu runs with no manual secret entry and secrets live encrypted in git like the repo's other secrets. - .sops.yaml: creation rule encrypting infra/tofu/secrets.sops.env (all values) to the existing age recipient. - secrets.sops.env: encrypted, PLACEHOLDER values only. Holds R2 S3 keys, the state-encryption passphrase (TF_STATE_PASSPHRASE scalar), and the Proxmox API token/endpoint. Operator fills real values via `sops`. - tf.sh: sops exec-env decrypts the dotenv, exports PROXMOX_VE_*/AWS_*, builds the multi-line TF_ENCRYPTION HCL from TF_STATE_PASSPHRASE, and runs tofu (injecting -backend-config=backend.r2.hcl on init). Proxmox token stays env-only, never a tofu variable/state. - backend.r2.hcl: committed non-secret R2 bucket/key/endpoint (account id is not a credential), passed to `tofu init`. - Retire .r2.env/backend.hcl(.example): drop the committed templates; keep the gitignore lines as a safety net for leftover local plaintext. - Update backend.tf/README/REMOTE-STATE for the new day-2 flow and DR story (repo + age key is sufficient). Verified: placeholder file round-trips sops -e/-d with the repo age key; tf.sh exports the expected VAR NAMES and builds valid TF_ENCRYPTION HCL (stub tofu, no real plan/secrets). Co-Authored-By: Claude Opus 4.8 <[email protected]>
101 lines
6.5 KiB
Markdown
101 lines
6.5 KiB
Markdown
# Tofu remote state — Cloudflare R2 (off-cluster, encrypted)
|
|
|
|
## Why
|
|
State was **local** (`terraform.tfstate` on one workstation, git-ignored). Losing that box meant a
|
|
`tofu apply` from anywhere else would try to **recreate the three live k3s nodes**. State must live
|
|
**off the cluster it manages** — the chain is `Proxmox host → k3s VMs → k3s cluster → Gitea/apps`, so
|
|
an in-cluster backend (Gitea) is unreadable in the one scenario it exists for (cluster gone, rebuild).
|
|
|
|
**Cloudflare R2** sits outside that chain entirely: a full cluster rebuild can still read state, it
|
|
survives total-site loss (off-site), and it works from any machine. State bytes are **encrypted at
|
|
rest** (OpenTofu native AES-GCM) so R2 only ever holds ciphertext.
|
|
|
|
## One-time setup (operator)
|
|
|
|
### 1. Create the R2 bucket + token (Cloudflare dashboard)
|
|
1. **R2 → Create bucket** → name `homelab-tofu-state` (any name; match `backend.hcl`).
|
|
2. **R2 → Manage API Tokens → Create API Token** → permission **Object Read & Write**, scoped to
|
|
that bucket. Copy the **Access Key ID** and **Secret Access Key** (shown once).
|
|
3. Note your **Account ID** (32 hex chars, in the dashboard URL / the R2 S3 endpoint).
|
|
|
|
### 2. Fill in the encrypted secrets file (edited in place; ciphertext is committed)
|
|
All five secrets live in ONE committed, SOPS/age-encrypted dotenv, `infra/tofu/secrets.sops.env`.
|
|
Edit it inside the dev container (it has `sops` + the age key). `sops` decrypts to your `$EDITOR`
|
|
and re-encrypts on save — plaintext never touches disk:
|
|
```bash
|
|
./infra/devcontainer/dev.sh run sops infra/tofu/secrets.sops.env
|
|
```
|
|
Replace every `REPLACE-ME`:
|
|
- `PROXMOX_VE_API_TOKEN` — `user@realm!tokenid=secret` (injected into the provider's env by tf.sh;
|
|
never a tofu variable, so it never lands in state).
|
|
- `PROXMOX_VE_ENDPOINT` / `PROXMOX_VE_INSECURE` — already set to `https://192.168.2.102:8006/` / `true`.
|
|
- `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` — the R2 Object-Read/Write token from step 1.
|
|
- `TF_STATE_PASSPHRASE` — the state-at-rest passphrase (≥ 16 chars). **tf.sh builds the multi-line
|
|
`TF_ENCRYPTION` blob from this one scalar.**
|
|
|
|
> ⚠️ **The passphrase and R2 keys MUST be the SAME values that already encrypt the live R2 state.**
|
|
> The state in R2 is already AES-GCM-encrypted with the current passphrase; a *different*
|
|
> `TF_STATE_PASSPHRASE` makes the existing state UNREADABLE (and wrong R2 keys can't reach it at
|
|
> all). Reuse your existing values from your password manager — do NOT invent new ones here.
|
|
|
|
The non-secret R2 bucket/key/endpoint live in the COMMITTED `infra/tofu/backend.r2.hcl` (an R2
|
|
account id is not a credential). Still keep the passphrase + R2 keys in your password manager as
|
|
the authoritative backup. The old plaintext `.r2.env` / `backend.hcl` are retired — delete any
|
|
local copies (they stay in `.gitignore` only as a safety net).
|
|
|
|
### 3. Migrate the existing local state -> R2 (one shot, moves state only — touches NO infra)
|
|
> The R2 state has ALREADY been migrated and encrypted. This step is historical — a fresh machine
|
|
> adopting the existing encrypted state does NOT run it; it just does the Day-2 flow below.
|
|
```bash
|
|
cp infra/tofu/terraform.tfstate infra/tofu/terraform.tfstate.premigrate.bak # belt-and-suspenders
|
|
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh init -migrate-state # answer "yes"
|
|
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan # MUST say "No changes."
|
|
```
|
|
If `plan` shows no changes, the migration is correct. If it wants to create/destroy nodes, STOP —
|
|
the state did not carry over; re-check backend.hcl/creds and do not apply.
|
|
|
|
> **Adopting encryption on pre-existing *unencrypted* state (one-time — already done on first
|
|
> migration).** OpenTofu refuses to read a plaintext state while encryption is configured
|
|
> (`encountered unencrypted payload without unencrypted method configured`). For that ONE migration
|
|
> only, `TF_ENCRYPTION` must include a fallback so tofu can read plaintext and rewrite it encrypted:
|
|
> add `method "unencrypted" "migrate" {}` and give the `state` block a
|
|
> `fallback { method = method.unencrypted.migrate }`. Run the migrate, then **remove both** and
|
|
> `tofu init -reconfigure` so encryption is enforced. A fresh machine adopting the ALREADY-encrypted
|
|
> R2 state does NOT need this — the encrypt-only `.r2.env.example` is correct as-is.
|
|
|
|
## Day-2 use (any machine)
|
|
Everything goes through the `tf.sh` wrapper inside the dev container. It decrypts
|
|
`secrets.sops.env`, exports `PROXMOX_VE_*` / `AWS_*`, builds `TF_ENCRYPTION` from
|
|
`TF_STATE_PASSPHRASE`, and passes `backend.r2.hcl` to `init`. NO manual secret entry, no plaintext
|
|
env file:
|
|
```bash
|
|
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh init
|
|
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan # expect "No changes."
|
|
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh apply
|
|
```
|
|
To CHANGE a secret, edit the encrypted file in place (re-encrypts on save):
|
|
```bash
|
|
./infra/devcontainer/dev.sh run sops infra/tofu/secrets.sops.env
|
|
```
|
|
|
|
That's the whole "work from a different computer" story: clone the repo, have the age PRIVATE key in
|
|
`~/.config/sops/age/keys.txt` (the dev container mounts it), then `tf.sh init` / `plan`. The secrets
|
|
ride along in git as ciphertext; only the age key lives outside the repo.
|
|
|
|
## DR: rebuild from nothing
|
|
Because state is in R2 (not the cluster) AND the tofu secrets are committed (encrypted), a destroyed
|
|
cluster is recoverable from just **the repo + the age private key**. On a fresh box: install the dev
|
|
container toolchain, drop the age key at `~/.config/sops/age/keys.txt`, then
|
|
`./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh init` and `... tf.sh apply` to rebuild the VMs
|
|
against the preserved state. Two things must survive off-cluster and off-repo: the **age private
|
|
key** (in `flux-system/sops-age` in a live cluster; also keep an offline copy) and the **R2 state
|
|
bucket** itself (covered by the off-cluster backup target OQ-BKP so state survives an R2/account loss).
|
|
The `TF_STATE_PASSPHRASE` and R2 keys are recoverable from the encrypted `secrets.sops.env` as long
|
|
as you hold the age key — keep them in your password manager too, belt and suspenders.
|
|
|
|
## Follow-up (tracked)
|
|
- **Locking.** OpenTofu 1.9.0 has no `use_lockfile`; state locking is currently OFF (safe for a single
|
|
operator). After bumping the toolchain to **OpenTofu ≥ 1.10** (`infra/devcontainer/versions.env`
|
|
+ `versions.tf` `required_version`), set `use_lockfile = true` in `backend.tf` for native R2
|
|
conditional-write locking (no DynamoDB) — do this before regular two-machine use.
|