From 485b9e388881fbbdc3eff342cec07de79508842b Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Thu, 27 Aug 2026 15:57:12 -0500 Subject: [PATCH] infra(tofu): scaffold off-cluster R2 remote state (encrypted) Local-only tfstate meant losing the workstation would strand a rebuild (apply would try to recreate the 3 live nodes). Move state to Cloudflare R2, which sits OUTSIDE the Proxmox->VM->cluster->Gitea chain, so a full cluster rebuild can still read it; encrypted at rest via OpenTofu native AES-GCM. Adds a partial s3 backend (R2 skip-flags), gitignored backend.hcl/.r2.env for the account-specific + secret bits, their .example templates, and a migration+DR runbook. Locking (use_lockfile) deferred to an OpenTofu >=1.10 bump (1.9.0 lacks it). Migration (tofu init -migrate-state) runs once the operator provisions the R2 bucket. Co-Authored-By: Claude Opus 4.8 --- .gitignore | 5 +++ infra/tofu/.r2.env.example | 24 +++++++++++++ infra/tofu/REMOTE-STATE.md | 61 ++++++++++++++++++++++++++++++++++ infra/tofu/backend.hcl.example | 12 +++++++ infra/tofu/backend.tf | 37 +++++++++++++++++++++ 5 files changed, 139 insertions(+) create mode 100644 infra/tofu/.r2.env.example create mode 100644 infra/tofu/REMOTE-STATE.md create mode 100644 infra/tofu/backend.hcl.example create mode 100644 infra/tofu/backend.tf diff --git a/.gitignore b/.gitignore index a787a1a..ca7a974 100644 --- a/.gitignore +++ b/.gitignore @@ -29,3 +29,8 @@ plan.log /reports/ *.junit.xml coverage/ + +# Tofu remote-state secrets (R2 creds + encryption passphrase + account-specific backend cfg). +# The *.example templates ARE committed; the real files are not. +infra/tofu/backend.hcl +infra/tofu/.r2.env diff --git a/infra/tofu/.r2.env.example b/infra/tofu/.r2.env.example new file mode 100644 index 0000000..0de1166 --- /dev/null +++ b/infra/tofu/.r2.env.example @@ -0,0 +1,24 @@ +# Copy to `.r2.env` (gitignored), fill in, then: source .r2.env +# These are the ONLY secrets. They never enter git or tofu state. + +# --- R2 S3 credentials (from Cloudflare R2 -> Manage API Tokens -> Object Read & Write) --- +export AWS_ACCESS_KEY_ID="" +export AWS_SECRET_ACCESS_KEY="" + +# --- State-at-rest encryption (OpenTofu native, AES-GCM via PBKDF2 passphrase) --- +# CRITICAL: store this passphrase in your PASSWORD MANAGER. It is required to read the state on any +# machine and after any rebuild. It must be >= 16 characters (OpenTofu enforces a minimum). +export TF_ENCRYPTION=' +key_provider "pbkdf2" "state" { + passphrase = "" +} +method "aes_gcm" "state" { + keys = key_provider.pbkdf2.state +} +state { + method = method.aes_gcm.state +} +plan { + method = method.aes_gcm.state +} +' diff --git a/infra/tofu/REMOTE-STATE.md b/infra/tofu/REMOTE-STATE.md new file mode 100644 index 0000000..40905ac --- /dev/null +++ b/infra/tofu/REMOTE-STATE.md @@ -0,0 +1,61 @@ +# Tofu remote state — Cloudflare R2 (off-cluster, encrypted) + +## Why +State was **local** (`terraform.tfstate` on one workstation, git-ignored). Losing that box meant a +`tofu apply` from anywhere else would try to **recreate the three live k3s nodes**. State must live +**off the cluster it manages** — the chain is `Proxmox host → k3s VMs → k3s cluster → Gitea/apps`, so +an in-cluster backend (Gitea) is unreadable in the one scenario it exists for (cluster gone, rebuild). + +**Cloudflare R2** sits outside that chain entirely: a full cluster rebuild can still read state, it +survives total-site loss (off-site), and it works from any machine. State bytes are **encrypted at +rest** (OpenTofu native AES-GCM) so R2 only ever holds ciphertext. + +## One-time setup (operator) + +### 1. Create the R2 bucket + token (Cloudflare dashboard) +1. **R2 → Create bucket** → name `homelab-tofu-state` (any name; match `backend.hcl`). +2. **R2 → Manage API Tokens → Create API Token** → permission **Object Read & Write**, scoped to + that bucket. Copy the **Access Key ID** and **Secret Access Key** (shown once). +3. Note your **Account ID** (32 hex chars, in the dashboard URL / the R2 S3 endpoint). + +### 2. Fill the two gitignored files (secrets stay out of git and out of chat) +```bash +cd infra/tofu +cp backend.hcl.example backend.hcl # set bucket/key + endpoints.s3 = https://.r2.cloudflarestorage.com +cp .r2.env.example .r2.env # set AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY + a strong TF_ENCRYPTION passphrase +``` +**Store the encryption passphrase in your password manager.** It is required to read state on any +machine and after any rebuild. Also record the R2 keys there. (Both files are in `.gitignore`.) + +### 3. Migrate the existing local state -> R2 (one shot, moves state only — touches NO infra) +```bash +cd infra/tofu +cp terraform.tfstate terraform.tfstate.premigrate.bak # extra belt-and-suspenders backup +source .r2.env +tofu init -backend-config=backend.hcl -migrate-state # answer "yes" to copy local -> R2 +tofu plan # MUST report "No changes." — proves parity +``` +If `plan` shows no changes, the migration is correct. If it wants to create/destroy nodes, STOP — +the state did not carry over; re-check backend.hcl/creds and do not apply. + +## Day-2 use (any machine) +```bash +cd infra/tofu +source .r2.env # R2 creds + TF_ENCRYPTION passphrase +tofu init -backend-config=backend.hcl +tofu plan +``` +That's the whole "work from a different computer" story: clone the repo, create the two local files +from the templates (creds + passphrase from your password manager), `init`, done. + +## DR: rebuild from nothing +Because state is in R2 (not the cluster), a destroyed cluster is recoverable: on a fresh box, install +OpenTofu, create `backend.hcl` + `.r2.env`, `tofu init -backend-config=backend.hcl`, then `tofu apply` +to rebuild the VMs against the preserved state. R2 itself should ALSO be covered by the off-cluster +backup target (OQ-BKP) so state survives even an R2/account loss. + +## Follow-up (tracked) +- **Locking.** OpenTofu 1.9.0 has no `use_lockfile`; state locking is currently OFF (safe for a single + operator). After bumping the toolchain to **OpenTofu ≥ 1.10** (`infra/devcontainer/versions.env` + + `versions.tf` `required_version`), set `use_lockfile = true` in `backend.tf` for native R2 + conditional-write locking (no DynamoDB) — do this before regular two-machine use. diff --git a/infra/tofu/backend.hcl.example b/infra/tofu/backend.hcl.example new file mode 100644 index 0000000..58a59bf --- /dev/null +++ b/infra/tofu/backend.hcl.example @@ -0,0 +1,12 @@ +# Copy to `backend.hcl` (gitignored) and fill in from your Cloudflare R2 bucket + API token. +# Passed to tofu via: source .r2.env && tofu init -backend-config=backend.hcl -migrate-state +# +# The R2 is the 32-hex string in your Cloudflare dashboard URL / R2 endpoint. It is +# not a credential, but is kept out of git here as defense-in-depth. + +bucket = "homelab-tofu-state" +key = "k3s-dev/terraform.tfstate" + +endpoints = { + s3 = "https://.r2.cloudflarestorage.com" +} diff --git a/infra/tofu/backend.tf b/infra/tofu/backend.tf new file mode 100644 index 0000000..c143648 --- /dev/null +++ b/infra/tofu/backend.tf @@ -0,0 +1,37 @@ +--- +# Remote state — Cloudflare R2 (S3-compatible), stored OFF the k3s cluster. +# +# WHY OFF-CLUSTER (the load-bearing decision): tofu state must live at or below the layer it +# manages. Its dependency chain is Proxmox host -> k3s VMs -> k3s cluster -> Gitea/apps, so state +# in Gitea (top of the chain) would be UNREADABLE in exactly the disaster it exists for — cluster +# gone, `tofu apply` to rebuild. R2 sits entirely outside that chain: a full cluster rebuild can +# still read state. It also survives total-site loss (off-site) and gives multi-machine access. +# +# SECRETS: the account-specific bucket/endpoint come from a gitignored backend.hcl; the S3 keys +# come from the environment (.r2.env: AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY). State bytes are +# encrypted at rest via the TF_ENCRYPTION env blob (see REMOTE-STATE.md) — the passphrase lives ONLY +# in your password manager and the gitignored .r2.env, never in git or in the state itself. +# +# INIT (partial config — non-secret bits here, secret bits from the gitignored file): +# cd infra/tofu && source .r2.env && tofu init -backend-config=backend.hcl -migrate-state +# +# NOTE: this `terraform {}` block is MERGED with the one in versions.tf (required_version + +# required_providers). OpenTofu allows multiple terraform blocks across files; `backend` appears once. +terraform { + backend "s3" { + region = "auto" # R2 ignores region, but the S3 backend requires a value. + + # R2 is S3-compatible but not AWS: disable the AWS-only preflight/validation paths. + skip_credentials_validation = true + skip_region_validation = true + skip_requesting_account_id = true + skip_metadata_api_check = true + skip_s3_checksum = true # R2 rejects AWS trailing (streaming) checksums. + + # bucket, key, endpoints{ s3 } -> backend.hcl (gitignored; keeps the R2 account id out of git). + # + # use_lockfile = true -> ENABLE after bumping OpenTofu to >= 1.10. Native S3 conditional-write + # locking (no DynamoDB); works on R2 via If-None-Match. 1.9.0 has no lockfile support, so state + # locking is OFF for now (single-operator workflow — see REMOTE-STATE.md follow-up). + } +}