infra(tofu): scaffold off-cluster R2 remote state (encrypted)
Local-only tfstate meant losing the workstation would strand a rebuild (apply would try to recreate the 3 live nodes). Move state to Cloudflare R2, which sits OUTSIDE the Proxmox->VM->cluster->Gitea chain, so a full cluster rebuild can still read it; encrypted at rest via OpenTofu native AES-GCM. Adds a partial s3 backend (R2 skip-flags), gitignored backend.hcl/.r2.env for the account-specific + secret bits, their .example templates, and a migration+DR runbook. Locking (use_lockfile) deferred to an OpenTofu >=1.10 bump (1.9.0 lacks it). Migration (tofu init -migrate-state) runs once the operator provisions the R2 bucket. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed
1 parent
91c82d2756
commit
485b9e3888
5 files changed
+139
No files matched your search
@@ -29,3 +29,8 @@ plan.log
|
||||
/reports/
|
||||
*.junit.xml
|
||||
coverage/
|
||||
|
||||
# Tofu remote-state secrets (R2 creds + encryption passphrase + account-specific backend cfg).
|
||||
# The *.example templates ARE committed; the real files are not.
|
||||
infra/tofu/backend.hcl
|
||||
infra/tofu/.r2.env
|
||||
@@ -0,0 +1,24 @@
|
||||
# Copy to `.r2.env` (gitignored), fill in, then: source .r2.env
|
||||
# These are the ONLY secrets. They never enter git or tofu state.
|
||||
|
||||
# --- R2 S3 credentials (from Cloudflare R2 -> Manage API Tokens -> Object Read & Write) ---
|
||||
export AWS_ACCESS_KEY_ID="<R2_ACCESS_KEY_ID>"
|
||||
export AWS_SECRET_ACCESS_KEY="<R2_SECRET_ACCESS_KEY>"
|
||||
|
||||
# --- State-at-rest encryption (OpenTofu native, AES-GCM via PBKDF2 passphrase) ---
|
||||
# CRITICAL: store this passphrase in your PASSWORD MANAGER. It is required to read the state on any
|
||||
# machine and after any rebuild. It must be >= 16 characters (OpenTofu enforces a minimum).
|
||||
export TF_ENCRYPTION='
|
||||
key_provider "pbkdf2" "state" {
|
||||
passphrase = "<STRONG_PASSPHRASE_MIN_16_CHARS>"
|
||||
}
|
||||
method "aes_gcm" "state" {
|
||||
keys = key_provider.pbkdf2.state
|
||||
}
|
||||
state {
|
||||
method = method.aes_gcm.state
|
||||
}
|
||||
plan {
|
||||
method = method.aes_gcm.state
|
||||
}
|
||||
'
|
||||
@@ -0,0 +1,61 @@
|
||||
# Tofu remote state — Cloudflare R2 (off-cluster, encrypted)
|
||||
|
||||
## Why
|
||||
State was **local** (`terraform.tfstate` on one workstation, git-ignored). Losing that box meant a
|
||||
`tofu apply` from anywhere else would try to **recreate the three live k3s nodes**. State must live
|
||||
**off the cluster it manages** — the chain is `Proxmox host → k3s VMs → k3s cluster → Gitea/apps`, so
|
||||
an in-cluster backend (Gitea) is unreadable in the one scenario it exists for (cluster gone, rebuild).
|
||||
|
||||
**Cloudflare R2** sits outside that chain entirely: a full cluster rebuild can still read state, it
|
||||
survives total-site loss (off-site), and it works from any machine. State bytes are **encrypted at
|
||||
rest** (OpenTofu native AES-GCM) so R2 only ever holds ciphertext.
|
||||
|
||||
## One-time setup (operator)
|
||||
|
||||
### 1. Create the R2 bucket + token (Cloudflare dashboard)
|
||||
1. **R2 → Create bucket** → name `homelab-tofu-state` (any name; match `backend.hcl`).
|
||||
2. **R2 → Manage API Tokens → Create API Token** → permission **Object Read & Write**, scoped to
|
||||
that bucket. Copy the **Access Key ID** and **Secret Access Key** (shown once).
|
||||
3. Note your **Account ID** (32 hex chars, in the dashboard URL / the R2 S3 endpoint).
|
||||
|
||||
### 2. Fill the two gitignored files (secrets stay out of git and out of chat)
|
||||
```bash
|
||||
cd infra/tofu
|
||||
cp backend.hcl.example backend.hcl # set bucket/key + endpoints.s3 = https://<ACCOUNT_ID>.r2.cloudflarestorage.com
|
||||
cp .r2.env.example .r2.env # set AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY + a strong TF_ENCRYPTION passphrase
|
||||
```
|
||||
**Store the encryption passphrase in your password manager.** It is required to read state on any
|
||||
machine and after any rebuild. Also record the R2 keys there. (Both files are in `.gitignore`.)
|
||||
|
||||
### 3. Migrate the existing local state -> R2 (one shot, moves state only — touches NO infra)
|
||||
```bash
|
||||
cd infra/tofu
|
||||
cp terraform.tfstate terraform.tfstate.premigrate.bak # extra belt-and-suspenders backup
|
||||
source .r2.env
|
||||
tofu init -backend-config=backend.hcl -migrate-state # answer "yes" to copy local -> R2
|
||||
tofu plan # MUST report "No changes." — proves parity
|
||||
```
|
||||
If `plan` shows no changes, the migration is correct. If it wants to create/destroy nodes, STOP —
|
||||
the state did not carry over; re-check backend.hcl/creds and do not apply.
|
||||
|
||||
## Day-2 use (any machine)
|
||||
```bash
|
||||
cd infra/tofu
|
||||
source .r2.env # R2 creds + TF_ENCRYPTION passphrase
|
||||
tofu init -backend-config=backend.hcl
|
||||
tofu plan
|
||||
```
|
||||
That's the whole "work from a different computer" story: clone the repo, create the two local files
|
||||
from the templates (creds + passphrase from your password manager), `init`, done.
|
||||
|
||||
## DR: rebuild from nothing
|
||||
Because state is in R2 (not the cluster), a destroyed cluster is recoverable: on a fresh box, install
|
||||
OpenTofu, create `backend.hcl` + `.r2.env`, `tofu init -backend-config=backend.hcl`, then `tofu apply`
|
||||
to rebuild the VMs against the preserved state. R2 itself should ALSO be covered by the off-cluster
|
||||
backup target (OQ-BKP) so state survives even an R2/account loss.
|
||||
|
||||
## Follow-up (tracked)
|
||||
- **Locking.** OpenTofu 1.9.0 has no `use_lockfile`; state locking is currently OFF (safe for a single
|
||||
operator). After bumping the toolchain to **OpenTofu ≥ 1.10** (`infra/devcontainer/versions.env`
|
||||
+ `versions.tf` `required_version`), set `use_lockfile = true` in `backend.tf` for native R2
|
||||
conditional-write locking (no DynamoDB) — do this before regular two-machine use.
|
||||
@@ -0,0 +1,12 @@
|
||||
# Copy to `backend.hcl` (gitignored) and fill in from your Cloudflare R2 bucket + API token.
|
||||
# Passed to tofu via: source .r2.env && tofu init -backend-config=backend.hcl -migrate-state
|
||||
#
|
||||
# The R2 <ACCOUNT_ID> is the 32-hex string in your Cloudflare dashboard URL / R2 endpoint. It is
|
||||
# not a credential, but is kept out of git here as defense-in-depth.
|
||||
|
||||
bucket = "homelab-tofu-state"
|
||||
key = "k3s-dev/terraform.tfstate"
|
||||
|
||||
endpoints = {
|
||||
s3 = "https://<ACCOUNT_ID>.r2.cloudflarestorage.com"
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
---
|
||||
# Remote state — Cloudflare R2 (S3-compatible), stored OFF the k3s cluster.
|
||||
#
|
||||
# WHY OFF-CLUSTER (the load-bearing decision): tofu state must live at or below the layer it
|
||||
# manages. Its dependency chain is Proxmox host -> k3s VMs -> k3s cluster -> Gitea/apps, so state
|
||||
# in Gitea (top of the chain) would be UNREADABLE in exactly the disaster it exists for — cluster
|
||||
# gone, `tofu apply` to rebuild. R2 sits entirely outside that chain: a full cluster rebuild can
|
||||
# still read state. It also survives total-site loss (off-site) and gives multi-machine access.
|
||||
#
|
||||
# SECRETS: the account-specific bucket/endpoint come from a gitignored backend.hcl; the S3 keys
|
||||
# come from the environment (.r2.env: AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY). State bytes are
|
||||
# encrypted at rest via the TF_ENCRYPTION env blob (see REMOTE-STATE.md) — the passphrase lives ONLY
|
||||
# in your password manager and the gitignored .r2.env, never in git or in the state itself.
|
||||
#
|
||||
# INIT (partial config — non-secret bits here, secret bits from the gitignored file):
|
||||
# cd infra/tofu && source .r2.env && tofu init -backend-config=backend.hcl -migrate-state
|
||||
#
|
||||
# NOTE: this `terraform {}` block is MERGED with the one in versions.tf (required_version +
|
||||
# required_providers). OpenTofu allows multiple terraform blocks across files; `backend` appears once.
|
||||
terraform {
|
||||
backend "s3" {
|
||||
region = "auto" # R2 ignores region, but the S3 backend requires a value.
|
||||
|
||||
# R2 is S3-compatible but not AWS: disable the AWS-only preflight/validation paths.
|
||||
skip_credentials_validation = true
|
||||
skip_region_validation = true
|
||||
skip_requesting_account_id = true
|
||||
skip_metadata_api_check = true
|
||||
skip_s3_checksum = true # R2 rejects AWS trailing (streaming) checksums.
|
||||
|
||||
# bucket, key, endpoints{ s3 } -> backend.hcl (gitignored; keeps the R2 account id out of git).
|
||||
#
|
||||
# use_lockfile = true -> ENABLE after bumping OpenTofu to >= 1.10. Native S3 conditional-write
|
||||
# locking (no DynamoDB); works on R2 via If-None-Match. 1.9.0 has no lockfile support, so state
|
||||
# locking is OFF for now (single-operator workflow — see REMOTE-STATE.md follow-up).
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user