Merge feature/tofu-remote-state-r2: off-cluster R2 encrypted tofu state scaffolding

.gitignore

@@ -29,3 +29,8 @@ plan.log
 /reports/
 *.junit.xml
 coverage/
+
+# Tofu remote-state secrets (R2 creds + encryption passphrase + account-specific backend cfg).
+# The *.example templates ARE committed; the real files are not.
+infra/tofu/backend.hcl
+infra/tofu/.r2.env

infra/tofu/.r2.env.example

@@ -0,0 +1,24 @@
+# Copy to `.r2.env` (gitignored), fill in, then:  source .r2.env
+# These are the ONLY secrets. They never enter git or tofu state.
+
+# --- R2 S3 credentials (from Cloudflare R2 -> Manage API Tokens -> Object Read & Write) ---
+export AWS_ACCESS_KEY_ID="<R2_ACCESS_KEY_ID>"
+export AWS_SECRET_ACCESS_KEY="<R2_SECRET_ACCESS_KEY>"
+
+# --- State-at-rest encryption (OpenTofu native, AES-GCM via PBKDF2 passphrase) ---
+# CRITICAL: store this passphrase in your PASSWORD MANAGER. It is required to read the state on any
+# machine and after any rebuild. It must be >= 16 characters (OpenTofu enforces a minimum).
+export TF_ENCRYPTION='
+key_provider "pbkdf2" "state" {
+  passphrase = "<STRONG_PASSPHRASE_MIN_16_CHARS>"
+}
+method "aes_gcm" "state" {
+  keys = key_provider.pbkdf2.state
+}
+state {
+  method = method.aes_gcm.state
+}
+plan {
+  method = method.aes_gcm.state
+}
+'

infra/tofu/REMOTE-STATE.md

@@ -0,0 +1,61 @@
+# Tofu remote state — Cloudflare R2 (off-cluster, encrypted)
+
+## Why
+State was **local** (`terraform.tfstate` on one workstation, git-ignored). Losing that box meant a
+`tofu apply` from anywhere else would try to **recreate the three live k3s nodes**. State must live
+**off the cluster it manages** — the chain is `Proxmox host → k3s VMs → k3s cluster → Gitea/apps`, so
+an in-cluster backend (Gitea) is unreadable in the one scenario it exists for (cluster gone, rebuild).
+
+**Cloudflare R2** sits outside that chain entirely: a full cluster rebuild can still read state, it
+survives total-site loss (off-site), and it works from any machine. State bytes are **encrypted at
+rest** (OpenTofu native AES-GCM) so R2 only ever holds ciphertext.
+
+## One-time setup (operator)
+
+### 1. Create the R2 bucket + token (Cloudflare dashboard)
+1. **R2 → Create bucket** → name `homelab-tofu-state` (any name; match `backend.hcl`).
+2. **R2 → Manage API Tokens → Create API Token** → permission **Object Read & Write**, scoped to
+   that bucket. Copy the **Access Key ID** and **Secret Access Key** (shown once).
+3. Note your **Account ID** (32 hex chars, in the dashboard URL / the R2 S3 endpoint).
+
+### 2. Fill the two gitignored files (secrets stay out of git and out of chat)
+```bash
+cd infra/tofu
+cp backend.hcl.example backend.hcl     # set bucket/key + endpoints.s3 = https://<ACCOUNT_ID>.r2.cloudflarestorage.com
+cp .r2.env.example     .r2.env         # set AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY + a strong TF_ENCRYPTION passphrase
+```
+**Store the encryption passphrase in your password manager.** It is required to read state on any
+machine and after any rebuild. Also record the R2 keys there. (Both files are in `.gitignore`.)
+
+### 3. Migrate the existing local state -> R2 (one shot, moves state only — touches NO infra)
+```bash
+cd infra/tofu
+cp terraform.tfstate terraform.tfstate.premigrate.bak   # extra belt-and-suspenders backup
+source .r2.env
+tofu init -backend-config=backend.hcl -migrate-state     # answer "yes" to copy local -> R2
+tofu plan                                                # MUST report "No changes." — proves parity
+```
+If `plan` shows no changes, the migration is correct. If it wants to create/destroy nodes, STOP —
+the state did not carry over; re-check backend.hcl/creds and do not apply.
+
+## Day-2 use (any machine)
+```bash
+cd infra/tofu
+source .r2.env                 # R2 creds + TF_ENCRYPTION passphrase
+tofu init -backend-config=backend.hcl
+tofu plan
+```
+That's the whole "work from a different computer" story: clone the repo, create the two local files
+from the templates (creds + passphrase from your password manager), `init`, done.
+
+## DR: rebuild from nothing
+Because state is in R2 (not the cluster), a destroyed cluster is recoverable: on a fresh box, install
+OpenTofu, create `backend.hcl` + `.r2.env`, `tofu init -backend-config=backend.hcl`, then `tofu apply`
+to rebuild the VMs against the preserved state. R2 itself should ALSO be covered by the off-cluster
+backup target (OQ-BKP) so state survives even an R2/account loss.
+
+## Follow-up (tracked)
+- **Locking.** OpenTofu 1.9.0 has no `use_lockfile`; state locking is currently OFF (safe for a single
+  operator). After bumping the toolchain to **OpenTofu ≥ 1.10** (`infra/devcontainer/versions.env`
+  + `versions.tf` `required_version`), set `use_lockfile = true` in `backend.tf` for native R2
+  conditional-write locking (no DynamoDB) — do this before regular two-machine use.

infra/tofu/backend.hcl.example

@@ -0,0 +1,12 @@
+# Copy to `backend.hcl` (gitignored) and fill in from your Cloudflare R2 bucket + API token.
+# Passed to tofu via:  source .r2.env && tofu init -backend-config=backend.hcl -migrate-state
+#
+# The R2 <ACCOUNT_ID> is the 32-hex string in your Cloudflare dashboard URL / R2 endpoint. It is
+# not a credential, but is kept out of git here as defense-in-depth.
+
+bucket = "homelab-tofu-state"
+key    = "k3s-dev/terraform.tfstate"
+
+endpoints = {
+  s3 = "https://<ACCOUNT_ID>.r2.cloudflarestorage.com"
+}

infra/tofu/backend.tf

@@ -0,0 +1,37 @@
+---
+# Remote state — Cloudflare R2 (S3-compatible), stored OFF the k3s cluster.
+#
+# WHY OFF-CLUSTER (the load-bearing decision): tofu state must live at or below the layer it
+# manages. Its dependency chain is  Proxmox host -> k3s VMs -> k3s cluster -> Gitea/apps, so state
+# in Gitea (top of the chain) would be UNREADABLE in exactly the disaster it exists for — cluster
+# gone, `tofu apply` to rebuild. R2 sits entirely outside that chain: a full cluster rebuild can
+# still read state. It also survives total-site loss (off-site) and gives multi-machine access.
+#
+# SECRETS: the account-specific bucket/endpoint come from a gitignored backend.hcl; the S3 keys
+# come from the environment (.r2.env: AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY). State bytes are
+# encrypted at rest via the TF_ENCRYPTION env blob (see REMOTE-STATE.md) — the passphrase lives ONLY
+# in your password manager and the gitignored .r2.env, never in git or in the state itself.
+#
+# INIT (partial config — non-secret bits here, secret bits from the gitignored file):
+#   cd infra/tofu && source .r2.env && tofu init -backend-config=backend.hcl -migrate-state
+#
+# NOTE: this `terraform {}` block is MERGED with the one in versions.tf (required_version +
+# required_providers). OpenTofu allows multiple terraform blocks across files; `backend` appears once.
+terraform {
+  backend "s3" {
+    region = "auto" # R2 ignores region, but the S3 backend requires a value.
+
+    # R2 is S3-compatible but not AWS: disable the AWS-only preflight/validation paths.
+    skip_credentials_validation = true
+    skip_region_validation      = true
+    skip_requesting_account_id  = true
+    skip_metadata_api_check     = true
+    skip_s3_checksum            = true # R2 rejects AWS trailing (streaming) checksums.
+
+    # bucket, key, endpoints{ s3 } -> backend.hcl (gitignored; keeps the R2 account id out of git).
+    #
+    # use_lockfile = true  -> ENABLE after bumping OpenTofu to >= 1.10. Native S3 conditional-write
+    # locking (no DynamoDB); works on R2 via If-None-Match. 1.9.0 has no lockfile support, so state
+    # locking is OFF for now (single-operator workflow — see REMOTE-STATE.md follow-up).
+  }
+}