home-network-k3s-impl/infra/tofu/README.md

infra/tofu — dev/test cluster nodes (OpenTofu + Proxmox VE)

Provisions the "Create" lifecycle stage (docs/02): 3 cloud-init'd VMs on the Proxmox VE box, each with an imported OS disk and a blank Longhorn data disk. (Migrated from the old memory-constrained libvirt host; provider is now bpg/proxmox.)

APPLIED AND LIVE. These three VMs are the running dev cluster — k3s-dev-1/2/3, k3s v1.36.4, carrying the whole platform. Treat any tofu apply/destroy here as an operation on production-for-dev, and note the state file caveat below.

Serves R-ENV-01/02, R-PLAT-01, ADR-0001 (k3s substrate).

What it builds (defaults = the LIVE cluster, verified 2026-08-26)

Resource	Value
Proxmox node	proxmox
Disk datastore	Storage (ZFS, 4.3TB)
Image/snippet datastore	local (dir; content = iso + snippets)
Bridge	vmbr0 (no VLAN)
Nodes	k3s-dev-1/2/3 at 192.168.1.50/.51/.52, gw 192.168.1.1, DNS 192.168.1.18
Per node	6 vCPU, 24 GiB RAM, cpu type host (nested virt for KubeVirt — real KVM confirmed in-guest)
OS disk	64 GiB, virtio0 (/dev/vda, boot), imported from the Debian 13 cloud image
Data disk	128 GiB, virtio1 (/dev/vdb) → /var/lib/longhorn (mounted by cloud-init; Longhorn reports ~125 GiB usable per node)
etcd disk	16 GiB SSD, virtio2 (/dev/vdc) on the local-zfs SSD datastore → etcd data dir (/var/lib/rancher/k3s/server/db); isolates etcd WAL fsync from the HDD Longhorn I/O (docs/plans/ssd-etcd-isolation.md)
Cloud-init	native initialization (static IP + DNS) + user-data snippet for the baseline
Reserved	192.168.1.53–.60 for Cilium L2 ingress VIPs (surfaced in outputs). In use: .53 shared Gateway, .54 UniFi

⚠️ State file

terraform.tfstate is gitignored and lives only on the workstation that ran the apply. On a fresh workstation it is absent — and a tofu apply with empty state will try to create three VMs that already exist. Recover the state file (or tofu import the existing VMs) before running anything that plans changes. tofu plan against empty state is itself harmless, but read the plan: "3 to add" means you are missing state, not that the cluster is gone.

All sizing/IPs/counts/placement are variables — see variables.tf.

Files

• versions.tf — OpenTofu 1.9.0 pin + bpg/proxmox ~> 0.66; provider block with endpoint scheme-normalization (see below).
• variables.tf — every knob (Proxmox placement, counts, sizing, IPs, image, SSH key).
• main.tf — cloud image download, per-node user-data snippet, and the 3 VMs.
• outputs.tf — node IPs/names, primary_node_ip, k3s_api_endpoint, rendered ansible_inventory.
• terraform.tfvars.example — copy to terraform.tfvars; only ssh_authorized_key required.

Operator prerequisites (before tofu apply)

1. Enable snippets content on the local datastore. The rich first-boot baseline (users, packages, kernel modules, Longhorn mount, swap-off) is shipped as a cloud-init user-data snippet. In the Proxmox UI: Datacenter → Storage → local → Edit → Content → add Snippets. Without this the proxmox_virtual_environment_file (user-data) resource fails. (Alternatively point image_datastore at any datastore that has snippets enabled.)
2. Endpoint scheme. If the operator env file exports PROXMOX_VE_ENDPOINT without https://, the provider talks HTTP and eats a 301. Either fix the env file to include the scheme, or export TF_VAR_proxmox_endpoint="$PROXMOX_VE_ENDPOINT" so tofu normalizes it (prepends https://).
3. pfSense NAT — DONE, live. The workstation reaches the nodes and the kube-API across it: SSH 192.168.2.102:2250/2251/2252 → 192.168.1.50/.51/.52:22, and kube-API 192.168.2.50:6443 → kube-vip VIP 192.168.1.49:6443. infra/ansible/inventory/hosts.ini is hand-maintained for this mapping — do not regenerate it from tofu output -raw ansible_inventory, which knows nothing about the NAT and would overwrite it.

Auth (env only — never in tofu vars/state)

The provider reads these from the environment; the API token is never a tofu variable, so it never enters state, vars, or plan output. You do NOT export these by hand — they live encrypted in secrets.sops.env and the tf.sh wrapper decrypts + exports them (and sets TF_VAR_proxmox_endpoint for scheme normalization) at runtime:

PROXMOX_VE_ENDPOINT=https://192.168.2.102:8006/
PROXMOX_VE_API_TOKEN=user@realm!tokenid=SECRET
PROXMOX_VE_INSECURE=true

See REMOTE-STATE.md for the encrypted-secrets + wrapper flow.

Usage (operator — NOT run here)

All secrets come from the encrypted secrets.sops.env via the tf.sh wrapper, run inside the dev container (which has sops + the age key):

cp infra/tofu/terraform.tfvars.example infra/tofu/terraform.tfvars   # set ssh_authorized_key
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh init
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh apply
# hand the node IPs to Ansible:
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh output -raw ansible_inventory > infra/ansible/inventory/hosts.ini

Hand-off contract

• primary_node_ip / k3s_api_endpoint (192.168.1.50:6443) is the value the Ansible Cilium bootstrap and the Flux platform/cilium HelmRelease must BOTH use for k8sServiceHost / k8sServicePort (docs/02 parity contract).
• Destroy/rebuild proof: name-keyed for_each means tofu destroy -target='proxmox_virtual_environment_vm.node["k3s-dev-2"]' then tofu apply re-creates just that node (stop_on_destroy allows the clean stop); Ansible re-joins it; Longhorn replicas reconverge (nodes are cattle).

Static validation (PM, offline)

tofu fmt -check -recursive
tofu init            # fetches the bpg provider (network, read-only)
tofu validate