home-network-k3s-impl/infra/tofu/README.md

infra/tofu — dev/test cluster nodes (OpenTofu + Proxmox VE)

Provisions the "Create" lifecycle stage (docs/02): 3 cloud-init'd VMs on the Proxmox VE box, each with an imported OS disk and a blank Longhorn data disk. Authoring only — nothing here has been applied. (Migrated from the old memory-constrained libvirt host; provider is now bpg/proxmox.)

Serves R-ENV-01/02, R-PLAT-01, ADR-0001 (k3s substrate).

What it builds (defaults = VERIFIED Proxmox targets)

Resource	Value
Proxmox node	proxmox
Disk datastore	Storage (ZFS, 4.3TB)
Image/snippet datastore	local (dir; content = iso + snippets)
Bridge	vmbr0 (no VLAN)
Nodes	k3s-dev-1/2/3 at 192.168.1.50/.51/.52, gw 192.168.1.1, DNS 192.168.1.18
Per node	6 vCPU, 16 GiB RAM, cpu type host (nested virt for KubeVirt)
OS disk	40 GiB, virtio0 (/dev/vda, boot), imported from the Debian 12 cloud image
Data disk	100 GiB, virtio1 (/dev/vdb) → /var/lib/longhorn (mounted by cloud-init)
Cloud-init	native initialization (static IP + DNS) + user-data snippet for the baseline
Reserved	192.168.1.53–.60 for Cilium L2 ingress VIPs (surfaced in outputs)

All sizing/IPs/counts/placement are variables — see variables.tf.

Files

• versions.tf — OpenTofu 1.9.0 pin + bpg/proxmox ~> 0.66; provider block with endpoint scheme-normalization (see below).
• variables.tf — every knob (Proxmox placement, counts, sizing, IPs, image, SSH key).
• main.tf — cloud image download, per-node user-data snippet, and the 3 VMs.
• outputs.tf — node IPs/names, primary_node_ip, k3s_api_endpoint, rendered ansible_inventory.
• terraform.tfvars.example — copy to terraform.tfvars; only ssh_authorized_key required.

Operator prerequisites (before tofu apply)

1. Enable snippets content on the local datastore. The rich first-boot baseline (users, packages, kernel modules, Longhorn mount, swap-off) is shipped as a cloud-init user-data snippet. In the Proxmox UI: Datacenter → Storage → local → Edit → Content → add Snippets. Without this the proxmox_virtual_environment_file (user-data) resource fails. (Alternatively point image_datastore at any datastore that has snippets enabled.)
2. Endpoint scheme. If the operator env file exports PROXMOX_VE_ENDPOINT without https://, the provider talks HTTP and eats a 301. Either fix the env file to include the scheme, or export TF_VAR_proxmox_endpoint="$PROXMOX_VE_ENDPOINT" so tofu normalizes it (prepends https://).
3. pfSense SSH NAT (network-dependent): the PM reaches these nodes and the k3s API over a pfSense NAT that is not yet created; the Ansible inventory carries a note to finalize ansible_host once the NAT rule exists.

Auth (env only — never in tofu vars/state)

The provider reads these from the environment; the API token is never a tofu variable, so it never enters state, vars, or plan output:

export PROXMOX_VE_ENDPOINT="https://192.168.2.102:8006/"
export PROXMOX_VE_API_TOKEN="user@realm!tokenid=SECRET"
export PROXMOX_VE_INSECURE="true"
export TF_VAR_proxmox_endpoint="$PROXMOX_VE_ENDPOINT"   # optional; enables normalization

Usage (operator — NOT run here)

cd infra/tofu
cp terraform.tfvars.example terraform.tfvars   # set ssh_authorized_key
# source the operator env file (exports the PROXMOX_VE_* above)
tofu init
tofu plan
tofu apply
# hand the node IPs to Ansible:
tofu output -raw ansible_inventory > ../ansible/inventory/hosts.ini

Hand-off contract

• primary_node_ip / k3s_api_endpoint (192.168.1.50:6443) is the value the Ansible Cilium bootstrap and the Flux platform/cilium HelmRelease must BOTH use for k8sServiceHost / k8sServicePort (docs/02 parity contract).
• Destroy/rebuild proof: name-keyed for_each means tofu destroy -target='proxmox_virtual_environment_vm.node["k3s-dev-2"]' then tofu apply re-creates just that node (stop_on_destroy allows the clean stop); Ansible re-joins it; Longhorn replicas reconverge (nodes are cattle).

Static validation (PM, offline)

tofu fmt -check -recursive
tofu init            # fetches the bpg provider (network, read-only)
tofu validate