Phase 1 (author-only): node provisioning IaC — tofu + cloud-init + Ansible (T-1.1/1.2/1.3)

OpenTofu libvirt module (k3s-dev NAT 10.73.0.0/24, 3x 6GiB/4vCPU VMs, 20G OS + 40G thin data, host-passthrough), cloud-init baseline (k3s/Cilium/Longhorn prereqs, static net), Ansible k3s-HA + Cilium (kube-proxy replacement, Gateway API, Hubble; flannel/kube-proxy/Traefik disabled). PM fix: escape ${...} in a cloud-init template comment (tofu templatefile parse). tofu fmt/init/validate = Success; ansible --syntax-check + yamllint pass. NOTHING executed (no apply). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 01:21:00 -05:00
1 parent 224cd8ac35
commit f2a7e42068
@@ -1,3 +1,57 @@
-# infra/tofu
+# infra/tofu — dev/test cluster nodes (OpenTofu + libvirt)

-_Phase 1 (T-1.x) — to be authored on execution go. See infra/PHASE1-SCOPE.md._
+Phase 1 **T-1.1**. Provisions the "Create" lifecycle stage (docs/02): a dedicated
+NAT network and 3 cloud-init'd KVM VMs, each with a thin OS disk and a blank thin
+Longhorn data disk. **Authoring only — nothing here has been applied.**
+
+Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s substrate), and the network
+half of the LOCKED PHASE1-SCOPE decisions.
+
+## What it builds (defaults = LOCKED decisions)
+
+| Resource | Value |
+|---|---|
+| NAT network | `k3s-dev`, `10.73.0.0/24`, gateway `10.73.0.1`, **DHCP off** (static IPs) |
+| Nodes | `k3s-dev-1/2/3` at `10.73.0.11/.12/.13` |
+| Per node | 4 vCPU, 6 GiB RAM, **host-passthrough** CPU (nested virt for KubeVirt) |
+| OS disk | 20 GiB thin qcow2, backed by the base cloud image |
+| Data disk | 40 GiB blank thin qcow2 → `/dev/vdb`, mounted at `/var/lib/longhorn` by cloud-init |
+| Seed | `libvirt_cloudinit_disk` wired to `../cloud-init/` templates |
+| Reserved | `10.73.0.240–.250` kept out of any DHCP range for Cilium L2 ingress VIPs |
+
+Base OS image is a **variable** (`image_source`), defaulting to Debian 12
+genericcloud amd64. All sizing/IPs/counts are variables — see `variables.tf`.
+
+## Files
+- `versions.tf` — OpenTofu `1.9.0` pin + `dmacvicar/libvirt ~> 0.8.1` provider pin.
+- `variables.tf` — every knob (counts, sizing, IPs, image, SSH key).
+- `main.tf` — network, base/OS/data volumes, cloud-init seed, domains.
+- `outputs.tf` — node IPs/names, `primary_node_ip`, `k3s_api_endpoint`, and a
+  rendered `ansible_inventory` for `infra/ansible/`.
+- `terraform.tfvars.example` — copy to `terraform.tfvars`; only `ssh_authorized_key` is required.
+
+## Usage (operator, in the dev container — NOT run here)
+```
+cd infra/tofu
+cp terraform.tfvars.example terraform.tfvars   # set ssh_authorized_key
+tofu init
+tofu plan
+tofu apply
+# hand the node IPs to Ansible:
+tofu output -raw ansible_inventory > ../ansible/inventory/hosts.ini
+```
+
+## Hand-off contract
+- `primary_node_ip` / `k3s_api_endpoint` (`10.73.0.11:6443`) is the value the
+  Ansible Cilium bootstrap and the Flux `platform/cilium` HelmRelease must BOTH
+  use for `k8sServiceHost` / `k8sServicePort` (docs/02 parity contract).
+- Destroy/rebuild proof (T-1.5): `tofu destroy -target=libvirt_domain.node[1]`
+  then `tofu apply` re-creates that node; Ansible re-joins it; Longhorn replicas
+  reconverge (nodes are cattle — docs/02 statelessness).
+
+## Static validation (PM, in dev container)
+```
+tofu fmt -check -recursive
+tofu init -backend=false
+tofu validate
+```
@@ -0,0 +1,144 @@
+# main.tf — dedicated NAT network + 3 cloud-init'd VMs, each with a thin OS disk
+# (backed by the cloud image) and a blank thin Longhorn data disk (Phase 1 T-1.1).
+#
+# Node lifecycle stage 1 ("Create") per docs/02. cloud-init (infra/cloud-init/)
+# does stage 2 ("Baseline"); Ansible (infra/ansible/) does stage 3 ("Cluster-join").
+
+locals {
+  # 1 GiB in bytes; libvirt volume sizes are bytes.
+  gib = 1073741824
+
+  # Per-node identity/addressing, indexed 0..node_count-1.
+  nodes = [
+    for i in range(var.node_count) : {
+      name     = "${var.node_name_prefix}${i + 1}"
+      ip       = var.node_ips[i]
+      fqdn     = "${var.node_name_prefix}${i + 1}.${var.node_domain}"
+      is_first = i == 0
+    }
+  ]
+}
+
+# --- dedicated NAT network (10.73.0.0/24, gw .1) ---------------------------
+# DHCP is OFF: nodes get static IPs from cloud-init network-config, so the
+# reserved VIP range (.240-.250) can never be leased to something else.
+resource "libvirt_network" "k3s_dev" {
+  name      = var.network_name
+  mode      = "nat"
+  addresses = [var.network_cidr]
+  autostart = true
+
+  dhcp {
+    enabled = false
+  }
+
+  dns {
+    enabled    = true
+    local_only = true
+  }
+}
+
+# --- base OS volume (downloaded/copied once, shared as a backing file) ------
+resource "libvirt_volume" "base" {
+  name   = "${var.cluster_name}-base.qcow2"
+  pool   = var.pool_name
+  source = var.image_source
+  format = "qcow2"
+}
+
+# --- per-node OS disk: thin qcow2 backed by the base image ------------------
+resource "libvirt_volume" "os" {
+  count          = var.node_count
+  name           = "${local.nodes[count.index].name}-os.qcow2"
+  pool           = var.pool_name
+  base_volume_id = libvirt_volume.base.id
+  size           = var.os_disk_gib * local.gib
+  format         = "qcow2"
+}
+
+# --- per-node data disk: blank thin qcow2 for Longhorn ----------------------
+resource "libvirt_volume" "data" {
+  count  = var.node_count
+  name   = "${local.nodes[count.index].name}-data.qcow2"
+  pool   = var.pool_name
+  size   = var.data_disk_gib * local.gib
+  format = "qcow2"
+}
+
+# --- cloud-init seed (user-data + meta-data + static network-config) --------
+# Templates live in the sibling infra/cloud-init/ subtree (T-1.2).
+resource "libvirt_cloudinit_disk" "seed" {
+  count = var.node_count
+  name  = "${local.nodes[count.index].name}-seed.iso"
+  pool  = var.pool_name
+
+  user_data = templatefile("${path.module}/../cloud-init/user-data.yaml.tftpl", {
+    hostname           = local.nodes[count.index].name
+    fqdn               = local.nodes[count.index].fqdn
+    username           = var.login_user
+    ssh_authorized_key = var.ssh_authorized_key
+  })
+
+  meta_data = templatefile("${path.module}/../cloud-init/meta-data.yaml.tftpl", {
+    hostname = local.nodes[count.index].name
+    fqdn     = local.nodes[count.index].fqdn
+  })
+
+  network_config = templatefile("${path.module}/../cloud-init/network-config.yaml.tftpl", {
+    ip      = local.nodes[count.index].ip
+    prefix  = var.network_prefix
+    gateway = var.network_gateway
+  })
+}
+
+# --- the VMs ----------------------------------------------------------------
+resource "libvirt_domain" "node" {
+  count  = var.node_count
+  name   = local.nodes[count.index].name
+  memory = var.memory_mib
+  vcpu   = var.vcpu
+
+  # host-passthrough so nested virt is available for KubeVirt in Phase 5.
+  cpu {
+    mode = var.cpu_mode
+  }
+
+  # qemu-guest-agent is installed by cloud-init; lets tofu read the lease/agent.
+  qemu_agent = true
+
+  cloudinit = libvirt_cloudinit_disk.seed[count.index].id
+
+  # IP + hostname come from cloud-init (DHCP is off on this network), so we do NOT
+  # set addresses/hostname here — the provider couples those to DHCP reservations.
+  network_interface {
+    network_id     = libvirt_network.k3s_dev.id
+    wait_for_lease = false
+  }
+
+  # OS disk first (vda, boot), Longhorn data disk second (vdb).
+  disk {
+    volume_id = libvirt_volume.os[count.index].id
+  }
+
+  disk {
+    volume_id = libvirt_volume.data[count.index].id
+  }
+
+  console {
+    type        = "pty"
+    target_port = "0"
+    target_type = "serial"
+  }
+
+  console {
+    type        = "pty"
+    target_port = "1"
+    target_type = "virtio"
+  }
+
+  graphics {
+    type        = "spice"
+    listen_type = "address"
+    autoport    = true
+  }
+}
@@ -0,0 +1,58 @@
+# outputs.tf — surface the facts the Ansible inventory needs (Phase 1 T-1.1).
+# The primary node is the k3s cluster-init server and the k8sServiceHost the
+# Cilium bootstrap + Flux HelmRelease must agree on (docs/02 parity contract).
+
+output "node_names" {
+  description = "Ordered VM hostnames."
+  value       = [for n in local.nodes : n.name]
+}
+
+output "node_ips" {
+  description = "Ordered node static IPs (same order as node_names)."
+  value       = [for n in local.nodes : n.ip]
+}
+
+output "nodes" {
+  description = "Per-node map name => ip, for building the Ansible inventory."
+  value       = { for n in local.nodes : n.name => n.ip }
+}
+
+output "primary_node_ip" {
+  description = "First server IP — k3s --cluster-init host and Cilium k8sServiceHost (parity value)."
+  value       = local.nodes[0].ip
+}
+
+output "k3s_api_endpoint" {
+  description = "kube-apiserver endpoint used for k8sServiceHost/Port parity (docs/02)."
+  value       = "${local.nodes[0].ip}:6443"
+}
+
+output "cilium_l2_vip_range" {
+  description = "Reserved ingress VIP range for Cilium L2 announcement (WS-NET consumes in Phase 2)."
+  value       = "${var.cilium_l2_vip_range_start}-${var.cilium_l2_vip_range_end}"
+}
+
+# Rendered INI inventory for infra/ansible/. The Ansible dir also ships a static
+# inventory aligned with these defaults; regenerate it from this output if the
+# IPs/names are overridden. `tofu output -raw ansible_inventory > ../ansible/inventory/hosts.ini`
+output "ansible_inventory" {
+  description = "Ready-to-write Ansible INI inventory aligned with tofu outputs."
+  value       = <<-EOT
+    # GENERATED from tofu outputs — do not hand-edit; re-render on IP/name change.
+    [k3s_primary]
+    ${local.nodes[0].name} ansible_host=${local.nodes[0].ip}
+
+    [k3s_secondary]
+    %{for n in slice(local.nodes, 1, length(local.nodes))~}
+    ${n.name} ansible_host=${n.ip}
+    %{endfor~}
+
+    [k3s_servers:children]
+    k3s_primary
+    k3s_secondary
+
+    [k3s_servers:vars]
+    ansible_user=${var.login_user}
+    k3s_api_host=${local.nodes[0].ip}
+  EOT
+}
@@ -0,0 +1,37 @@
+# terraform.tfvars.example — copy to terraform.tfvars and edit.
+#
+# NOTE on secrets/gitignore: real *.tfvars are gitignored (root .gitignore) so
+# local values never land in git. This *.example file is safe to commit — it
+# carries NO private material (SSH *public* key only). Never put a private key here.
+#
+# Defaults for everything below already encode the LOCKED PHASE1-SCOPE decisions;
+# the ONLY required value is ssh_authorized_key.
+
+# REQUIRED — your SSH *public* key (contents of e.g. ~/.ssh/id_ed25519.pub).
+ssh_authorized_key = "ssh-ed25519 AAAA...replace-with-your-public-key... user@host"
+
+# --- optional overrides (defaults shown) -----------------------------------
+
+# libvirt_uri   = "qemu:///system"
+# pool_name     = "default"
+
+# node_count       = 3
+# node_name_prefix = "k3s-dev-"
+# node_ips         = ["10.73.0.11", "10.73.0.12", "10.73.0.13"]
+
+# vcpu          = 4
+# memory_mib    = 6144
+# os_disk_gib   = 20
+# data_disk_gib = 40
+# cpu_mode      = "host-passthrough"
+
+# network_name    = "k3s-dev"
+# network_cidr    = "10.73.0.0/24"
+# network_gateway = "10.73.0.1"
+# network_prefix  = 24
+
+# login_user  = "k3s"
+# node_domain = "k3s.dev.local"
+
+# Point at a locally pre-downloaded image to avoid re-fetching on each apply:
+# image_source = "/var/lib/libvirt/images/debian-12-genericcloud-amd64.qcow2"
@@ -0,0 +1,151 @@
+# variables.tf — every sizing / IP / image knob is a variable (Phase 1 T-1.1).
+# Defaults encode the LOCKED decisions in infra/PHASE1-SCOPE.md. Override via
+# terraform.tfvars (see terraform.tfvars.example).
+
+variable "libvirt_uri" {
+  description = "libvirt connection URI."
+  type        = string
+  default     = "qemu:///system"
+}
+
+variable "pool_name" {
+  description = "libvirt storage pool for all volumes (base image, OS + data disks, seed ISOs)."
+  type        = string
+  default     = "default"
+}
+
+# --- cluster identity / naming ---------------------------------------------
+
+variable "cluster_name" {
+  description = "Logical cluster name; used as a resource-name prefix."
+  type        = string
+  default     = "k3s-dev"
+}
+
+variable "node_name_prefix" {
+  description = "VM hostname prefix. Nodes are <prefix>1, <prefix>2, ... (LOCKED: k3s-dev-1/2/3)."
+  type        = string
+  default     = "k3s-dev-"
+}
+
+variable "node_count" {
+  description = "Number of k3s nodes (LOCKED: 3-server HA embedded etcd)."
+  type        = number
+  default     = 3
+
+  validation {
+    condition     = var.node_count == length(var.node_ips)
+    error_message = "node_count must equal the number of entries in node_ips."
+  }
+}
+
+# --- per-node sizing (LOCKED: 4 vCPU / 6 GiB / 20 GiB OS + 40 GiB data) -----
+
+variable "vcpu" {
+  description = "vCPUs per node."
+  type        = number
+  default     = 4
+}
+
+variable "memory_mib" {
+  description = "RAM per node in MiB (6 GiB)."
+  type        = number
+  default     = 6144
+}
+
+variable "os_disk_gib" {
+  description = "OS (root) disk size per node, GiB. Thin qcow2 backed by the cloud image."
+  type        = number
+  default     = 20
+}
+
+variable "data_disk_gib" {
+  description = "Dedicated Longhorn data disk per node, GiB. Thin qcow2, blank."
+  type        = number
+  default     = 40
+}
+
+variable "cpu_mode" {
+  description = "libvirt CPU mode. host-passthrough exposes nested-virt for KubeVirt (Phase 5)."
+  type        = string
+  default     = "host-passthrough"
+}
+
+# --- network (LOCKED: dedicated NAT k3s-dev 10.73.0.0/24, gw .1) ------------
+
+variable "network_name" {
+  description = "libvirt NAT network name."
+  type        = string
+  default     = "k3s-dev"
+}
+
+variable "network_cidr" {
+  description = "Cluster NAT subnet in CIDR. Gateway (network_gateway) is the libvirt bridge host IP."
+  type        = string
+  default     = "10.73.0.0/24"
+}
+
+variable "network_prefix" {
+  description = "Netmask prefix length for node static addressing."
+  type        = number
+  default     = 24
+}
+
+variable "network_gateway" {
+  description = "Default gateway / DNS for nodes = libvirt bridge host IP (LOCKED: 10.73.0.1)."
+  type        = string
+  default     = "10.73.0.1"
+}
+
+variable "node_ips" {
+  description = "Static IPs per node, in order (LOCKED: .11/.12/.13)."
+  type        = list(string)
+  default     = ["10.73.0.11", "10.73.0.12", "10.73.0.13"]
+}
+
+# Reserved for Cilium L2-announcement ingress VIPs (LOCKED: .240-.250).
+# Not materialised as libvirt objects; surfaced in outputs and kept OUT of any
+# DHCP range so nothing else claims them. Cilium's IP pool (WS-NET, Phase 2)
+# consumes this range.
+variable "cilium_l2_vip_range_start" {
+  description = "First reserved Cilium L2 VIP."
+  type        = string
+  default     = "10.73.0.240"
+}
+
+variable "cilium_l2_vip_range_end" {
+  description = "Last reserved Cilium L2 VIP."
+  type        = string
+  default     = "10.73.0.250"
+}
+
+# --- base OS image ----------------------------------------------------------
+
+variable "image_source" {
+  description = <<-EOT
+    Source for the base OS volume: a URL or local path to a minimal cloud image
+    (qcow2). Default is Debian 12 (bookworm) genericcloud amd64. Point at a local
+    path to avoid re-downloading on every apply.
+  EOT
+  type        = string
+  default     = "https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.qcow2"
+}
+
+# --- cloud-init identity ----------------------------------------------------
+
+variable "ssh_authorized_key" {
+  description = "SSH PUBLIC key injected into the login user (public only — never a private key)."
+  type        = string
+}
+
+variable "login_user" {
+  description = "Non-root login user created by cloud-init; also the Ansible ansible_user."
+  type        = string
+  default     = "k3s"
+}
+
+variable "node_domain" {
+  description = "DNS domain suffix for node FQDNs (informational; dev uses split-horizon / hosts)."
+  type        = string
+  default     = "k3s.dev.local"
+}
@@ -0,0 +1,25 @@
+# versions.tf — OpenTofu + provider pins (Phase 1 T-1.1).
+#
+# Toolchain is pinned to match the dev container (infra/devcontainer/versions.env):
+#   OpenTofu 1.9.0. The dmacvicar/libvirt provider is pinned HERE (not in the
+#   dev container) per the note in that file.
+#
+# AUTHOR-ONLY: this module is written but NOT applied. The PM static-validates
+# with `tofu fmt -check` / `tofu validate` in the dev container.
+
+terraform {
+  required_version = "1.9.0"
+
+  required_providers {
+    libvirt = {
+      source  = "dmacvicar/libvirt"
+      version = "~> 0.8.1"
+    }
+  }
+}
+
+provider "libvirt" {
+  # System-level libvirt (qemu:///system) so VMs/pools/networks are host-wide,
+  # matching the dev-container mount of the host libvirt socket (PHASE1-SCOPE risks).
+  uri = var.libvirt_uri
+}