feat(tofu,ansible): SSD-backed etcd disk per k3s node (Phase 1: code + plan)

Interim bridge to stop Longhorn replica-rebuild I/O storms on the shared HDD
Storage pool from starving etcd WAL fsync (crash-looping k3s / flapping the
kube-vip VIP). Adds a small dedicated SSD disk per node for etcd only; Longhorn
stays on HDD. See docs/plans/ssd-etcd-isolation.md.

tofu (infra/tofu):
- New vars etcd_ssd_datastore (default "local-zfs" = rpool/data, Micron 5100
  SATA SSD mirror w/ PLP) and etcd_disk_gib (default 16).
- New virtio2 disk block on the SSD datastore per VM: discard=on, cache=none
  (honor guest fsync straight to the ZFS ZIL — safe on PLP SSD), iothread=false.
  Kept last (ascending interface) so bpg does not reorder virtio0/virtio1.
- Revert the debian_cloud_image resource to the long type name
  proxmox_virtual_environment_download_file to match the R2 state: bpg >= 0.111
  dropped the cross-type moved{} linkage to the short proxmox_download_file, so
  the stale moved block failed EVERY plan with "Resource type mismatch" (broken
  on main too). Aligning code to state yields a clean plan with no image
  re-download. Short-name migration is deferred to a deliberate `tofu state mv`.

ansible (node_baseline):
- Format /dev/vdc (label k3s-etcd) and mount it at
  /var/lib/rancher/k3s/server/db (defaults,noatime; no nofail so a missing etcd
  disk fails loudly rather than silently landing etcd on the root HDD) before
  k3s starts, so a rebuilt node comes up with etcd isolated on SSD. Data-safe:
  filesystem module never reformats an existing fs.

Phase 1 only: non-disruptive. No tofu apply, no node/VM/etcd/Longhorn changes.
tofu plan verified: 0 to add, 3 to change, 0 to destroy — every VM an in-place
disk ADD (virtio2), dev-3 also an in-place memory 16384->24576; no VM replace.
Merge gate green (render==golden included).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

infra/ansible/roles/node_baseline/defaults/main.yml

@@ -15,3 +15,10 @@ node_baseline_kernel_modules:
 node_baseline_longhorn_device: /dev/vdb
 node_baseline_longhorn_mount: /var/lib/longhorn
 node_baseline_longhorn_fs_label: longhorn
+# Dedicated SSD etcd disk (tofu virtio2 -> /dev/vdc), mounted at the k3s etcd data dir
+# so etcd WAL fsync is isolated from Longhorn rebuild I/O on the HDD data disk.
+# See docs/plans/ssd-etcd-isolation.md. FS label keeps the fstab entry stable across
+# device renames (matches the Longhorn LABEL= approach).
+node_baseline_etcd_device: /dev/vdc
+node_baseline_etcd_mount: /var/lib/rancher/k3s/server/db
+node_baseline_etcd_fs_label: k3s-etcd

infra/ansible/roles/node_baseline/tasks/main.yml

@@ -72,6 +72,31 @@
     opts: "defaults,nofail,x-systemd.device-timeout=5s"
     state: mounted
 
+# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
+# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
+# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
+# a disk that already carries a filesystem, so re-running against an already-migrated
+# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
+# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
+# role would mount over a populated directory.
+- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
+  community.general.filesystem:
+    fstype: ext4
+    dev: "{{ node_baseline_etcd_device }}"
+    opts: "-L {{ node_baseline_etcd_fs_label }}"
+
+- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
+  ansible.posix.mount:
+    path: "{{ node_baseline_etcd_mount }}"
+    src: "LABEL={{ node_baseline_etcd_fs_label }}"
+    fstype: ext4
+    # noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
+    # missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
+    opts: "defaults,noatime"
+    dump: "0"
+    passno: "2"
+    state: mounted
+
 - name: Disable swap now
   ansible.builtin.command: swapoff -a
   changed_when: false

infra/tofu/main.tf

@@ -41,13 +41,17 @@ locals {
 # -> Storage -> local -> Content -> check "Import"), else the disk import fails with
 # "wrong type 'iso' - needs to be 'images' or 'import'".
 #
-# Resource type is the SHORT name `proxmox_download_file` (bpg/proxmox ADR-007). The old
-# `proxmox_virtual_environment_download_file` is deprecated ("Use `proxmox_download_file`
-# instead. This resource will be removed in v1.0.") — both are the same dual-registered
-# Framework resource, so the schema is identical. The `moved` block below migrates existing
-# state in place (provider implements MoveState / migration.PrefixMoveState), so no
-# re-download of the image and no change to the VMs.
-resource "proxmox_download_file" "debian_cloud_image" {
+# The Debian cloud image is a bpg `download_file` resource. This intentionally uses the
+# LONG (deprecated) type name `proxmox_virtual_environment_download_file`: the R2 backend
+# state still holds this resource under the long name, and bpg >= 0.111 dropped the
+# cross-type `moved{}` linkage to the short `proxmox_download_file` (ADR-007) — a `moved`
+# block between the two now fails plan with "Resource type mismatch" (this broke plan on
+# main entirely). Keeping the long name makes config match state, so the plan is clean and
+# the ~450 MB image is NOT re-downloaded. Migrating to the short name is a SEPARATE,
+# deliberate step (`tofu state mv proxmox_virtual_environment_download_file.debian_cloud_image
+# proxmox_download_file.debian_cloud_image`, then rename here) — not bundled with the
+# etcd-disk change so this bounce stays focused and low-risk.
+resource "proxmox_virtual_environment_download_file" "debian_cloud_image" {
   content_type = "import"
   datastore_id = var.image_datastore
   node_name    = var.proxmox_node
@@ -56,15 +60,6 @@ resource "proxmox_download_file" "debian_cloud_image" {
   overwrite    = false
 }
 
-# State migration for the deprecated -> short resource-type rename (ADR-007). OpenTofu
-# 1.9 (>= 1.8 required) drives the provider's MoveState RPC: the download_file state is
-# transferred in place, keeping the same id (datastore_id:content_type/file_name), so the
-# already-downloaded Debian image is NOT re-downloaded and the VMs' import_from is unchanged.
-moved {
-  from = proxmox_virtual_environment_download_file.debian_cloud_image
-  to   = proxmox_download_file.debian_cloud_image
-}
-
 # --- cloud-init: NATIVE Proxmox cloud-init only (API-driven) ------------------
 # The rich first-boot baseline (packages, kernel modules, Longhorn disk, swap-off)
 # is NOT delivered as a custom snippet any more — custom snippets require SSH/SFTP
@@ -118,7 +113,7 @@ resource "proxmox_virtual_environment_vm" "node" {
   # fs_setup/mount for /var/lib/longhorn already expects.
   disk {
     datastore_id = var.datastore
-    import_from  = proxmox_download_file.debian_cloud_image.id
+    import_from  = proxmox_virtual_environment_download_file.debian_cloud_image.id
     interface    = "virtio0"
     size         = var.os_disk_gib
     discard      = "on"
@@ -131,6 +126,22 @@ resource "proxmox_virtual_environment_vm" "node" {
     discard      = "on"
   }
 
+  # Dedicated SSD etcd disk (virtio2 -> /dev/vdc). Isolates etcd WAL fsync from the HDD
+  # `Storage` pool where the OS + Longhorn disks live — the node_baseline role formats
+  # it and mounts it at /var/lib/rancher/k3s/server/db before k3s starts. On the SSD
+  # `local-zfs` (rpool/data) datastore: an enterprise Micron SATA SSD mirror WITH
+  # power-loss protection, so cache=none (honor guest fsync straight through to the ZFS
+  # ZIL) is both durable and fast. Kept last (ascending interface) so bpg won't reorder
+  # the existing virtio0/virtio1 disks. See docs/plans/ssd-etcd-isolation.md.
+  disk {
+    datastore_id = var.etcd_ssd_datastore
+    interface    = "virtio2"
+    size         = var.etcd_disk_gib
+    discard      = "on"
+    cache        = "none"
+    iothread     = false
+  }
+
   network_device {
     bridge = var.bridge
   }

infra/tofu/variables.tf

@@ -100,6 +100,24 @@ variable "data_disk_gib" {
   default     = 128
 }
 
+# --- dedicated SSD etcd disk (interim etcd-on-SSD isolation, docs/plans/ssd-etcd-isolation.md) --
+# etcd WAL fsync was being starved by Longhorn replica-rebuild I/O on the shared HDD
+# `Storage` pool (OS + Longhorn disks). A tiny dedicated SSD disk per node moves the
+# etcd data dir (/var/lib/rancher/k3s/server/db, mounted by node_baseline) off the HDD.
+# Interim bridge until the all-SSD mini-PCs land; Longhorn stays on HDD (does not fit).
+
+variable "etcd_ssd_datastore" {
+  description = "SSD-backed datastore for the dedicated etcd disk (VERIFIED: local-zfs = rpool/data, ZFS mirror on Micron 5100 SATA SSDs w/ power-loss protection)."
+  type        = string
+  default     = "local-zfs"
+}
+
+variable "etcd_disk_gib" {
+  description = "Dedicated SSD etcd disk per node, GiB (etcd DB is ~83 MB; 16 GiB is generous headroom)."
+  type        = number
+  default     = 16
+}
+
 variable "cpu_type" {
   description = "Proxmox vCPU type. `host` passes through the physical CPU (nested-virt for KubeVirt, Phase 5)."
   type        = string