From af5fc5b25f5f3e2854d53b92d3b98ae577d5b8f0 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 29 Aug 2026 17:21:06 -0500 Subject: [PATCH] feat(tofu,ansible): SSD-backed etcd disk per k3s node (Phase 1: code + plan) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Interim bridge to stop Longhorn replica-rebuild I/O storms on the shared HDD Storage pool from starving etcd WAL fsync (crash-looping k3s / flapping the kube-vip VIP). Adds a small dedicated SSD disk per node for etcd only; Longhorn stays on HDD. See docs/plans/ssd-etcd-isolation.md. tofu (infra/tofu): - New vars etcd_ssd_datastore (default "local-zfs" = rpool/data, Micron 5100 SATA SSD mirror w/ PLP) and etcd_disk_gib (default 16). - New virtio2 disk block on the SSD datastore per VM: discard=on, cache=none (honor guest fsync straight to the ZFS ZIL — safe on PLP SSD), iothread=false. Kept last (ascending interface) so bpg does not reorder virtio0/virtio1. - Revert the debian_cloud_image resource to the long type name proxmox_virtual_environment_download_file to match the R2 state: bpg >= 0.111 dropped the cross-type moved{} linkage to the short proxmox_download_file, so the stale moved block failed EVERY plan with "Resource type mismatch" (broken on main too). Aligning code to state yields a clean plan with no image re-download. Short-name migration is deferred to a deliberate `tofu state mv`. ansible (node_baseline): - Format /dev/vdc (label k3s-etcd) and mount it at /var/lib/rancher/k3s/server/db (defaults,noatime; no nofail so a missing etcd disk fails loudly rather than silently landing etcd on the root HDD) before k3s starts, so a rebuilt node comes up with etcd isolated on SSD. Data-safe: filesystem module never reformats an existing fs. Phase 1 only: non-disruptive. No tofu apply, no node/VM/etcd/Longhorn changes. tofu plan verified: 0 to add, 3 to change, 0 to destroy — every VM an in-place disk ADD (virtio2), dev-3 also an in-place memory 16384->24576; no VM replace. Merge gate green (render==golden included). Co-Authored-By: Claude Opus 4.8 --- docs/plans/ssd-etcd-isolation.md | 270 ++++++++++++++++++ .../roles/node_baseline/defaults/main.yml | 7 + .../roles/node_baseline/tasks/main.yml | 25 ++ infra/tofu/main.tf | 45 +-- infra/tofu/variables.tf | 18 ++ 5 files changed, 348 insertions(+), 17 deletions(-) create mode 100644 docs/plans/ssd-etcd-isolation.md diff --git a/docs/plans/ssd-etcd-isolation.md b/docs/plans/ssd-etcd-isolation.md new file mode 100644 index 0000000..cfeef0b --- /dev/null +++ b/docs/plans/ssd-etcd-isolation.md @@ -0,0 +1,270 @@ +# SSD-backed etcd isolation for the k3s dev cluster + +**Status:** PROPOSAL — awaiting user approval. No changes have been made. +**Author:** investigation agent (read-only), 2026-08-29 +**Problem:** Longhorn replica-rebuild I/O storms on the HDD-backed VM disks starve +etcd's WAL fsync, crash-looping k3s and flapping the kube-vip VIP +(`192.168.2.50:6443`). Even `concurrent-replica-rebuild-per-node-limit=1` tips etcd. +**Question:** Can the Proxmox host's ~1 TB system SSD (`rpool`) host dedicated +SSD-backed storage for etcd (and/or Longhorn) to end these incidents? + +**Verdict: YES for etcd (strongly recommended). NO for Longhorn.** + +--- + +## 1. Measured facts (all collected read-only on 2026-08-29) + +### 1.1 `rpool` — the system/boot SSD pool + +| Property | Value | +|---|---| +| Pool | `rpool`, **mirror** (`mirror-0`) of 2 devices | +| Devices | `ata-MTFDDAK960TDS_...` × 2 — **Micron 5100/52x0 960 GB, 2.5" SATA** | +| Class | **Enterprise SATA SSD with power-loss protection (PLP)** — safe for sync/fsync | +| Health | `Percent_Lifetime_Remain` = **97%** (3% used), 13 904 power-on hours, 0 reallocated sectors, ~96 TB written. Excellent. | +| Pool size / alloc / **free** | 888 G / 538 G / **350 G free** (`zpool list`, 60% cap) | +| ZFS `AVAIL` | **~323 GiB** available to any new dataset/zvol | +| System root | `rpool/ROOT/pve-1` = **430 G used** (Proxmox host `/`) | +| Proxmox storage on it | `local-zfs` → pool `rpool/data`, `content images,rootdir`, **`sparse 1`** (thin), ~338 GiB available per `pvesm status` | +| Pool settings | `compression on`, **`sync standard`** (inherited), `dedup off`, recordsize 128K | + +`local-zfs` already advertises `images,rootdir` content, so **Proxmox can place VM +disks on the SSD today with no storage-config change.** + +### 1.2 Current VM disk backing — confirms the HDD assumption + +VMs (bpg/OpenTofu-provisioned): **dev-1 = vmid 105, dev-2 = 102, dev-3 = 101**. +Each has two disks, **both on the `Storage` datastore**: + +- `Storage` = ZFS pool on **2× WDC WUH721818ALE6L4 18 TB HDD** (`sdc`/`sdd`, `ROTA=1`), mirror, 16.4 T. +- Per node: `virtio0 = vm--disk-0` (64 G, OS/root) and `virtio1 = vm--disk-1` (128 G, Longhorn data). `volblocksize 16K`. + +So **etcd, the OS, and Longhorn all currently sit on spinning rust.** That is the +root cause: etcd WAL fsync competes with Longhorn rebuild I/O on the same HDD spindles. + +(`Storage2` = 2× Seagate ST8000VN004 8 TB HDD is a separate HDD pool, not involved.) + +### 1.3 Size of the hot data + +- **etcd DB: ~82.6 MB.** Measured via apiserver metric + `apiserver_storage_size_bytes{storage_cluster_id="etcd-0"} = 8.264e7`. + A 4-day-old cluster running ~18 services. Even allowing generous WAL + growth + headroom, **etcd fits in single-digit GB with room to spare — trivially ×3.** +- **Longhorn: ~22 GB actual data per node right now**, but the data disk is + **128 GiB** and Longhorn reports `storageScheduled ≈ 152 GiB` (thin-overcommitted) + vs `storageMaximum ≈ 125 GiB` per node. 18 PVCs, `numberOfReplicas=3`. + Provisioned total ≈ 152 GiB. Longhorn is the **churn source we want to isolate + etcd from**, and it can grow toward 128 GiB/node. + +### 1.4 Sizing conclusion + +| Candidate | Per-node footprint | ×3 on rpool | Fits in 323 GiB avail? | +|---|---|---|---| +| **etcd** | ~83 MB now; size a 16 GiB disk | ~48 GiB reserved (thin) | **Yes, easily** | +| **Longhorn** | ~22 GB now, up to 128 GiB provisioned | up to **~384 GiB** | **No — would fill/threaten the boot disk** | + +--- + +## 2. Viability verdict + +**etcd on SSD: viable and high-leverage.** etcd is ~83 MB; three 16 GiB SSD-backed +disks cost ~48 GiB of thin allocation (actual ~250 MB) against 323 GiB free on an +enterprise, PLP-equipped, 97%-healthy SSD mirror. This directly removes the WAL +fsync path from the HDD contention domain — the smallest-footprint fix for the exact +failure mode. **This is the recommendation.** + +**Longhorn on SSD: not viable here.** Its provisioned footprint (~384 GiB across 3 +replicas at full size) does not fit alongside the 430 GiB Proxmox root on a 888 GiB +pool, and co-locating heavy Longhorn rebuild I/O on the boot SSD reintroduces +contention with the host itself. Keep Longhorn on HDD (`Storage`). Rejected. + +--- + +## 3. Options evaluated + +- **Option A — dedicated small SSD disk per node, etcd-only (RECOMMENDED).** + Add a ~16 GiB SSD-backed virtual disk (zvol on `rpool/data` via `local-zfs`) to + each of the 3 VMs, mount it at the k3s etcd directory + (`/var/lib/rancher/k3s/server/db`), migrate etcd data onto it one node at a time. + Isolates WAL fsync from HDD Longhorn churn. Minimal space, minimal blast radius. +- **Option B — Longhorn replicas/disk on SSD.** Rejected (§2): does not fit and + re-couples host I/O. +- **Option C — do nothing; wait for the SSD mini-PCs.** Zero risk, zero work, but + leaves the cluster fragile to every node bounce until the mini-PC migration lands + (deferred, no committed date). Option A is a cheap, reversible bridge that Option C + eventually supersedes (§7). + +**Recommendation: Option A now, as an interim fix, carried by tofu.** + +--- + +## 4. Recommended implementation — Option A, step-by-step + +### 4.1 Model it in tofu (preferred over manual) + +The VMs are bpg/OpenTofu-managed (`infra/tofu/`). Disks are declared as `disk` +blocks in `main.tf`; both current disks (`virtio0`, `virtio1`) point at +`var.datastore` (`Storage`). Add a **third disk on the SSD datastore**: + +1. **New variables** in `infra/tofu/variables.tf`: + ```hcl + variable "etcd_ssd_datastore" { + description = "SSD-backed datastore for the dedicated etcd disk (VERIFIED: local-zfs = rpool/data, ZFS mirror on Micron 5100 SATA SSDs w/ PLP)." + type = string + default = "local-zfs" + } + variable "etcd_disk_gib" { + description = "Dedicated SSD etcd disk per node, GiB (etcd DB is ~83 MB; 16 GiB is generous headroom)." + type = number + default = 16 + } + ``` +2. **New disk block** in the VM resource in `main.tf`, appended AFTER the existing + `virtio1` block (keep interface order ascending so bpg does not reorder): + ```hcl + # Dedicated SSD etcd disk (virtio2 -> /dev/vdc). Isolates etcd WAL fsync from + # the HDD Storage pool where the OS + Longhorn disks live. + disk { + datastore_id = var.etcd_ssd_datastore + interface = "virtio2" + size = var.etcd_disk_gib + discard = "on" + cache = "none" # honor guest fsync through to ZFS ZIL (PLP SSD = safe+fast) + iothread = false + } + ``` +3. **Plan and inspect before applying.** `tofu plan` MUST show an **in-place update + (disk add)** for the targeted VM — NOT a replacement. If it shows replace, stop and + reassess (do not destroy a running etcd node). +4. **Apply one node at a time** with `-target` (see §4.3 for quorum ordering), e.g. + `-target='proxmox_virtual_environment_vm.node["k3s-dev-3"]'` (adjust to the actual + resource address in this repo). A disk add is applied online by bpg, but treat the + node as going down for the etcd migration regardless. + +> **Coupling caveat — dev-3 memory drift.** dev-1/dev-2 are at `memory_mib = 24576` +> but **dev-3 is live at 16384** (drift from tofu default 24576). A tofu apply that +> touches dev-3 will also try to reset its memory to 24576. Per the Proxmox rule, +> memory changes need a **full stop/start**, not a reboot. Decide deliberately: +> either (a) let the same maintenance window fix the memory drift on dev-3 (stop → +> set → start), or (b) `-target` narrowly and accept that dev-3's plan will include +> the memory change. Do dev-3 in its own window with eyes on it. + +**Manual alternative (if not using tofu):** on the Proxmox host, +`qm set -virtio2 local-zfs:16,discard=on,cache=none` then stop/start the VM. +This drifts from tofu state and is discouraged — if done, import/reconcile into tofu +afterward. Prefer the tofu path. + +### 4.2 Optional space guarantee + +`local-zfs` is `sparse` (thin), so the etcd zvols consume only real usage. To +guarantee etcd can never be squeezed if the system disk fills, set a modest +`refreservation` on each etcd zvol (e.g. `zfs set refreservation=16G +rpool/data/vm--disk-2`). 3×16 GiB = 48 GiB reserved out of 323 GiB avail — safe. +This is a host-side ZFS tweak, not expressible in bpg; apply manually if desired. + +### 4.3 Migrate etcd onto the new disk — quorum-safe, ONE node at a time + +k3s has no flag to relocate only the etcd directory (`--data-dir` moves everything), +so the clean approach is to **mount the SSD disk at +`/var/lib/rancher/k3s/server/db`** (this directory holds `etcd/` — member data, WAL, +snapshots — and nothing else that matters). + +**Pre-flight:** confirm all 3 etcd members healthy and the cluster is quiet (no active +Longhorn rebuild). `kubectl get nodes` all Ready; check etcd member health. + +**Per node, repeat for dev-3, then dev-2, then dev-1 (leader last):** + +1. Verify 3/3 etcd members healthy BEFORE touching this node (losing one of three + keeps quorum at 2/3; never take a second node until the previous is fully back). +2. Stop k3s on the node: `systemctl stop k3s`. (Node leaves quorum; 2/3 remain.) +3. Identify the new disk (`/dev/vdc` — the third virtio disk) and format it: + `mkfs.ext4 -L k3s-etcd /dev/vdc` (ext4 or xfs; ext4 is fine and simple). +4. Preserve and move existing data: + ``` + mv /var/lib/rancher/k3s/server/db /var/lib/rancher/k3s/server/db.bak + mkdir /var/lib/rancher/k3s/server/db + mount /dev/vdc /var/lib/rancher/k3s/server/db + rsync -aHAX --numeric-ids /var/lib/rancher/k3s/server/db.bak/ /var/lib/rancher/k3s/server/db/ + ``` +5. Persist the mount in `/etc/fstab` **by UUID** (`blkid /dev/vdc`), e.g. + `UUID= /var/lib/rancher/k3s/server/db ext4 defaults,noatime 0 2`, then + `systemctl daemon-reload`. +6. Start k3s: `systemctl start k3s`. Wait until the node is `Ready` AND its etcd + member is healthy and has re-joined (member list shows 3 started, no learner + stuck). Confirm the VIP `192.168.2.50:6443` is stable. +7. Only after full recovery, remove the backup: `rm -rf + /var/lib/rancher/k3s/server/db.bak`. Then proceed to the next node. + +**Reproducibility for rebuilds:** the SSD disk + mount should also be baked into the +node **cloud-init first-boot baseline** (the same template that already provisions the +Longhorn `virtio1` disk and swap-off), so a `tofu`-rebuilt node comes up with +`/dev/vdc` formatted and mounted at the etcd path automatically. Update that template +in the same change set as the tofu disk block. + +### 4.4 Validation + +- `kubectl get nodes` — all 3 Ready; no flapping over a 30–60 min soak. +- etcd: 3 members, all started/healthy, one leader; DB size sane (~tens–hundreds MB). +- On each node, `df -h /var/lib/rancher/k3s/server/db` shows the new `/dev/vdc` mount. +- **Regression test the original failure:** with the cluster healthy, trigger a + controlled Longhorn replica rebuild (e.g. bounce ONE node after approval) and + confirm etcd stays healthy and the VIP does not flap — the whole point of the change. +- Host-side: `zpool status rpool` clean; `zfs list rpool/data` shows the 3 new + `vm-*-disk-2` zvols with small real usage. + +### 4.5 Rollback + +Per node, fully reversible while `db.bak` still exists (before step 7): +`systemctl stop k3s` → `umount .../server/db` → `rmdir` the empty mountpoint → +`mv db.bak db` → remove the fstab line → `systemctl start k3s`. To fully revert the +disk, detach `virtio2` (`qm set -delete virtio2` after stop) or `tofu` revert +the disk block, then stop/start. Because etcd is a replicated 3-member cluster, a +botched single node can also simply be reset and re-joined from the other two. + +--- + +## 5. Risks + +1. **System-disk contention / exhaustion (top risk).** `rpool` is also the Proxmox + boot/root disk (430 GiB used of 888 GiB). Mitigations: keep the etcd disks tiny + (16 GiB, thin), optional `refreservation` (§4.2), and **never** put Longhorn here + (§2). etcd's steady-state real usage is ~hundreds of MB; the marginal I/O it adds + to the SSD is negligible next to what it removes from the HDD. +2. **Consumer-SSD fsync/PLP caveat — N/A here, but assert it.** etcd correctness under + power loss depends on honest fsync. These are **Micron 5100/52x0 enterprise SATA + SSDs with power-loss protection**, so `sync=standard` + `cache=none` is both fast + and durable. (If the disks were consumer SSDs without PLP this concern would be + real — it is not, for this hardware.) +3. **Quorum during migration.** Mitigated by strict one-node-at-a-time with full + recovery gates (§4.3) and leader-last ordering. Never take a second node down + before the previous rejoins. +4. **tofu coupling with dev-3 memory drift + apply semantics (§4.1).** An apply + touching dev-3 also resets memory 16384→24576 (needs stop/start). Verify `tofu + plan` shows disk *add*, not VM *replace*, before applying. Use `-target`. +5. **Backup/restore assumptions.** k3s etcd snapshots (`/var/lib/rancher/k3s/server/db/snapshots`) + now live on the SSD too — fine, but confirm any snapshot/backup job still points at + the (unchanged) path. + +--- + +## 6. Recommendation summary + +Implement **Option A**: a 16 GiB SSD-backed `virtio2` disk per k3s VM (zvol on +`rpool/data` via `local-zfs`), etcd-only, migrated one node at a time with the +quorum-safe procedure in §4.3, modeled in tofu (+ cloud-init baseline) for +reproducibility. Do **not** move Longhorn to SSD. This is a small, reversible, +high-leverage fix targeting the exact WAL-starvation failure. + +--- + +## 7. Relationship to the deferred SSD-mini-PC migration + +This is explicitly an **interim bridge** on the current Proxmox dev host +(`192.168.2.102`), whose bulk storage is HDD. Once the k3s cluster migrates to the +planned **all-SSD mini-PCs**, every I/O path (OS, etcd, Longhorn) is already on SSD +and the HDD-contention failure mode disappears — the dedicated etcd disk becomes +unnecessary. At that point the isolation can be either dropped or carried forward +harmlessly (a dedicated etcd disk is still good hygiene). Because Option A is captured +in tofu + cloud-init, it travels with the cluster definition and can be removed in one +reviewed change when the mini-PCs land. Until then it stops the recurring incidents +at minimal cost. diff --git a/infra/ansible/roles/node_baseline/defaults/main.yml b/infra/ansible/roles/node_baseline/defaults/main.yml index 37ce87c..6fe924d 100644 --- a/infra/ansible/roles/node_baseline/defaults/main.yml +++ b/infra/ansible/roles/node_baseline/defaults/main.yml @@ -15,3 +15,10 @@ node_baseline_kernel_modules: node_baseline_longhorn_device: /dev/vdb node_baseline_longhorn_mount: /var/lib/longhorn node_baseline_longhorn_fs_label: longhorn +# Dedicated SSD etcd disk (tofu virtio2 -> /dev/vdc), mounted at the k3s etcd data dir +# so etcd WAL fsync is isolated from Longhorn rebuild I/O on the HDD data disk. +# See docs/plans/ssd-etcd-isolation.md. FS label keeps the fstab entry stable across +# device renames (matches the Longhorn LABEL= approach). +node_baseline_etcd_device: /dev/vdc +node_baseline_etcd_mount: /var/lib/rancher/k3s/server/db +node_baseline_etcd_fs_label: k3s-etcd diff --git a/infra/ansible/roles/node_baseline/tasks/main.yml b/infra/ansible/roles/node_baseline/tasks/main.yml index bf84e9d..c8cf0b1 100644 --- a/infra/ansible/roles/node_baseline/tasks/main.yml +++ b/infra/ansible/roles/node_baseline/tasks/main.yml @@ -72,6 +72,31 @@ opts: "defaults,nofail,x-systemd.device-timeout=5s" state: mounted +# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and +# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with +# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats +# a disk that already carries a filesystem, so re-running against an already-migrated +# node leaves the etcd data intact. On EXISTING nodes the one-time data migration +# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this +# role would mount over a populated directory. +- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present) + community.general.filesystem: + fstype: ext4 + dev: "{{ node_baseline_etcd_device }}" + opts: "-L {{ node_baseline_etcd_fs_label }}" + +- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir + ansible.posix.mount: + path: "{{ node_baseline_etcd_mount }}" + src: "LABEL={{ node_baseline_etcd_fs_label }}" + fstype: ext4 + # noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a + # missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD. + opts: "defaults,noatime" + dump: "0" + passno: "2" + state: mounted + - name: Disable swap now ansible.builtin.command: swapoff -a changed_when: false diff --git a/infra/tofu/main.tf b/infra/tofu/main.tf index cfa2f5d..1b502bb 100644 --- a/infra/tofu/main.tf +++ b/infra/tofu/main.tf @@ -41,13 +41,17 @@ locals { # -> Storage -> local -> Content -> check "Import"), else the disk import fails with # "wrong type 'iso' - needs to be 'images' or 'import'". # -# Resource type is the SHORT name `proxmox_download_file` (bpg/proxmox ADR-007). The old -# `proxmox_virtual_environment_download_file` is deprecated ("Use `proxmox_download_file` -# instead. This resource will be removed in v1.0.") — both are the same dual-registered -# Framework resource, so the schema is identical. The `moved` block below migrates existing -# state in place (provider implements MoveState / migration.PrefixMoveState), so no -# re-download of the image and no change to the VMs. -resource "proxmox_download_file" "debian_cloud_image" { +# The Debian cloud image is a bpg `download_file` resource. This intentionally uses the +# LONG (deprecated) type name `proxmox_virtual_environment_download_file`: the R2 backend +# state still holds this resource under the long name, and bpg >= 0.111 dropped the +# cross-type `moved{}` linkage to the short `proxmox_download_file` (ADR-007) — a `moved` +# block between the two now fails plan with "Resource type mismatch" (this broke plan on +# main entirely). Keeping the long name makes config match state, so the plan is clean and +# the ~450 MB image is NOT re-downloaded. Migrating to the short name is a SEPARATE, +# deliberate step (`tofu state mv proxmox_virtual_environment_download_file.debian_cloud_image +# proxmox_download_file.debian_cloud_image`, then rename here) — not bundled with the +# etcd-disk change so this bounce stays focused and low-risk. +resource "proxmox_virtual_environment_download_file" "debian_cloud_image" { content_type = "import" datastore_id = var.image_datastore node_name = var.proxmox_node @@ -56,15 +60,6 @@ resource "proxmox_download_file" "debian_cloud_image" { overwrite = false } -# State migration for the deprecated -> short resource-type rename (ADR-007). OpenTofu -# 1.9 (>= 1.8 required) drives the provider's MoveState RPC: the download_file state is -# transferred in place, keeping the same id (datastore_id:content_type/file_name), so the -# already-downloaded Debian image is NOT re-downloaded and the VMs' import_from is unchanged. -moved { - from = proxmox_virtual_environment_download_file.debian_cloud_image - to = proxmox_download_file.debian_cloud_image -} - # --- cloud-init: NATIVE Proxmox cloud-init only (API-driven) ------------------ # The rich first-boot baseline (packages, kernel modules, Longhorn disk, swap-off) # is NOT delivered as a custom snippet any more — custom snippets require SSH/SFTP @@ -118,7 +113,7 @@ resource "proxmox_virtual_environment_vm" "node" { # fs_setup/mount for /var/lib/longhorn already expects. disk { datastore_id = var.datastore - import_from = proxmox_download_file.debian_cloud_image.id + import_from = proxmox_virtual_environment_download_file.debian_cloud_image.id interface = "virtio0" size = var.os_disk_gib discard = "on" @@ -131,6 +126,22 @@ resource "proxmox_virtual_environment_vm" "node" { discard = "on" } + # Dedicated SSD etcd disk (virtio2 -> /dev/vdc). Isolates etcd WAL fsync from the HDD + # `Storage` pool where the OS + Longhorn disks live — the node_baseline role formats + # it and mounts it at /var/lib/rancher/k3s/server/db before k3s starts. On the SSD + # `local-zfs` (rpool/data) datastore: an enterprise Micron SATA SSD mirror WITH + # power-loss protection, so cache=none (honor guest fsync straight through to the ZFS + # ZIL) is both durable and fast. Kept last (ascending interface) so bpg won't reorder + # the existing virtio0/virtio1 disks. See docs/plans/ssd-etcd-isolation.md. + disk { + datastore_id = var.etcd_ssd_datastore + interface = "virtio2" + size = var.etcd_disk_gib + discard = "on" + cache = "none" + iothread = false + } + network_device { bridge = var.bridge } diff --git a/infra/tofu/variables.tf b/infra/tofu/variables.tf index 471e0a5..88dbfe4 100644 --- a/infra/tofu/variables.tf +++ b/infra/tofu/variables.tf @@ -100,6 +100,24 @@ variable "data_disk_gib" { default = 128 } +# --- dedicated SSD etcd disk (interim etcd-on-SSD isolation, docs/plans/ssd-etcd-isolation.md) -- +# etcd WAL fsync was being starved by Longhorn replica-rebuild I/O on the shared HDD +# `Storage` pool (OS + Longhorn disks). A tiny dedicated SSD disk per node moves the +# etcd data dir (/var/lib/rancher/k3s/server/db, mounted by node_baseline) off the HDD. +# Interim bridge until the all-SSD mini-PCs land; Longhorn stays on HDD (does not fit). + +variable "etcd_ssd_datastore" { + description = "SSD-backed datastore for the dedicated etcd disk (VERIFIED: local-zfs = rpool/data, ZFS mirror on Micron 5100 SATA SSDs w/ power-loss protection)." + type = string + default = "local-zfs" +} + +variable "etcd_disk_gib" { + description = "Dedicated SSD etcd disk per node, GiB (etcd DB is ~83 MB; 16 GiB is generous headroom)." + type = number + default = 16 +} + variable "cpu_type" { description = "Proxmox vCPU type. `host` passes through the physical CPU (nested-virt for KubeVirt, Phase 5)." type = string