home-network-k3s-impl/docs/plans/ssd-etcd-isolation.md

SSD-backed etcd isolation for the k3s dev cluster

✅ COMPLETED / historical — executed 2026-08-30, see git log

This proposal was approved and carried out. Each node now has a dedicated virtio2 SSD etcd disk (/dev/vdc, local-zfs) hosting the etcd data dir. The Status: PROPOSAL line below and the sizing/RAM figures are the original investigation record — retained for the reasoning, not as a live to-do. Current state is tracked in infra/PHASE1-STATUS.md and the 2026-08-30 DECISIONS-LOG entry.

Status: PROPOSAL — awaiting user approval. No changes have been made. Author: investigation agent (read-only), 2026-08-29 Problem: Longhorn replica-rebuild I/O storms on the HDD-backed VM disks starve etcd's WAL fsync, crash-looping k3s and flapping the kube-vip VIP (192.168.2.50:6443). Even concurrent-replica-rebuild-per-node-limit=1 tips etcd. Question: Can the Proxmox host's ~1 TB system SSD (rpool) host dedicated SSD-backed storage for etcd (and/or Longhorn) to end these incidents?

Verdict: YES for etcd (strongly recommended). NO for Longhorn.

---

1. Measured facts (all collected read-only on 2026-08-29)

1.1 rpool — the system/boot SSD pool

Property	Value
Pool	rpool, mirror (mirror-0) of 2 devices
Devices	ata-MTFDDAK960TDS_... × 2 — Micron 5100/52x0 960 GB, 2.5" SATA
Class	Enterprise SATA SSD with power-loss protection (PLP) — safe for sync/fsync
Health	Percent_Lifetime_Remain = 97% (3% used), 13 904 power-on hours, 0 reallocated sectors, ~96 TB written. Excellent.
Pool size / alloc / free	888 G / 538 G / 350 G free (zpool list, 60% cap)
ZFS AVAIL	~323 GiB available to any new dataset/zvol
System root	rpool/ROOT/pve-1 = 430 G used (Proxmox host /)
Proxmox storage on it	local-zfs → pool rpool/data, content images,rootdir, sparse 1 (thin), ~338 GiB available per pvesm status
Pool settings	compression on, sync standard (inherited), dedup off, recordsize 128K

local-zfs already advertises images,rootdir content, so Proxmox can place VM disks on the SSD today with no storage-config change.

1.2 Current VM disk backing — confirms the HDD assumption

VMs (bpg/OpenTofu-provisioned): dev-1 = vmid 105, dev-2 = 102, dev-3 = 101. Each has two disks, both on the Storage datastore:

• Storage = ZFS pool on 2× WDC WUH721818ALE6L4 18 TB HDD (sdc/sdd, ROTA=1), mirror, 16.4 T.
• Per node: virtio0 = vm-<id>-disk-0 (64 G, OS/root) and virtio1 = vm-<id>-disk-1 (128 G, Longhorn data). volblocksize 16K.

So etcd, the OS, and Longhorn all currently sit on spinning rust. That is the root cause: etcd WAL fsync competes with Longhorn rebuild I/O on the same HDD spindles.

(Storage2 = 2× Seagate ST8000VN004 8 TB HDD is a separate HDD pool, not involved.)

1.3 Size of the hot data

• etcd DB: ~82.6 MB. Measured via apiserver metric apiserver_storage_size_bytes{storage_cluster_id="etcd-0"} = 8.264e7. A 4-day-old cluster running ~18 services. Even allowing generous WAL + growth headroom, etcd fits in single-digit GB with room to spare — trivially ×3.
• Longhorn: ~22 GB actual data per node right now, but the data disk is 128 GiB and Longhorn reports storageScheduled ≈ 152 GiB (thin-overcommitted) vs storageMaximum ≈ 125 GiB per node. 18 PVCs, numberOfReplicas=3. Provisioned total ≈ 152 GiB. Longhorn is the churn source we want to isolate etcd from, and it can grow toward 128 GiB/node.

1.4 Sizing conclusion

Candidate	Per-node footprint	×3 on rpool	Fits in 323 GiB avail?
etcd	~83 MB now; size a 16 GiB disk	~48 GiB reserved (thin)	Yes, easily
Longhorn	~22 GB now, up to 128 GiB provisioned	up to ~384 GiB	No — would fill/threaten the boot disk

---

2. Viability verdict

etcd on SSD: viable and high-leverage. etcd is ~83 MB; three 16 GiB SSD-backed disks cost ~48 GiB of thin allocation (actual ~250 MB) against 323 GiB free on an enterprise, PLP-equipped, 97%-healthy SSD mirror. This directly removes the WAL fsync path from the HDD contention domain — the smallest-footprint fix for the exact failure mode. This is the recommendation.

Longhorn on SSD: not viable here. Its provisioned footprint (~384 GiB across 3 replicas at full size) does not fit alongside the 430 GiB Proxmox root on a 888 GiB pool, and co-locating heavy Longhorn rebuild I/O on the boot SSD reintroduces contention with the host itself. Keep Longhorn on HDD (Storage). Rejected.

---

3. Options evaluated

• Option A — dedicated small SSD disk per node, etcd-only (RECOMMENDED). Add a ~16 GiB SSD-backed virtual disk (zvol on rpool/data via local-zfs) to each of the 3 VMs, mount it at the k3s etcd directory (/var/lib/rancher/k3s/server/db), migrate etcd data onto it one node at a time. Isolates WAL fsync from HDD Longhorn churn. Minimal space, minimal blast radius.
• Option B — Longhorn replicas/disk on SSD. Rejected (§2): does not fit and re-couples host I/O.
• Option C — do nothing; wait for the SSD mini-PCs. Zero risk, zero work, but leaves the cluster fragile to every node bounce until the mini-PC migration lands (deferred, no committed date). Option A is a cheap, reversible bridge that Option C eventually supersedes (§7).

Recommendation: Option A now, as an interim fix, carried by tofu.

---

4. Recommended implementation — Option A, step-by-step

4.1 Model it in tofu (preferred over manual)

The VMs are bpg/OpenTofu-managed (infra/tofu/). Disks are declared as disk blocks in main.tf; both current disks (virtio0, virtio1) point at var.datastore (Storage). Add a third disk on the SSD datastore:

1. New variables in infra/tofu/variables.tf:

   variable "etcd_ssd_datastore" {
     description = "SSD-backed datastore for the dedicated etcd disk (VERIFIED: local-zfs = rpool/data, ZFS mirror on Micron 5100 SATA SSDs w/ PLP)."
     type        = string
     default     = "local-zfs"
   }
   variable "etcd_disk_gib" {
     description = "Dedicated SSD etcd disk per node, GiB (etcd DB is ~83 MB; 16 GiB is generous headroom)."
     type        = number
     default     = 16
   }
   

2. New disk block in the VM resource in main.tf, appended AFTER the existing virtio1 block (keep interface order ascending so bpg does not reorder):

   # Dedicated SSD etcd disk (virtio2 -> /dev/vdc). Isolates etcd WAL fsync from
   # the HDD Storage pool where the OS + Longhorn disks live.
   disk {
     datastore_id = var.etcd_ssd_datastore
     interface    = "virtio2"
     size         = var.etcd_disk_gib
     discard      = "on"
     cache        = "none"   # honor guest fsync through to ZFS ZIL (PLP SSD = safe+fast)
     iothread     = false
   }
   

3. Plan and inspect before applying. tofu plan MUST show an in-place update (disk add) for the targeted VM — NOT a replacement. If it shows replace, stop and reassess (do not destroy a running etcd node).
4. Apply one node at a time with -target (see §4.3 for quorum ordering), e.g. -target='proxmox_virtual_environment_vm.node["k3s-dev-3"]' (adjust to the actual resource address in this repo). A disk add is applied online by bpg, but treat the node as going down for the etcd migration regardless.

Coupling caveat — dev-3 memory drift. dev-1/dev-2 are at memory_mib = 24576 but dev-3 is live at 16384 (drift from tofu default 24576). A tofu apply that touches dev-3 will also try to reset its memory to 24576. Per the Proxmox rule, memory changes need a full stop/start, not a reboot. Decide deliberately: either (a) let the same maintenance window fix the memory drift on dev-3 (stop → set → start), or (b) -target narrowly and accept that dev-3's plan will include the memory change. Do dev-3 in its own window with eyes on it.

Manual alternative (if not using tofu): on the Proxmox host, qm set <vmid> -virtio2 local-zfs:16,discard=on,cache=none then stop/start the VM. This drifts from tofu state and is discouraged — if done, import/reconcile into tofu afterward. Prefer the tofu path.

4.2 Optional space guarantee

local-zfs is sparse (thin), so the etcd zvols consume only real usage. To guarantee etcd can never be squeezed if the system disk fills, set a modest refreservation on each etcd zvol (e.g. zfs set refreservation=16G rpool/data/vm-<id>-disk-2). 3×16 GiB = 48 GiB reserved out of 323 GiB avail — safe. This is a host-side ZFS tweak, not expressible in bpg; apply manually if desired.

4.3 Migrate etcd onto the new disk — quorum-safe, ONE node at a time

k3s has no flag to relocate only the etcd directory (--data-dir moves everything), so the clean approach is to mount the SSD disk at /var/lib/rancher/k3s/server/db (this directory holds etcd/ — member data, WAL, snapshots — and nothing else that matters).

Pre-flight: confirm all 3 etcd members healthy and the cluster is quiet (no active Longhorn rebuild). kubectl get nodes all Ready; check etcd member health.

Per node, repeat for dev-3, then dev-2, then dev-1 (leader last):

1. Verify 3/3 etcd members healthy BEFORE touching this node (losing one of three keeps quorum at 2/3; never take a second node until the previous is fully back).
2. Stop k3s on the node: systemctl stop k3s. (Node leaves quorum; 2/3 remain.)
3. Identify the new disk (/dev/vdc — the third virtio disk) and format it: mkfs.ext4 -L k3s-etcd /dev/vdc (ext4 or xfs; ext4 is fine and simple).
4. Preserve and move existing data:

   mv /var/lib/rancher/k3s/server/db /var/lib/rancher/k3s/server/db.bak
   mkdir /var/lib/rancher/k3s/server/db
   mount /dev/vdc /var/lib/rancher/k3s/server/db
   rsync -aHAX --numeric-ids /var/lib/rancher/k3s/server/db.bak/ /var/lib/rancher/k3s/server/db/
   

5. Persist the mount in /etc/fstab by UUID (blkid /dev/vdc), e.g. UUID=<uuid> /var/lib/rancher/k3s/server/db ext4 defaults,noatime 0 2, then systemctl daemon-reload.
6. Start k3s: systemctl start k3s. Wait until the node is Ready AND its etcd member is healthy and has re-joined (member list shows 3 started, no learner stuck). Confirm the VIP 192.168.2.50:6443 is stable.
7. Only after full recovery, remove the backup: rm -rf /var/lib/rancher/k3s/server/db.bak. Then proceed to the next node.

Reproducibility for rebuilds: the SSD disk + mount should also be baked into the node cloud-init first-boot baseline (the same template that already provisions the Longhorn virtio1 disk and swap-off), so a tofu-rebuilt node comes up with /dev/vdc formatted and mounted at the etcd path automatically. Update that template in the same change set as the tofu disk block.

4.4 Validation

• kubectl get nodes — all 3 Ready; no flapping over a 30–60 min soak.
• etcd: 3 members, all started/healthy, one leader; DB size sane (~tens–hundreds MB).
• On each node, df -h /var/lib/rancher/k3s/server/db shows the new /dev/vdc mount.
• Regression test the original failure: with the cluster healthy, trigger a controlled Longhorn replica rebuild (e.g. bounce ONE node after approval) and confirm etcd stays healthy and the VIP does not flap — the whole point of the change.
• Host-side: zpool status rpool clean; zfs list rpool/data shows the 3 new vm-*-disk-2 zvols with small real usage.

4.5 Rollback

Per node, fully reversible while db.bak still exists (before step 7): systemctl stop k3s → umount .../server/db → rmdir the empty mountpoint → mv db.bak db → remove the fstab line → systemctl start k3s. To fully revert the disk, detach virtio2 (qm set <vmid> -delete virtio2 after stop) or tofu revert the disk block, then stop/start. Because etcd is a replicated 3-member cluster, a botched single node can also simply be reset and re-joined from the other two.

---

5. Risks

1. System-disk contention / exhaustion (top risk). rpool is also the Proxmox boot/root disk (430 GiB used of 888 GiB). Mitigations: keep the etcd disks tiny (16 GiB, thin), optional refreservation (§4.2), and never put Longhorn here (§2). etcd's steady-state real usage is ~hundreds of MB; the marginal I/O it adds to the SSD is negligible next to what it removes from the HDD.
2. Consumer-SSD fsync/PLP caveat — N/A here, but assert it. etcd correctness under power loss depends on honest fsync. These are Micron 5100/52x0 enterprise SATA SSDs with power-loss protection, so sync=standard + cache=none is both fast and durable. (If the disks were consumer SSDs without PLP this concern would be real — it is not, for this hardware.)
3. Quorum during migration. Mitigated by strict one-node-at-a-time with full recovery gates (§4.3) and leader-last ordering. Never take a second node down before the previous rejoins.
4. tofu coupling with dev-3 memory drift + apply semantics (§4.1). An apply touching dev-3 also resets memory 16384→24576 (needs stop/start). Verify tofu plan shows disk add, not VM replace, before applying. Use -target.
5. Backup/restore assumptions. k3s etcd snapshots (/var/lib/rancher/k3s/server/db/snapshots) now live on the SSD too — fine, but confirm any snapshot/backup job still points at the (unchanged) path.

---

6. Recommendation summary

Implement Option A: a 16 GiB SSD-backed virtio2 disk per k3s VM (zvol on rpool/data via local-zfs), etcd-only, migrated one node at a time with the quorum-safe procedure in §4.3, modeled in tofu (+ cloud-init baseline) for reproducibility. Do not move Longhorn to SSD. This is a small, reversible, high-leverage fix targeting the exact WAL-starvation failure.

---

7. Relationship to the deferred SSD-mini-PC migration

This is explicitly an interim bridge on the current Proxmox dev host (192.168.2.102), whose bulk storage is HDD. Once the k3s cluster migrates to the planned all-SSD mini-PCs, every I/O path (OS, etcd, Longhorn) is already on SSD and the HDD-contention failure mode disappears — the dedicated etcd disk becomes unnecessary. At that point the isolation can be either dropped or carried forward harmlessly (a dedicated etcd disk is still good hygiene). Because Option A is captured in tofu + cloud-init, it travels with the cluster definition and can be removed in one reviewed change when the mini-PCs land. Until then it stops the recurring incidents at minimal cost.