# SSD-backed etcd isolation for the k3s dev cluster > ## ✅ COMPLETED / historical — executed 2026-08-30, see git log > > **This proposal was approved and carried out.** Each node now has a dedicated `virtio2` SSD etcd > disk (`/dev/vdc`, `local-zfs`) hosting the etcd data dir. The **Status: PROPOSAL** line below and > the sizing/RAM figures are the original investigation record — retained for the reasoning, not as a > live to-do. Current state is tracked in `infra/PHASE1-STATUS.md` and the 2026-08-30 DECISIONS-LOG entry. **Status:** PROPOSAL — awaiting user approval. No changes have been made. **Author:** investigation agent (read-only), 2026-08-29 **Problem:** Longhorn replica-rebuild I/O storms on the HDD-backed VM disks starve etcd's WAL fsync, crash-looping k3s and flapping the kube-vip VIP (`192.168.2.50:6443`). Even `concurrent-replica-rebuild-per-node-limit=1` tips etcd. **Question:** Can the Proxmox host's ~1 TB system SSD (`rpool`) host dedicated SSD-backed storage for etcd (and/or Longhorn) to end these incidents? **Verdict: YES for etcd (strongly recommended). NO for Longhorn.** --- ## 1. Measured facts (all collected read-only on 2026-08-29) ### 1.1 `rpool` — the system/boot SSD pool | Property | Value | |---|---| | Pool | `rpool`, **mirror** (`mirror-0`) of 2 devices | | Devices | `ata-MTFDDAK960TDS_...` × 2 — **Micron 5100/52x0 960 GB, 2.5" SATA** | | Class | **Enterprise SATA SSD with power-loss protection (PLP)** — safe for sync/fsync | | Health | `Percent_Lifetime_Remain` = **97%** (3% used), 13 904 power-on hours, 0 reallocated sectors, ~96 TB written. Excellent. | | Pool size / alloc / **free** | 888 G / 538 G / **350 G free** (`zpool list`, 60% cap) | | ZFS `AVAIL` | **~323 GiB** available to any new dataset/zvol | | System root | `rpool/ROOT/pve-1` = **430 G used** (Proxmox host `/`) | | Proxmox storage on it | `local-zfs` → pool `rpool/data`, `content images,rootdir`, **`sparse 1`** (thin), ~338 GiB available per `pvesm status` | | Pool settings | `compression on`, **`sync standard`** (inherited), `dedup off`, recordsize 128K | `local-zfs` already advertises `images,rootdir` content, so **Proxmox can place VM disks on the SSD today with no storage-config change.** ### 1.2 Current VM disk backing — confirms the HDD assumption VMs (bpg/OpenTofu-provisioned): **dev-1 = vmid 105, dev-2 = 102, dev-3 = 101**. Each has two disks, **both on the `Storage` datastore**: - `Storage` = ZFS pool on **2× WDC WUH721818ALE6L4 18 TB HDD** (`sdc`/`sdd`, `ROTA=1`), mirror, 16.4 T. - Per node: `virtio0 = vm--disk-0` (64 G, OS/root) and `virtio1 = vm--disk-1` (128 G, Longhorn data). `volblocksize 16K`. So **etcd, the OS, and Longhorn all currently sit on spinning rust.** That is the root cause: etcd WAL fsync competes with Longhorn rebuild I/O on the same HDD spindles. (`Storage2` = 2× Seagate ST8000VN004 8 TB HDD is a separate HDD pool, not involved.) ### 1.3 Size of the hot data - **etcd DB: ~82.6 MB.** Measured via apiserver metric `apiserver_storage_size_bytes{storage_cluster_id="etcd-0"} = 8.264e7`. A 4-day-old cluster running ~18 services. Even allowing generous WAL + growth headroom, **etcd fits in single-digit GB with room to spare — trivially ×3.** - **Longhorn: ~22 GB actual data per node right now**, but the data disk is **128 GiB** and Longhorn reports `storageScheduled ≈ 152 GiB` (thin-overcommitted) vs `storageMaximum ≈ 125 GiB` per node. 18 PVCs, `numberOfReplicas=3`. Provisioned total ≈ 152 GiB. Longhorn is the **churn source we want to isolate etcd from**, and it can grow toward 128 GiB/node. ### 1.4 Sizing conclusion | Candidate | Per-node footprint | ×3 on rpool | Fits in 323 GiB avail? | |---|---|---|---| | **etcd** | ~83 MB now; size a 16 GiB disk | ~48 GiB reserved (thin) | **Yes, easily** | | **Longhorn** | ~22 GB now, up to 128 GiB provisioned | up to **~384 GiB** | **No — would fill/threaten the boot disk** | --- ## 2. Viability verdict **etcd on SSD: viable and high-leverage.** etcd is ~83 MB; three 16 GiB SSD-backed disks cost ~48 GiB of thin allocation (actual ~250 MB) against 323 GiB free on an enterprise, PLP-equipped, 97%-healthy SSD mirror. This directly removes the WAL fsync path from the HDD contention domain — the smallest-footprint fix for the exact failure mode. **This is the recommendation.** **Longhorn on SSD: not viable here.** Its provisioned footprint (~384 GiB across 3 replicas at full size) does not fit alongside the 430 GiB Proxmox root on a 888 GiB pool, and co-locating heavy Longhorn rebuild I/O on the boot SSD reintroduces contention with the host itself. Keep Longhorn on HDD (`Storage`). Rejected. --- ## 3. Options evaluated - **Option A — dedicated small SSD disk per node, etcd-only (RECOMMENDED).** Add a ~16 GiB SSD-backed virtual disk (zvol on `rpool/data` via `local-zfs`) to each of the 3 VMs, mount it at the k3s etcd directory (`/var/lib/rancher/k3s/server/db`), migrate etcd data onto it one node at a time. Isolates WAL fsync from HDD Longhorn churn. Minimal space, minimal blast radius. - **Option B — Longhorn replicas/disk on SSD.** Rejected (§2): does not fit and re-couples host I/O. - **Option C — do nothing; wait for the SSD mini-PCs.** Zero risk, zero work, but leaves the cluster fragile to every node bounce until the mini-PC migration lands (deferred, no committed date). Option A is a cheap, reversible bridge that Option C eventually supersedes (§7). **Recommendation: Option A now, as an interim fix, carried by tofu.** --- ## 4. Recommended implementation — Option A, step-by-step ### 4.1 Model it in tofu (preferred over manual) The VMs are bpg/OpenTofu-managed (`infra/tofu/`). Disks are declared as `disk` blocks in `main.tf`; both current disks (`virtio0`, `virtio1`) point at `var.datastore` (`Storage`). Add a **third disk on the SSD datastore**: 1. **New variables** in `infra/tofu/variables.tf`: ```hcl variable "etcd_ssd_datastore" { description = "SSD-backed datastore for the dedicated etcd disk (VERIFIED: local-zfs = rpool/data, ZFS mirror on Micron 5100 SATA SSDs w/ PLP)." type = string default = "local-zfs" } variable "etcd_disk_gib" { description = "Dedicated SSD etcd disk per node, GiB (etcd DB is ~83 MB; 16 GiB is generous headroom)." type = number default = 16 } ``` 2. **New disk block** in the VM resource in `main.tf`, appended AFTER the existing `virtio1` block (keep interface order ascending so bpg does not reorder): ```hcl # Dedicated SSD etcd disk (virtio2 -> /dev/vdc). Isolates etcd WAL fsync from # the HDD Storage pool where the OS + Longhorn disks live. disk { datastore_id = var.etcd_ssd_datastore interface = "virtio2" size = var.etcd_disk_gib discard = "on" cache = "none" # honor guest fsync through to ZFS ZIL (PLP SSD = safe+fast) iothread = false } ``` 3. **Plan and inspect before applying.** `tofu plan` MUST show an **in-place update (disk add)** for the targeted VM — NOT a replacement. If it shows replace, stop and reassess (do not destroy a running etcd node). 4. **Apply one node at a time** with `-target` (see §4.3 for quorum ordering), e.g. `-target='proxmox_virtual_environment_vm.node["k3s-dev-3"]'` (adjust to the actual resource address in this repo). A disk add is applied online by bpg, but treat the node as going down for the etcd migration regardless. > **Coupling caveat — dev-3 memory drift.** dev-1/dev-2 are at `memory_mib = 24576` > but **dev-3 is live at 16384** (drift from tofu default 24576). A tofu apply that > touches dev-3 will also try to reset its memory to 24576. Per the Proxmox rule, > memory changes need a **full stop/start**, not a reboot. Decide deliberately: > either (a) let the same maintenance window fix the memory drift on dev-3 (stop → > set → start), or (b) `-target` narrowly and accept that dev-3's plan will include > the memory change. Do dev-3 in its own window with eyes on it. **Manual alternative (if not using tofu):** on the Proxmox host, `qm set -virtio2 local-zfs:16,discard=on,cache=none` then stop/start the VM. This drifts from tofu state and is discouraged — if done, import/reconcile into tofu afterward. Prefer the tofu path. ### 4.2 Optional space guarantee `local-zfs` is `sparse` (thin), so the etcd zvols consume only real usage. To guarantee etcd can never be squeezed if the system disk fills, set a modest `refreservation` on each etcd zvol (e.g. `zfs set refreservation=16G rpool/data/vm--disk-2`). 3×16 GiB = 48 GiB reserved out of 323 GiB avail — safe. This is a host-side ZFS tweak, not expressible in bpg; apply manually if desired. ### 4.3 Migrate etcd onto the new disk — quorum-safe, ONE node at a time k3s has no flag to relocate only the etcd directory (`--data-dir` moves everything), so the clean approach is to **mount the SSD disk at `/var/lib/rancher/k3s/server/db`** (this directory holds `etcd/` — member data, WAL, snapshots — and nothing else that matters). **Pre-flight:** confirm all 3 etcd members healthy and the cluster is quiet (no active Longhorn rebuild). `kubectl get nodes` all Ready; check etcd member health. **Per node, repeat for dev-3, then dev-2, then dev-1 (leader last):** 1. Verify 3/3 etcd members healthy BEFORE touching this node (losing one of three keeps quorum at 2/3; never take a second node until the previous is fully back). 2. Stop k3s on the node: `systemctl stop k3s`. (Node leaves quorum; 2/3 remain.) 3. Identify the new disk (`/dev/vdc` — the third virtio disk) and format it: `mkfs.ext4 -L k3s-etcd /dev/vdc` (ext4 or xfs; ext4 is fine and simple). 4. Preserve and move existing data: ``` mv /var/lib/rancher/k3s/server/db /var/lib/rancher/k3s/server/db.bak mkdir /var/lib/rancher/k3s/server/db mount /dev/vdc /var/lib/rancher/k3s/server/db rsync -aHAX --numeric-ids /var/lib/rancher/k3s/server/db.bak/ /var/lib/rancher/k3s/server/db/ ``` 5. Persist the mount in `/etc/fstab` **by UUID** (`blkid /dev/vdc`), e.g. `UUID= /var/lib/rancher/k3s/server/db ext4 defaults,noatime 0 2`, then `systemctl daemon-reload`. 6. Start k3s: `systemctl start k3s`. Wait until the node is `Ready` AND its etcd member is healthy and has re-joined (member list shows 3 started, no learner stuck). Confirm the VIP `192.168.2.50:6443` is stable. 7. Only after full recovery, remove the backup: `rm -rf /var/lib/rancher/k3s/server/db.bak`. Then proceed to the next node. **Reproducibility for rebuilds:** the SSD disk + mount should also be baked into the node **cloud-init first-boot baseline** (the same template that already provisions the Longhorn `virtio1` disk and swap-off), so a `tofu`-rebuilt node comes up with `/dev/vdc` formatted and mounted at the etcd path automatically. Update that template in the same change set as the tofu disk block. ### 4.4 Validation - `kubectl get nodes` — all 3 Ready; no flapping over a 30–60 min soak. - etcd: 3 members, all started/healthy, one leader; DB size sane (~tens–hundreds MB). - On each node, `df -h /var/lib/rancher/k3s/server/db` shows the new `/dev/vdc` mount. - **Regression test the original failure:** with the cluster healthy, trigger a controlled Longhorn replica rebuild (e.g. bounce ONE node after approval) and confirm etcd stays healthy and the VIP does not flap — the whole point of the change. - Host-side: `zpool status rpool` clean; `zfs list rpool/data` shows the 3 new `vm-*-disk-2` zvols with small real usage. ### 4.5 Rollback Per node, fully reversible while `db.bak` still exists (before step 7): `systemctl stop k3s` → `umount .../server/db` → `rmdir` the empty mountpoint → `mv db.bak db` → remove the fstab line → `systemctl start k3s`. To fully revert the disk, detach `virtio2` (`qm set -delete virtio2` after stop) or `tofu` revert the disk block, then stop/start. Because etcd is a replicated 3-member cluster, a botched single node can also simply be reset and re-joined from the other two. --- ## 5. Risks 1. **System-disk contention / exhaustion (top risk).** `rpool` is also the Proxmox boot/root disk (430 GiB used of 888 GiB). Mitigations: keep the etcd disks tiny (16 GiB, thin), optional `refreservation` (§4.2), and **never** put Longhorn here (§2). etcd's steady-state real usage is ~hundreds of MB; the marginal I/O it adds to the SSD is negligible next to what it removes from the HDD. 2. **Consumer-SSD fsync/PLP caveat — N/A here, but assert it.** etcd correctness under power loss depends on honest fsync. These are **Micron 5100/52x0 enterprise SATA SSDs with power-loss protection**, so `sync=standard` + `cache=none` is both fast and durable. (If the disks were consumer SSDs without PLP this concern would be real — it is not, for this hardware.) 3. **Quorum during migration.** Mitigated by strict one-node-at-a-time with full recovery gates (§4.3) and leader-last ordering. Never take a second node down before the previous rejoins. 4. **tofu coupling with dev-3 memory drift + apply semantics (§4.1).** An apply touching dev-3 also resets memory 16384→24576 (needs stop/start). Verify `tofu plan` shows disk *add*, not VM *replace*, before applying. Use `-target`. 5. **Backup/restore assumptions.** k3s etcd snapshots (`/var/lib/rancher/k3s/server/db/snapshots`) now live on the SSD too — fine, but confirm any snapshot/backup job still points at the (unchanged) path. --- ## 6. Recommendation summary Implement **Option A**: a 16 GiB SSD-backed `virtio2` disk per k3s VM (zvol on `rpool/data` via `local-zfs`), etcd-only, migrated one node at a time with the quorum-safe procedure in §4.3, modeled in tofu (+ cloud-init baseline) for reproducibility. Do **not** move Longhorn to SSD. This is a small, reversible, high-leverage fix targeting the exact WAL-starvation failure. --- ## 7. Relationship to the deferred SSD-mini-PC migration This is explicitly an **interim bridge** on the current Proxmox dev host (`192.168.2.102`), whose bulk storage is HDD. Once the k3s cluster migrates to the planned **all-SSD mini-PCs**, every I/O path (OS, etcd, Longhorn) is already on SSD and the HDD-contention failure mode disappears — the dedicated etcd disk becomes unnecessary. At that point the isolation can be either dropped or carried forward harmlessly (a dedicated etcd disk is still good hygiene). Because Option A is captured in tofu + cloud-init, it travels with the cluster definition and can be removed in one reviewed change when the mini-PCs land. Until then it stops the recurring incidents at minimal cost.