home-network-k3s-impl/docs/plans/ssd-etcd-isolation.md

# SSD-backed etcd isolation for the k3s dev cluster

> ## ✅ COMPLETED / historical — executed 2026-08-30, see git log
>
> **This proposal was approved and carried out.** Each node now has a dedicated `virtio2` SSD etcd
> disk (`/dev/vdc`, `local-zfs`) hosting the etcd data dir. The **Status: PROPOSAL** line below and
> the sizing/RAM figures are the original investigation record — retained for the reasoning, not as a
> live to-do. Current state is tracked in `infra/PHASE1-STATUS.md` and the 2026-08-30 DECISIONS-LOG entry.

**Status:** PROPOSAL — awaiting user approval. No changes have been made.
**Author:** investigation agent (read-only), 2026-08-29
**Problem:** Longhorn replica-rebuild I/O storms on the HDD-backed VM disks starve
etcd's WAL fsync, crash-looping k3s and flapping the kube-vip VIP
(`192.168.2.50:6443`). Even `concurrent-replica-rebuild-per-node-limit=1` tips etcd.
**Question:** Can the Proxmox host's ~1 TB system SSD (`rpool`) host dedicated
SSD-backed storage for etcd (and/or Longhorn) to end these incidents?

**Verdict: YES for etcd (strongly recommended). NO for Longhorn.**

---

## 1. Measured facts (all collected read-only on 2026-08-29)

### 1.1 `rpool` — the system/boot SSD pool

| Property | Value |
|---|---|
| Pool | `rpool`, **mirror** (`mirror-0`) of 2 devices |
| Devices | `ata-MTFDDAK960TDS_...` × 2 — **Micron 5100/52x0 960 GB, 2.5" SATA** |
| Class | **Enterprise SATA SSD with power-loss protection (PLP)** — safe for sync/fsync |
| Health | `Percent_Lifetime_Remain` = **97%** (3% used), 13 904 power-on hours, 0 reallocated sectors, ~96 TB written. Excellent. |
| Pool size / alloc / **free** | 888 G / 538 G / **350 G free** (`zpool list`, 60% cap) |
| ZFS `AVAIL` | **~323 GiB** available to any new dataset/zvol |
| System root | `rpool/ROOT/pve-1` = **430 G used** (Proxmox host `/`) |
| Proxmox storage on it | `local-zfs` → pool `rpool/data`, `content images,rootdir`, **`sparse 1`** (thin), ~338 GiB available per `pvesm status` |
| Pool settings | `compression on`, **`sync standard`** (inherited), `dedup off`, recordsize 128K |

`local-zfs` already advertises `images,rootdir` content, so **Proxmox can place VM
disks on the SSD today with no storage-config change.**

### 1.2 Current VM disk backing — confirms the HDD assumption

VMs (bpg/OpenTofu-provisioned): **dev-1 = vmid 105, dev-2 = 102, dev-3 = 101**.
Each has two disks, **both on the `Storage` datastore**:

- `Storage` = ZFS pool on **2× WDC WUH721818ALE6L4 18 TB HDD** (`sdc`/`sdd`, `ROTA=1`), mirror, 16.4 T.
- Per node: `virtio0 = vm-<id>-disk-0` (64 G, OS/root) and `virtio1 = vm-<id>-disk-1` (128 G, Longhorn data). `volblocksize 16K`.

So **etcd, the OS, and Longhorn all currently sit on spinning rust.** That is the
root cause: etcd WAL fsync competes with Longhorn rebuild I/O on the same HDD spindles.

(`Storage2` = 2× Seagate ST8000VN004 8 TB HDD is a separate HDD pool, not involved.)

### 1.3 Size of the hot data

- **etcd DB: ~82.6 MB.** Measured via apiserver metric
  `apiserver_storage_size_bytes{storage_cluster_id="etcd-0"} = 8.264e7`.
  A 4-day-old cluster running ~18 services. Even allowing generous WAL + growth
  headroom, **etcd fits in single-digit GB with room to spare — trivially ×3.**
- **Longhorn: ~22 GB actual data per node right now**, but the data disk is
  **128 GiB** and Longhorn reports `storageScheduled ≈ 152 GiB` (thin-overcommitted)
  vs `storageMaximum ≈ 125 GiB` per node. 18 PVCs, `numberOfReplicas=3`.
  Provisioned total ≈ 152 GiB. Longhorn is the **churn source we want to isolate
  etcd from**, and it can grow toward 128 GiB/node.

### 1.4 Sizing conclusion

| Candidate | Per-node footprint | ×3 on rpool | Fits in 323 GiB avail? |
|---|---|---|---|
| **etcd** | ~83 MB now; size a 16 GiB disk | ~48 GiB reserved (thin) | **Yes, easily** |
| **Longhorn** | ~22 GB now, up to 128 GiB provisioned | up to **~384 GiB** | **No — would fill/threaten the boot disk** |

---

## 2. Viability verdict

**etcd on SSD: viable and high-leverage.** etcd is ~83 MB; three 16 GiB SSD-backed
disks cost ~48 GiB of thin allocation (actual ~250 MB) against 323 GiB free on an
enterprise, PLP-equipped, 97%-healthy SSD mirror. This directly removes the WAL
fsync path from the HDD contention domain — the smallest-footprint fix for the exact
failure mode. **This is the recommendation.**

**Longhorn on SSD: not viable here.** Its provisioned footprint (~384 GiB across 3
replicas at full size) does not fit alongside the 430 GiB Proxmox root on a 888 GiB
pool, and co-locating heavy Longhorn rebuild I/O on the boot SSD reintroduces
contention with the host itself. Keep Longhorn on HDD (`Storage`). Rejected.

---

## 3. Options evaluated

- **Option A — dedicated small SSD disk per node, etcd-only (RECOMMENDED).**
  Add a ~16 GiB SSD-backed virtual disk (zvol on `rpool/data` via `local-zfs`) to
  each of the 3 VMs, mount it at the k3s etcd directory
  (`/var/lib/rancher/k3s/server/db`), migrate etcd data onto it one node at a time.
  Isolates WAL fsync from HDD Longhorn churn. Minimal space, minimal blast radius.
- **Option B — Longhorn replicas/disk on SSD.** Rejected (§2): does not fit and
  re-couples host I/O.
- **Option C — do nothing; wait for the SSD mini-PCs.** Zero risk, zero work, but
  leaves the cluster fragile to every node bounce until the mini-PC migration lands
  (deferred, no committed date). Option A is a cheap, reversible bridge that Option C
  eventually supersedes (§7).

**Recommendation: Option A now, as an interim fix, carried by tofu.**

---

## 4. Recommended implementation — Option A, step-by-step

### 4.1 Model it in tofu (preferred over manual)

The VMs are bpg/OpenTofu-managed (`infra/tofu/`). Disks are declared as `disk`
blocks in `main.tf`; both current disks (`virtio0`, `virtio1`) point at
`var.datastore` (`Storage`). Add a **third disk on the SSD datastore**:

1. **New variables** in `infra/tofu/variables.tf`:
   ```hcl
   variable "etcd_ssd_datastore" {
     description = "SSD-backed datastore for the dedicated etcd disk (VERIFIED: local-zfs = rpool/data, ZFS mirror on Micron 5100 SATA SSDs w/ PLP)."
     type        = string
     default     = "local-zfs"
   }
   variable "etcd_disk_gib" {
     description = "Dedicated SSD etcd disk per node, GiB (etcd DB is ~83 MB; 16 GiB is generous headroom)."
     type        = number
     default     = 16
   }
   ```
2. **New disk block** in the VM resource in `main.tf`, appended AFTER the existing
   `virtio1` block (keep interface order ascending so bpg does not reorder):
   ```hcl
   # Dedicated SSD etcd disk (virtio2 -> /dev/vdc). Isolates etcd WAL fsync from
   # the HDD Storage pool where the OS + Longhorn disks live.
   disk {
     datastore_id = var.etcd_ssd_datastore
     interface    = "virtio2"
     size         = var.etcd_disk_gib
     discard      = "on"
     cache        = "none"   # honor guest fsync through to ZFS ZIL (PLP SSD = safe+fast)
     iothread     = false
   }
   ```
3. **Plan and inspect before applying.** `tofu plan` MUST show an **in-place update
   (disk add)** for the targeted VM — NOT a replacement. If it shows replace, stop and
   reassess (do not destroy a running etcd node).
4. **Apply one node at a time** with `-target` (see §4.3 for quorum ordering), e.g.
   `-target='proxmox_virtual_environment_vm.node["k3s-dev-3"]'` (adjust to the actual
   resource address in this repo). A disk add is applied online by bpg, but treat the
   node as going down for the etcd migration regardless.

> **Coupling caveat — dev-3 memory drift.** dev-1/dev-2 are at `memory_mib = 24576`
> but **dev-3 is live at 16384** (drift from tofu default 24576). A tofu apply that
> touches dev-3 will also try to reset its memory to 24576. Per the Proxmox rule,
> memory changes need a **full stop/start**, not a reboot. Decide deliberately:
> either (a) let the same maintenance window fix the memory drift on dev-3 (stop →
> set → start), or (b) `-target` narrowly and accept that dev-3's plan will include
> the memory change. Do dev-3 in its own window with eyes on it.

**Manual alternative (if not using tofu):** on the Proxmox host,
`qm set <vmid> -virtio2 local-zfs:16,discard=on,cache=none` then stop/start the VM.
This drifts from tofu state and is discouraged — if done, import/reconcile into tofu
afterward. Prefer the tofu path.

### 4.2 Optional space guarantee

`local-zfs` is `sparse` (thin), so the etcd zvols consume only real usage. To
guarantee etcd can never be squeezed if the system disk fills, set a modest
`refreservation` on each etcd zvol (e.g. `zfs set refreservation=16G
rpool/data/vm-<id>-disk-2`). 3×16 GiB = 48 GiB reserved out of 323 GiB avail — safe.
This is a host-side ZFS tweak, not expressible in bpg; apply manually if desired.

### 4.3 Migrate etcd onto the new disk — quorum-safe, ONE node at a time

k3s has no flag to relocate only the etcd directory (`--data-dir` moves everything),
so the clean approach is to **mount the SSD disk at
`/var/lib/rancher/k3s/server/db`** (this directory holds `etcd/` — member data, WAL,
snapshots — and nothing else that matters).

**Pre-flight:** confirm all 3 etcd members healthy and the cluster is quiet (no active
Longhorn rebuild). `kubectl get nodes` all Ready; check etcd member health.

**Per node, repeat for dev-3, then dev-2, then dev-1 (leader last):**

1. Verify 3/3 etcd members healthy BEFORE touching this node (losing one of three
   keeps quorum at 2/3; never take a second node until the previous is fully back).
2. Stop k3s on the node: `systemctl stop k3s`. (Node leaves quorum; 2/3 remain.)
3. Identify the new disk (`/dev/vdc` — the third virtio disk) and format it:
   `mkfs.ext4 -L k3s-etcd /dev/vdc` (ext4 or xfs; ext4 is fine and simple).
4. Preserve and move existing data:
   ```
   mv /var/lib/rancher/k3s/server/db /var/lib/rancher/k3s/server/db.bak
   mkdir /var/lib/rancher/k3s/server/db
   mount /dev/vdc /var/lib/rancher/k3s/server/db
   rsync -aHAX --numeric-ids /var/lib/rancher/k3s/server/db.bak/ /var/lib/rancher/k3s/server/db/
   ```
5. Persist the mount in `/etc/fstab` **by UUID** (`blkid /dev/vdc`), e.g.
   `UUID=<uuid> /var/lib/rancher/k3s/server/db ext4 defaults,noatime 0 2`, then
   `systemctl daemon-reload`.
6. Start k3s: `systemctl start k3s`. Wait until the node is `Ready` AND its etcd
   member is healthy and has re-joined (member list shows 3 started, no learner
   stuck). Confirm the VIP `192.168.2.50:6443` is stable.
7. Only after full recovery, remove the backup: `rm -rf
   /var/lib/rancher/k3s/server/db.bak`. Then proceed to the next node.

**Reproducibility for rebuilds:** the SSD disk + mount should also be baked into the
node **cloud-init first-boot baseline** (the same template that already provisions the
Longhorn `virtio1` disk and swap-off), so a `tofu`-rebuilt node comes up with
`/dev/vdc` formatted and mounted at the etcd path automatically. Update that template
in the same change set as the tofu disk block.

### 4.4 Validation

- `kubectl get nodes` — all 3 Ready; no flapping over a 30–60 min soak.
- etcd: 3 members, all started/healthy, one leader; DB size sane (~tens–hundreds MB).
- On each node, `df -h /var/lib/rancher/k3s/server/db` shows the new `/dev/vdc` mount.
- **Regression test the original failure:** with the cluster healthy, trigger a
  controlled Longhorn replica rebuild (e.g. bounce ONE node after approval) and
  confirm etcd stays healthy and the VIP does not flap — the whole point of the change.
- Host-side: `zpool status rpool` clean; `zfs list rpool/data` shows the 3 new
  `vm-*-disk-2` zvols with small real usage.

### 4.5 Rollback

Per node, fully reversible while `db.bak` still exists (before step 7):
`systemctl stop k3s` → `umount .../server/db` → `rmdir` the empty mountpoint →
`mv db.bak db` → remove the fstab line → `systemctl start k3s`. To fully revert the
disk, detach `virtio2` (`qm set <vmid> -delete virtio2` after stop) or `tofu` revert
the disk block, then stop/start. Because etcd is a replicated 3-member cluster, a
botched single node can also simply be reset and re-joined from the other two.

---

## 5. Risks

1. **System-disk contention / exhaustion (top risk).** `rpool` is also the Proxmox
   boot/root disk (430 GiB used of 888 GiB). Mitigations: keep the etcd disks tiny
   (16 GiB, thin), optional `refreservation` (§4.2), and **never** put Longhorn here
   (§2). etcd's steady-state real usage is ~hundreds of MB; the marginal I/O it adds
   to the SSD is negligible next to what it removes from the HDD.
2. **Consumer-SSD fsync/PLP caveat — N/A here, but assert it.** etcd correctness under
   power loss depends on honest fsync. These are **Micron 5100/52x0 enterprise SATA
   SSDs with power-loss protection**, so `sync=standard` + `cache=none` is both fast
   and durable. (If the disks were consumer SSDs without PLP this concern would be
   real — it is not, for this hardware.)
3. **Quorum during migration.** Mitigated by strict one-node-at-a-time with full
   recovery gates (§4.3) and leader-last ordering. Never take a second node down
   before the previous rejoins.
4. **tofu coupling with dev-3 memory drift + apply semantics (§4.1).** An apply
   touching dev-3 also resets memory 16384→24576 (needs stop/start). Verify `tofu
   plan` shows disk *add*, not VM *replace*, before applying. Use `-target`.
5. **Backup/restore assumptions.** k3s etcd snapshots (`/var/lib/rancher/k3s/server/db/snapshots`)
   now live on the SSD too — fine, but confirm any snapshot/backup job still points at
   the (unchanged) path.

---

## 6. Recommendation summary

Implement **Option A**: a 16 GiB SSD-backed `virtio2` disk per k3s VM (zvol on
`rpool/data` via `local-zfs`), etcd-only, migrated one node at a time with the
quorum-safe procedure in §4.3, modeled in tofu (+ cloud-init baseline) for
reproducibility. Do **not** move Longhorn to SSD. This is a small, reversible,
high-leverage fix targeting the exact WAL-starvation failure.

---

## 7. Relationship to the deferred SSD-mini-PC migration

This is explicitly an **interim bridge** on the current Proxmox dev host
(`192.168.2.102`), whose bulk storage is HDD. Once the k3s cluster migrates to the
planned **all-SSD mini-PCs**, every I/O path (OS, etcd, Longhorn) is already on SSD
and the HDD-contention failure mode disappears — the dedicated etcd disk becomes
unnecessary. At that point the isolation can be either dropped or carried forward
harmlessly (a dedicated etcd disk is still good hygiene). Because Option A is captured
in tofu + cloud-init, it travels with the cluster definition and can be removed in one
reviewed change when the mini-PCs land. Until then it stops the recurring incidents
at minimal cost.