home-network-k3s-impl/docs/plans/longhorn-ssd-tiering.md

# Longhorn SSD tiering for the k3s dev cluster

**Status:** PROPOSAL — awaiting user approval. No storage, tofu, Longhorn, or live
infrastructure has been changed. This is a read-only investigation + reviewable plan.
**Author:** investigation agent (read-only), 2026-08-31
**Problem:** All 18 Longhorn data volumes live on the slow/failing Seagate-class HDD
pool (`Storage`). Every node reboot marks that node's replicas stale and triggers a
full 18-volume rebuild that spikes Proxmox IO-delay to 30 %+ and runs for a long time,
making every upgrade / resize / reboot expensive and risky. etcd was already isolated
onto a dedicated per-node SSD disk (`virtio2`, `local-zfs`) for exactly this reason
(`docs/plans/ssd-etcd-isolation.md`). The recorded blocker was that the SSD "does not
fit" the 3 × 128 GiB Longhorn data disks, so Longhorn stayed on HDD "interim until the
all-SSD mini-PCs land."

**Verdict (short):** The "doesn't fit" blocker was computed against the **provisioned**
128 GiB disks. The **actual working set is ~26 GB per node (~79 GB across all three
nodes)** — bulk media is on NFS, *not* Longhorn, so every Longhorn volume is a small
config/DB volume. **A hot-tier move onto SSD is viable and recommended**; a full move
is also feasible on space but re-couples rebuild I/O with etcd/host on the same SSD
pool. Recommendation: **Phase 1 hot-tier move (Prometheus + the CNPG DBs + Loki +
gitea) onto a tagged `longhorn-ssd` disk**, one node at a time.

---

## 1. Measured facts (all collected read-only, 2026-08-31)

### 1.1 The Longhorn working set — provisioned vs *actual*

`kubectl -n longhorn-system get volumes.longhorn.io` — capacity (`spec.size`) vs
**actual consumed** (`status.actualSize`, the real on-disk footprint of one replica,
snapshots included). All 18 volumes are `numberOfReplicas: 3`, so each node holds one
replica of each and a node's on-disk footprint ≈ the actualSize column below.

| Volume / PVC | Namespace | App class | Provisioned | **Actual (per replica)** | Tier |
|---|---|---|---:|---:|---|
| prometheus-…-db-0 | monitoring | Prometheus TSDB | 8 GiB | **11.59 GB** | HOT |
| unifi-rootdisk | unifi | KubeVirt VM disk (`longhorn-vm`) | 40 GiB | 6.20 GB | cold |
| nextcloud-data | nextcloud | file data | 20 GiB | 1.44 GB | cold |
| nextcloud-db-1 | nextcloud-db | CNPG Postgres | 5 GiB | 1.41 GB | HOT |
| storage-loki-0 | monitoring | Loki chunks/index | 5 GiB | 1.24 GB | HOT |
| keycloak-db-1 | keycloak | CNPG Postgres | 5 GiB | 0.93 GB | HOT |
| gitea-data | gitea | git repos + SQLite/meta | 10 GiB | 0.90 GB | HOT |
| semaphore-db-1 | semaphore-db | CNPG Postgres | 5 GiB | 0.79 GB | HOT |
| crafty-data | crafty | Minecraft world | 20 GiB | 0.54 GB | cold |
| plex-config | plex | Plex metadata/db | 8 GiB | 0.23 GB | cold |
| seerr-config | seerr | config | 5 GiB | 0.17 GB | cold |
| sonarr-config | sonarr | config db | 5 GiB | 0.15 GB | cold |
| qbittorrent-config | qbittorrent | config | 5 GiB | 0.15 GB | cold |
| radarr-config | radarr | config db | 5 GiB | 0.15 GB | cold |
| pihole-config | pihole | config | 2 GiB | 0.10 GB | cold |
| jackett-config | jackett | config | 2 GiB | 0.10 GB | cold |
| seerr/act-runner-data | gitea-actions | CI cache | 1 GiB | 0.05 GB | cold |
| semaphore-config | semaphore | config | 1 GiB | 0.05 GB | cold |
| **TOTALS (per node / one replica)** | | | **152 GiB** | **~26.2 GB** | |

- **Per node working set: ~26.2 GB actual** vs **152 GiB provisioned** (× 3 replicas =
  456 GiB nominal, but Longhorn/ZFS are thin — nominal is not what lands on disk).
- **Whole-cluster working set: ~79 GB** across the three nodes.
- **Bulk media is not on Longhorn.** The StorageClass survey (§1.3) shows Plex media
  libraries and the *arr download shares use the `bulk` NFS class
  (`nfs.csi.k8s.io`), not Longhorn. So the "bulk stays on HDD while small/hot moves to
  SSD" framing mostly dissolves: **there is no large Longhorn volume.** The biggest is
  `unifi-rootdisk` at 6.2 GB actual, and it is a cold KubeVirt root disk.

### 1.2 Longhorn per-node disk (the current single HDD data disk)

`kubectl -n longhorn-system get nodes.longhorn.io` (all three nodes identical):

| Field | Value |
|---|---|
| Disk | `default-disk`, path `/var/lib/longhorn`, tags `[]`, `allowScheduling: true` |
| `storageMaximum` | 124.9 GiB (the ext4 fs on the 128 GiB `virtio1` HDD disk) |
| `storageAvailable` | 100.4 GiB free |
| `storageScheduled` | **152.0 GiB** — over-scheduled vs the 124.9 GiB max |

The 152 GiB *scheduled* > 124.9 GiB *max* is Longhorn honouring
`storage-over-provisioning-percentage: 200`. Actual usage is only ~24.5 GiB
(124.9 − 100.4), matching §1.1.

### 1.3 StorageClasses in play

`kubectl get storageclass`:

| Class | Provisioner | Notes |
|---|---|---|
| `longhorn` | `driver.longhorn.io` | default class, 3 replicas, ext4 — **17 of 18 PVCs** |
| `longhorn-vm` | `driver.longhorn.io` | 3 replicas, `migratable: true` — `unifi-rootdisk` only |
| `longhorn-scratch` | `driver.longhorn.io` | 1 replica, best-effort locality (ephemeral) |
| `longhorn-static` | `driver.longhorn.io` | static-provisioning class |
| `bulk` | `nfs.csi.k8s.io` | NFS — bulk media (unresolved `REPLACE-ME` server/share) |

### 1.4 Longhorn settings relevant to a migration / rebuild storm

`kubectl -n longhorn-system get settings.longhorn.io …`:

| Setting | Value | Relevance |
|---|---|---|
| `concurrent-replica-rebuild-per-node-limit` | **1** | already throttled — one rebuild at a time per node (set during etcd work) |
| `storage-over-provisioning-percentage` | 200 | dangerous on a *small* SSD disk — must lower for the SSD disk sizing |
| `storage-minimal-available-percentage` | 25 | Longhorn stops scheduling when a disk is <25 % free — a safety floor |
| `replica-auto-balance` | best-effort | will help spread replicas after a new disk appears |
| `default-replica-count` | 3 | — |

Longhorn version: **v1.12.1**. **No backup target is configured**
(`backuptargets.longhorn.io/default`: URL empty, `available: false`) — so any
migration path that relies on Longhorn *backup → restore-to-new-SC* is **not
available** until an S3/NFS backup target is set up. This constrains the migration
mechanism (§4) to in-cluster options: CSI clone, per-volume `diskSelector` rebuild, or
app-level redeploy.

### 1.5 The SSD pool (`local-zfs` = `rpool/data`)

Measured read-only on 2026-08-29 during the etcd-isolation work
(`docs/plans/ssd-etcd-isolation.md` §1.1) and unchanged in substance since — the only
addition was the three 16 GiB etcd disks on 2026-08-30, which are thin and hold ~83 MB
of etcd data each (~250 MB total actual):

| Property | Value |
|---|---|
| Pool | `rpool`, mirror of 2 × Micron 5100/52x0 960 GB **enterprise SATA SSD w/ PLP** |
| Health | 97 % lifetime remaining, 0 reallocated sectors — excellent |
| `zpool` size / free | 888 G / **~350 G free** (60 % capacity) |
| ZFS `AVAIL` to a new dataset/zvol | **~323 GiB** |
| Proxmox host root on it | `rpool/ROOT/pve-1` = 430 G |
| Storage entry | `local-zfs`, content `images,rootdir`, `sparse 1` (thin) |
| Already carries | 3 × 16 GiB etcd zvols (thin, ~250 MB actual) |

> **Re-verify at execution time.** Before applying, re-run `zpool list rpool` and
> `pvesm status` on the Proxmox host to confirm current free space (this figure is
> dated 2026-08-29). The vault-gated ansible path (`proxmox_hosts` group) or a Proxmox
> UI check both work.

### 1.6 Is the HDD *failing* — data-loss urgency?

The problem statement calls the Seagate HDD "slow/failing." **SMART was not
re-collected in this investigation** (the Proxmox host query is vault-gated in the
dev container). The etcd-isolation doc characterised `Storage` as 2 × WDC WUH721818
18 TB (healthy) and `Storage2` as 2 × Seagate ST8000VN004 — the Longhorn `virtio1`
disks are on `Storage` (the WD mirror), a ZFS **mirror**, so a single-disk SMART fault
is not immediate data loss. **Action item for the executor:** collect
`smartctl -a` for both `Storage`-pool members and `zpool status -x` before scheduling;
if a member is degraded, that raises urgency and argues for moving the *hot* data off
sooner. Treat "failing" as unconfirmed until SMART is read — but the mirror + 3-way
Longhorn replication means the plan is not gated on it.

---

## 2. Capacity verdict — does it fit?

| Scenario | Per-node SSD footprint (actual) | ×3 across nodes | Provisioned to size the disk | Fits in ~323 GiB SSD `AVAIL`? |
|---|---:|---:|---:|---|
| **Hot tier** (Prometheus + 3 CNPG DBs + Loki + gitea) | **~16.9 GB** | **~50.6 GB** | 38 GiB/node → suggest a **48 GiB** SSD disk/node | **Yes, comfortably** (~145 GiB of thin zvols max, ~51 GB actual) |
| **Full move** (all 18 Longhorn volumes) | ~26.2 GB | ~79 GB | 152 GiB/node nominal, but thin | **Yes on space** (~79 GB actual) — the "doesn't fit" blocker was a *provisioned*-number artefact |
| All-SSD mini-PC path (no move now) | — | — | — | N/A — hardware-blocked, no committed date |

**The recorded blocker is overturned by the actual numbers.** 3 × 128 GiB = 384 GiB
*provisioned* does not fit next to the 430 GiB host root on an 888 GiB pool — that is
true and is why the earlier doc said "NO for Longhorn." But the **real data is ~79 GB**,
and both ZFS (`sparse 1`) and Longhorn are thin, so what actually lands on the SSD is
the working set, not the nominal disk size. Both a hot-tier move and a full move fit on
free space today.

The reason to still prefer the **hot tier** over a full move is **not space** — it is
**I/O coupling** (see §3.4 and §6): the only SSD pool is `rpool`, which now also carries
the Proxmox host root *and* the freshly-isolated etcd disks. A full move puts every
Longhorn rebuild back onto the same pool as etcd, partially undoing the isolation just
completed. A hot-tier move keeps the SSD footprint and the rebuild-I/O burst small while
capturing the single largest and churniest contributor (Prometheus = 11.6 GB = 44 % of
the whole working set) and all the latency-sensitive Postgres DBs.

---

## 3. Options evaluated

### 3.1 Option A — Hot-tier move onto a tagged SSD Longhorn disk (RECOMMENDED)

Add a second, SSD-backed Longhorn data disk per node (a new `virtio3` zvol on
`local-zfs`), tag it `ssd`, tag the existing HDD disk `hdd`, publish a `longhorn-ssd`
StorageClass with `diskSelector: ["ssd"]`, and migrate the hot volumes onto it. Cold
config volumes stay on the HDD `default-disk`.

- **Hot set:** `prometheus-…-db-0` (monitoring), `storage-loki-0` (monitoring),
  `keycloak-db-1`, `nextcloud-db-1`, `semaphore-db-1` (the three CNPG Postgres DBs),
  `gitea-data`. ~16.9 GB/node actual, 38 GiB provisioned.
- **Why these:** Prometheus is the biggest and churniest (TSDB compaction + snapshot
  bloat — 11.6 GB actual on an 8 GiB nominal volume). Postgres DBs are fsync-latency
  sensitive and small. Loki and gitea are moderate-churn. Moving these removes the bulk
  of rebuild bytes *and* the latency-sensitive I/O from the HDD.
- **Footprint:** small and bounded → the I/O-coupling risk on `rpool` is minimal.
- **Downside:** the ~12 cold config volumes (~9 GB/node) still rebuild on the HDD on a
  reboot — but they are tiny and low-churn, so the storm shrinks from 18 volumes /
  ~26 GB to ~12 volumes / ~9 GB, and the SSD ones rebuild fast.

### 3.2 Option B — Full move (all 18 Longhorn volumes to SSD)

Size the SSD disk per node for the full working set (e.g. 96 GiB thin, lower
over-provisioning) and repoint the default StorageClass / migrate every volume.

- **Pro:** eliminates HDD Longhorn rebuild entirely — reboots no longer touch the HDD
  for storage; SSD rebuild is fast and low-IO-delay.
- **Con:** re-couples *all* Longhorn rebuild I/O with etcd + host root on `rpool`,
  partially undoing `docs/plans/ssd-etcd-isolation.md`. SSD IOPS make this far less
  harmful than the HDD case, but it is a real regression of the isolation just done.
  Also consumes more of the small SSD pool's headroom.
- **Verdict:** feasible, but hold it as a *Phase 2* only if the hot-tier move proves
  insufficient. Not the opening move.

### 3.3 Option C — Do nothing; wait for the all-SSD mini-PCs

Zero work, zero risk, but leaves the cluster fragile to every node bounce
indefinitely — the mini-PC migration has no committed date. What that path *requires*:
new hardware (all-SSD mini-PCs), a fresh Proxmox/k3s build, and a full Longhorn data
migration anyway. Option A is a cheap, reversible bridge that the mini-PC path later
supersedes — it does not waste that work (the `longhorn-ssd` class and tagging carry
over).

### 3.4 The I/O-coupling caveat (applies to A and B)

`rpool` is the **only** SSD pool. It already carries the host root and the isolated
etcd disks. Putting Longhorn rebuild I/O back on it re-introduces *some* contention
with etcd's WAL fsync — the exact thing the etcd disk was meant to escape. Two facts
keep Option A safe: (1) SSD random IOPS are ~2–3 orders of magnitude above the HDD, so
concurrent fsync + rebuild is tolerable where it was fatal on spinning rust; (2) the
hot-tier footprint is small so rebuild bursts are short. **Mitigation:** keep
`concurrent-replica-rebuild-per-node-limit: 1`, and watch
`etcd_disk_wal_fsync_duration_seconds` (Prometheus) during the first SSD rebuild — if
p99 fsync climbs materially, stop and reassess before proceeding to the next node.

---

## 4. Recommended implementation — Option A, phased, one node at a time

> Every step below is a *proposal*. Nothing here has been executed. Apply per node,
> lowest-quorum-risk node first, verifying Longhorn/etcd health between nodes.

### Phase 0 — Pre-flight (read-only, do before touching anything)

1. Re-verify SSD free space: on the Proxmox host `zpool list rpool` and `pvesm status`
   (confirm ≳ 100 GiB `AVAIL` remains after the etcd zvols).
2. Collect HDD SMART: `smartctl -a` on both `Storage`-pool members + `zpool status -x`.
   If a member is degraded, note the added urgency but the mirror + 3× replication mean
   the plan is unchanged.
3. Confirm Longhorn is fully healthy (all volumes `robustness: healthy`) and no rebuild
   is in flight. Do not start during a rebuild.
4. Snapshot the CNPG DBs (CNPG can take an on-demand base backup) and note that gitea /
   Prometheus / Loki data is reproducible or non-critical — belt-and-braces before any
   volume move, since there is no Longhorn backup target.

### Phase 1 — Add an SSD Longhorn disk per node (tofu, mirrors the etcd-disk pattern)

The VMs are bpg/OpenTofu-managed (`infra/tofu/main.tf`,
`resource "proxmox_virtual_environment_vm" "node"`, `for_each = local.nodes`). Disks
are `disk {}` blocks; `virtio0` (OS) and `virtio1` (Longhorn HDD) are on `var.datastore`
(`Storage`), `virtio2` (etcd) is on `var.etcd_ssd_datastore` (`local-zfs`). Add a
**fourth disk on the SSD datastore**, kept last so bpg does not reorder existing disks.

1. New variables in `infra/tofu/variables.tf`:
   ```hcl
   variable "longhorn_ssd_datastore" {
     description = "SSD-backed datastore for the tagged Longhorn SSD disk (local-zfs = rpool/data, Micron 5100 SATA SSD mirror w/ PLP)."
     type        = string
     default     = "local-zfs"
   }
   variable "longhorn_ssd_disk_gib" {
     description = "Per-node SSD Longhorn disk, GiB. Hot tier is ~17 GiB actual / 38 GiB provisioned; 48 GiB gives headroom without crowding rpool."
     type        = number
     default     = 48
   }
   ```
2. New `disk {}` block in the VM resource in `main.tf`, appended **after** the
   `virtio2` etcd block (ascending interface → `virtio3` → `/dev/vdd`):
   ```hcl
   # Tagged SSD Longhorn disk (virtio3 -> /dev/vdd) for the hot tier (Prometheus, CNPG
   # DBs, Loki, gitea). See docs/plans/longhorn-ssd-tiering.md. Kept last so bpg won't
   # reorder virtio0/1/2. discard=on so ext4/Longhorn TRIM reclaims thin zvol space.
   disk {
     datastore_id = var.longhorn_ssd_datastore
     interface    = "virtio3"
     size         = var.longhorn_ssd_disk_gib
     discard      = "on"
     cache        = "none"
     iothread     = false
   }
   ```
3. `tofu plan` **must** show an in-place *disk add* (update) for each node — **not** a
   replacement. If it shows replace, stop (never destroy a running etcd node).
4. Apply **one node at a time** with `-target` on the name-keyed resource address,
   e.g. `-target='proxmox_virtual_environment_vm.node["k3s-dev-3"]'`. A disk add is
   applied online by bpg, but treat the node as maintenance-affected regardless.

### Phase 2 — Format + mount the SSD disk (ansible `node_baseline`, mirrors etcd)

`roles/node_baseline` already formats/mounts the HDD Longhorn disk (`/dev/vdb` →
`/var/lib/longhorn`) and the etcd disk (`/dev/vdc`). Add the same idempotent,
data-safe pattern for `/dev/vdd`:

1. New defaults in `roles/node_baseline/defaults/main.yml`:
   ```yaml
   node_baseline_longhorn_ssd_device: /dev/vdd
   node_baseline_longhorn_ssd_mount: /var/lib/longhorn-ssd
   node_baseline_longhorn_ssd_fs_label: longhorn-ssd
   ```
2. New tasks in `roles/node_baseline/tasks/main.yml` (copy the HDD Longhorn
   filesystem+mount pair; `community.general.filesystem` never reformats an existing
   fs, so re-runs are safe):
   ```yaml
   - name: Create ext4 filesystem on the SSD Longhorn disk (data-safe; skips if present)
     community.general.filesystem:
       fstype: ext4
       dev: "{{ node_baseline_longhorn_ssd_device }}"
       opts: "-L {{ node_baseline_longhorn_ssd_fs_label }}"
   - name: Mount the SSD Longhorn disk
     ansible.posix.mount:
       path: "{{ node_baseline_longhorn_ssd_mount }}"
       src: "LABEL={{ node_baseline_longhorn_ssd_fs_label }}"
       fstype: ext4
       opts: "defaults,nofail,x-systemd.device-timeout=5s"
       state: mounted
   ```
   Use `LABEL=` (not `/dev/vdd`) so the fstab entry survives device renames, and
   `nofail` so a missing SSD Longhorn disk does not block boot (unlike etcd, an absent
   Longhorn extra-disk should not wedge the node).

### Phase 3 — Register + tag the disks in Longhorn

Per node (via the Longhorn `nodes.longhorn.io` CR or UI):

1. **Tag the existing HDD disk** `default-disk` with `hdd` and set the new
   `longhorn-ssd` StorageClass's counterpart default class to select it (so cold
   volumes stay put and nothing hot accidentally lands on HDD once the SSD class
   exists).
2. **Add the SSD disk** at `/var/lib/longhorn-ssd` as a new Longhorn disk on each node,
   `allowScheduling: true`, tag `ssd`.
   ```yaml
   # patch each nodes.longhorn.io/<node> .spec.disks
   disk-ssd:
     path: /var/lib/longhorn-ssd
     allowScheduling: true
     tags: ["ssd"]
     storageReserved: 0
   ```
3. Confirm each node now reports two disks with correct `storageMaximum` (~48 GiB for
   the SSD one) and `Schedulable` condition true.

### Phase 4 — Publish the `longhorn-ssd` StorageClass

New GitOps manifest (wherever the Longhorn StorageClasses are declared in
`platform/` — mirror the existing `longhorn` class and add the selector):

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-ssd
provisioner: driver.longhorn.io
allowVolumeExpansion: true
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "30"
  fsType: ext4
  dataLocality: disabled
  diskSelector: "ssd"          # only schedules replicas on ssd-tagged disks
  disableRevisionCounter: "true"
  unmapMarkSnapChainRemoved: "ignored"
```

Optionally add a matching `diskSelector: "hdd"` to the existing default `longhorn`
class so newly-created cold volumes stay on HDD.

### Phase 5 — Migrate the hot volumes (no Longhorn backup target → in-cluster methods)

Because no backup target exists (§1.4), use one of these, **one volume at a time**,
respecting `concurrent-replica-rebuild-per-node-limit: 1`:

- **CNPG DBs (keycloak / nextcloud / semaphore) — cleanest via the operator.** Change
  the CNPG `Cluster` `.spec.storage.storageClass` to `longhorn-ssd` and let CNPG do a
  rolling replica re-creation (it provisions a fresh instance on the new class, streams
  from the primary, then promotes) — no manual data copy, no downtime beyond a
  switchover. Do one CNPG cluster at a time; verify replication caught up before the
  next.
- **Prometheus / Loki / gitea — per-volume `diskSelector` rebuild (in place, no PVC
  swap).** Patch the existing Longhorn volume to select the SSD disk and let Longhorn
  rebuild the replicas onto it, then drop the HDD replicas:
  ```
  kubectl -n longhorn-system patch volumes.longhorn.io <vol> --type merge \
    -p '{"spec":{"diskSelector":["ssd"]}}'
  ```
  Then trigger a controlled rebuild by deleting **one** HDD replica at a time (Longhorn
  reschedules onto an `ssd`-tagged disk; `replica-auto-balance: best-effort` helps).
  Watch the volume return to `robustness: healthy` before touching the next replica or
  the next volume. This avoids recreating PVCs and keeps the app running.
  - Alternative if the in-place rebuild misbehaves: scale the app to 0, CSI-**clone**
    the PVC onto `longhorn-ssd` (`dataSource` = existing PVC), repoint the workload to
    the clone, scale back up. More disruptive; keep as fallback.

### Phase 6 — Validate

1. All hot volumes `robustness: healthy`, replicas only on `ssd`-tagged disks
   (`kubectl -n longhorn-system get replicas.longhorn.io -o wide`).
2. SSD Longhorn disk usage ≈ 17 GiB/node; HDD disk down to the cold ~9 GiB/node.
3. Deliberately reboot **one** node and confirm the rebuild storm is materially smaller
   (fewer volumes, SSD-fast) and Proxmox IO-delay stays low. Confirm etcd fsync p99 did
   not regress during the SSD rebuild (§3.4).

---

## 5. Rebuild-storm mitigation (built into the plan)

- **One node at a time** for every phase (tofu `-target`, ansible `--limit`, one CNPG
  cluster / one replica at a time) — etcd quorum is 3-node, never take two down.
- **`concurrent-replica-rebuild-per-node-limit: 1`** already set — a rebooted node
  rebuilds serially, capping IO-delay.
- **Migrate during a quiet window**, never while another rebuild is in flight.
- **Lower `storage-over-provisioning-percentage`** *for the small SSD disk sizing* —
  200 % on a 48 GiB disk would let Longhorn schedule ~96 GiB of nominal claims onto
  ~17 GiB of real data; keep the hot set well under `storageMaximum`, and rely on
  `storage-minimal-available-percentage: 25` as the floor. (The setting is global —
  consider dropping it to ~120–150 % cluster-wide, or simply keep the SSD disk sized so
  200 % of the hot claims still fits.)
- **Verify etcd fsync latency** during the first SSD rebuild before proceeding (§3.4).

---

## 6. Risks and rollback

| Risk | Likelihood | Mitigation / rollback |
|---|---|---|
| SSD rebuild I/O re-starves etcd on shared `rpool` | Low (SSD IOPS) | rebuild-limit 1; watch `etcd_disk_wal_fsync_duration`; stop after node 1 if p99 regresses |
| SSD pool fills (host root 430 G + etcd + Longhorn on 888 G) | Low (hot set ~51 GB) | keep SSD disk small (48 GiB), `storage-minimal-available-percentage: 25`; re-check `zpool list` between nodes |
| tofu shows *replace* instead of disk-add | Medium if module drift | inspect `tofu plan` per node; abort on replace — never destroy a running etcd node |
| Volume-migration data loss (no backup target) | Low | CNPG native rolling; per-volume rebuild keeps HDD replica until SSD healthy; CNPG base backup + reproducible data taken in Phase 0 |
| Data disk device rename (`/dev/vdd` shifts) | Low | mount by `LABEL=longhorn-ssd`, not device path |
| HDD actually failing mid-migration | Unknown (SMART unread) | 3× replication + mirror; Phase 0 SMART check; hot data lands on SSD first, reducing exposure |

**Rollback:** every phase is reversible. To back out: migrate the hot volumes back to
the HDD `default-disk` (reverse the `diskSelector` patch / point CNPG back at
`longhorn`), remove the `disk-ssd` from each Longhorn node, delete the `longhorn-ssd`
StorageClass, then remove the `virtio3` disk block from tofu and re-apply one node at a
time. The SSD zvols are thin and independent of the HDD data, so removal is clean.

---

## 7. Recommendation

**Adopt Option A (hot-tier move), phased, one node at a time.** The recorded "SSD
doesn't fit Longhorn" blocker was based on the 384 GiB *provisioned* figure; the
**actual working set is ~79 GB cluster-wide (~26 GB/node)** because all bulk media
lives on NFS and every Longhorn volume is a small config/DB volume. The hot tier
(Prometheus + the three CNPG Postgres DBs + Loki + gitea) is **~51 GB actual /
114 GiB provisioned** across three nodes and fits the ~323 GiB SSD `AVAIL` with wide
headroom, while capturing the single largest and churniest rebuild contributor
(Prometheus, 44 % of the working set) and the latency-sensitive databases.

A **full move is feasible on space** and would fully eliminate HDD Longhorn rebuilds,
but it re-couples all rebuild I/O with etcd + host root on the only SSD pool, partially
undoing the just-completed etcd isolation — hold it as an optional Phase 2 gated on the
hot-tier move proving insufficient. **Option C (wait for mini-PCs) is not a solution**
tonight — no committed date, and the tagging + `longhorn-ssd` class built here carry
over to that migration anyway.

**Do this first, before any apply:** re-verify live SSD free space (`zpool list
rpool`, `pvesm status`) and read HDD SMART (`smartctl -a`, `zpool status -x`). If the
Seagate/`Storage` pool shows a degraded member, the urgency rises and the hot-tier move
should be scheduled sooner — but the mirror + 3× Longhorn replication mean the plan is
not gated on the disk's health.

> This document is a plan only. No storage, tofu, Longhorn, StorageClass, or live
> infrastructure has been modified.