# Longhorn SSD tiering for the k3s dev cluster **Status:** PROPOSAL — awaiting user approval. No storage, tofu, Longhorn, or live infrastructure has been changed. This is a read-only investigation + reviewable plan. **Author:** investigation agent (read-only), 2026-08-31 **Problem:** All 18 Longhorn data volumes live on the slow/failing Seagate-class HDD pool (`Storage`). Every node reboot marks that node's replicas stale and triggers a full 18-volume rebuild that spikes Proxmox IO-delay to 30 %+ and runs for a long time, making every upgrade / resize / reboot expensive and risky. etcd was already isolated onto a dedicated per-node SSD disk (`virtio2`, `local-zfs`) for exactly this reason (`docs/plans/ssd-etcd-isolation.md`). The recorded blocker was that the SSD "does not fit" the 3 × 128 GiB Longhorn data disks, so Longhorn stayed on HDD "interim until the all-SSD mini-PCs land." **Verdict (short):** The "doesn't fit" blocker was computed against the **provisioned** 128 GiB disks. The **actual working set is ~26 GB per node (~79 GB across all three nodes)** — bulk media is on NFS, *not* Longhorn, so every Longhorn volume is a small config/DB volume. **A hot-tier move onto SSD is viable and recommended**; a full move is also feasible on space but re-couples rebuild I/O with etcd/host on the same SSD pool. Recommendation: **Phase 1 hot-tier move (Prometheus + the CNPG DBs + Loki + gitea) onto a tagged `longhorn-ssd` disk**, one node at a time. --- ## 1. Measured facts (all collected read-only, 2026-08-31) ### 1.1 The Longhorn working set — provisioned vs *actual* `kubectl -n longhorn-system get volumes.longhorn.io` — capacity (`spec.size`) vs **actual consumed** (`status.actualSize`, the real on-disk footprint of one replica, snapshots included). All 18 volumes are `numberOfReplicas: 3`, so each node holds one replica of each and a node's on-disk footprint ≈ the actualSize column below. | Volume / PVC | Namespace | App class | Provisioned | **Actual (per replica)** | Tier | |---|---|---|---:|---:|---| | prometheus-…-db-0 | monitoring | Prometheus TSDB | 8 GiB | **11.59 GB** | HOT | | unifi-rootdisk | unifi | KubeVirt VM disk (`longhorn-vm`) | 40 GiB | 6.20 GB | cold | | nextcloud-data | nextcloud | file data | 20 GiB | 1.44 GB | cold | | nextcloud-db-1 | nextcloud-db | CNPG Postgres | 5 GiB | 1.41 GB | HOT | | storage-loki-0 | monitoring | Loki chunks/index | 5 GiB | 1.24 GB | HOT | | keycloak-db-1 | keycloak | CNPG Postgres | 5 GiB | 0.93 GB | HOT | | gitea-data | gitea | git repos + SQLite/meta | 10 GiB | 0.90 GB | HOT | | semaphore-db-1 | semaphore-db | CNPG Postgres | 5 GiB | 0.79 GB | HOT | | crafty-data | crafty | Minecraft world | 20 GiB | 0.54 GB | cold | | plex-config | plex | Plex metadata/db | 8 GiB | 0.23 GB | cold | | seerr-config | seerr | config | 5 GiB | 0.17 GB | cold | | sonarr-config | sonarr | config db | 5 GiB | 0.15 GB | cold | | qbittorrent-config | qbittorrent | config | 5 GiB | 0.15 GB | cold | | radarr-config | radarr | config db | 5 GiB | 0.15 GB | cold | | pihole-config | pihole | config | 2 GiB | 0.10 GB | cold | | jackett-config | jackett | config | 2 GiB | 0.10 GB | cold | | seerr/act-runner-data | gitea-actions | CI cache | 1 GiB | 0.05 GB | cold | | semaphore-config | semaphore | config | 1 GiB | 0.05 GB | cold | | **TOTALS (per node / one replica)** | | | **152 GiB** | **~26.2 GB** | | - **Per node working set: ~26.2 GB actual** vs **152 GiB provisioned** (× 3 replicas = 456 GiB nominal, but Longhorn/ZFS are thin — nominal is not what lands on disk). - **Whole-cluster working set: ~79 GB** across the three nodes. - **Bulk media is not on Longhorn.** The StorageClass survey (§1.3) shows Plex media libraries and the *arr download shares use the `bulk` NFS class (`nfs.csi.k8s.io`), not Longhorn. So the "bulk stays on HDD while small/hot moves to SSD" framing mostly dissolves: **there is no large Longhorn volume.** The biggest is `unifi-rootdisk` at 6.2 GB actual, and it is a cold KubeVirt root disk. ### 1.2 Longhorn per-node disk (the current single HDD data disk) `kubectl -n longhorn-system get nodes.longhorn.io` (all three nodes identical): | Field | Value | |---|---| | Disk | `default-disk`, path `/var/lib/longhorn`, tags `[]`, `allowScheduling: true` | | `storageMaximum` | 124.9 GiB (the ext4 fs on the 128 GiB `virtio1` HDD disk) | | `storageAvailable` | 100.4 GiB free | | `storageScheduled` | **152.0 GiB** — over-scheduled vs the 124.9 GiB max | The 152 GiB *scheduled* > 124.9 GiB *max* is Longhorn honouring `storage-over-provisioning-percentage: 200`. Actual usage is only ~24.5 GiB (124.9 − 100.4), matching §1.1. ### 1.3 StorageClasses in play `kubectl get storageclass`: | Class | Provisioner | Notes | |---|---|---| | `longhorn` | `driver.longhorn.io` | default class, 3 replicas, ext4 — **17 of 18 PVCs** | | `longhorn-vm` | `driver.longhorn.io` | 3 replicas, `migratable: true` — `unifi-rootdisk` only | | `longhorn-scratch` | `driver.longhorn.io` | 1 replica, best-effort locality (ephemeral) | | `longhorn-static` | `driver.longhorn.io` | static-provisioning class | | `bulk` | `nfs.csi.k8s.io` | NFS — bulk media (unresolved `REPLACE-ME` server/share) | ### 1.4 Longhorn settings relevant to a migration / rebuild storm `kubectl -n longhorn-system get settings.longhorn.io …`: | Setting | Value | Relevance | |---|---|---| | `concurrent-replica-rebuild-per-node-limit` | **1** | already throttled — one rebuild at a time per node (set during etcd work) | | `storage-over-provisioning-percentage` | 200 | dangerous on a *small* SSD disk — must lower for the SSD disk sizing | | `storage-minimal-available-percentage` | 25 | Longhorn stops scheduling when a disk is <25 % free — a safety floor | | `replica-auto-balance` | best-effort | will help spread replicas after a new disk appears | | `default-replica-count` | 3 | — | Longhorn version: **v1.12.1**. **No backup target is configured** (`backuptargets.longhorn.io/default`: URL empty, `available: false`) — so any migration path that relies on Longhorn *backup → restore-to-new-SC* is **not available** until an S3/NFS backup target is set up. This constrains the migration mechanism (§4) to in-cluster options: CSI clone, per-volume `diskSelector` rebuild, or app-level redeploy. ### 1.5 The SSD pool (`local-zfs` = `rpool/data`) Measured read-only on 2026-08-29 during the etcd-isolation work (`docs/plans/ssd-etcd-isolation.md` §1.1) and unchanged in substance since — the only addition was the three 16 GiB etcd disks on 2026-08-30, which are thin and hold ~83 MB of etcd data each (~250 MB total actual): | Property | Value | |---|---| | Pool | `rpool`, mirror of 2 × Micron 5100/52x0 960 GB **enterprise SATA SSD w/ PLP** | | Health | 97 % lifetime remaining, 0 reallocated sectors — excellent | | `zpool` size / free | 888 G / **~350 G free** (60 % capacity) | | ZFS `AVAIL` to a new dataset/zvol | **~323 GiB** | | Proxmox host root on it | `rpool/ROOT/pve-1` = 430 G | | Storage entry | `local-zfs`, content `images,rootdir`, `sparse 1` (thin) | | Already carries | 3 × 16 GiB etcd zvols (thin, ~250 MB actual) | > **Re-verify at execution time.** Before applying, re-run `zpool list rpool` and > `pvesm status` on the Proxmox host to confirm current free space (this figure is > dated 2026-08-29). The vault-gated ansible path (`proxmox_hosts` group) or a Proxmox > UI check both work. ### 1.6 Is the HDD *failing* — data-loss urgency? The problem statement calls the Seagate HDD "slow/failing." **SMART was not re-collected in this investigation** (the Proxmox host query is vault-gated in the dev container). The etcd-isolation doc characterised `Storage` as 2 × WDC WUH721818 18 TB (healthy) and `Storage2` as 2 × Seagate ST8000VN004 — the Longhorn `virtio1` disks are on `Storage` (the WD mirror), a ZFS **mirror**, so a single-disk SMART fault is not immediate data loss. **Action item for the executor:** collect `smartctl -a` for both `Storage`-pool members and `zpool status -x` before scheduling; if a member is degraded, that raises urgency and argues for moving the *hot* data off sooner. Treat "failing" as unconfirmed until SMART is read — but the mirror + 3-way Longhorn replication means the plan is not gated on it. --- ## 2. Capacity verdict — does it fit? | Scenario | Per-node SSD footprint (actual) | ×3 across nodes | Provisioned to size the disk | Fits in ~323 GiB SSD `AVAIL`? | |---|---:|---:|---:|---| | **Hot tier** (Prometheus + 3 CNPG DBs + Loki + gitea) | **~16.9 GB** | **~50.6 GB** | 38 GiB/node → suggest a **48 GiB** SSD disk/node | **Yes, comfortably** (~145 GiB of thin zvols max, ~51 GB actual) | | **Full move** (all 18 Longhorn volumes) | ~26.2 GB | ~79 GB | 152 GiB/node nominal, but thin | **Yes on space** (~79 GB actual) — the "doesn't fit" blocker was a *provisioned*-number artefact | | All-SSD mini-PC path (no move now) | — | — | — | N/A — hardware-blocked, no committed date | **The recorded blocker is overturned by the actual numbers.** 3 × 128 GiB = 384 GiB *provisioned* does not fit next to the 430 GiB host root on an 888 GiB pool — that is true and is why the earlier doc said "NO for Longhorn." But the **real data is ~79 GB**, and both ZFS (`sparse 1`) and Longhorn are thin, so what actually lands on the SSD is the working set, not the nominal disk size. Both a hot-tier move and a full move fit on free space today. The reason to still prefer the **hot tier** over a full move is **not space** — it is **I/O coupling** (see §3.4 and §6): the only SSD pool is `rpool`, which now also carries the Proxmox host root *and* the freshly-isolated etcd disks. A full move puts every Longhorn rebuild back onto the same pool as etcd, partially undoing the isolation just completed. A hot-tier move keeps the SSD footprint and the rebuild-I/O burst small while capturing the single largest and churniest contributor (Prometheus = 11.6 GB = 44 % of the whole working set) and all the latency-sensitive Postgres DBs. --- ## 3. Options evaluated ### 3.1 Option A — Hot-tier move onto a tagged SSD Longhorn disk (RECOMMENDED) Add a second, SSD-backed Longhorn data disk per node (a new `virtio3` zvol on `local-zfs`), tag it `ssd`, tag the existing HDD disk `hdd`, publish a `longhorn-ssd` StorageClass with `diskSelector: ["ssd"]`, and migrate the hot volumes onto it. Cold config volumes stay on the HDD `default-disk`. - **Hot set:** `prometheus-…-db-0` (monitoring), `storage-loki-0` (monitoring), `keycloak-db-1`, `nextcloud-db-1`, `semaphore-db-1` (the three CNPG Postgres DBs), `gitea-data`. ~16.9 GB/node actual, 38 GiB provisioned. - **Why these:** Prometheus is the biggest and churniest (TSDB compaction + snapshot bloat — 11.6 GB actual on an 8 GiB nominal volume). Postgres DBs are fsync-latency sensitive and small. Loki and gitea are moderate-churn. Moving these removes the bulk of rebuild bytes *and* the latency-sensitive I/O from the HDD. - **Footprint:** small and bounded → the I/O-coupling risk on `rpool` is minimal. - **Downside:** the ~12 cold config volumes (~9 GB/node) still rebuild on the HDD on a reboot — but they are tiny and low-churn, so the storm shrinks from 18 volumes / ~26 GB to ~12 volumes / ~9 GB, and the SSD ones rebuild fast. ### 3.2 Option B — Full move (all 18 Longhorn volumes to SSD) Size the SSD disk per node for the full working set (e.g. 96 GiB thin, lower over-provisioning) and repoint the default StorageClass / migrate every volume. - **Pro:** eliminates HDD Longhorn rebuild entirely — reboots no longer touch the HDD for storage; SSD rebuild is fast and low-IO-delay. - **Con:** re-couples *all* Longhorn rebuild I/O with etcd + host root on `rpool`, partially undoing `docs/plans/ssd-etcd-isolation.md`. SSD IOPS make this far less harmful than the HDD case, but it is a real regression of the isolation just done. Also consumes more of the small SSD pool's headroom. - **Verdict:** feasible, but hold it as a *Phase 2* only if the hot-tier move proves insufficient. Not the opening move. ### 3.3 Option C — Do nothing; wait for the all-SSD mini-PCs Zero work, zero risk, but leaves the cluster fragile to every node bounce indefinitely — the mini-PC migration has no committed date. What that path *requires*: new hardware (all-SSD mini-PCs), a fresh Proxmox/k3s build, and a full Longhorn data migration anyway. Option A is a cheap, reversible bridge that the mini-PC path later supersedes — it does not waste that work (the `longhorn-ssd` class and tagging carry over). ### 3.4 The I/O-coupling caveat (applies to A and B) `rpool` is the **only** SSD pool. It already carries the host root and the isolated etcd disks. Putting Longhorn rebuild I/O back on it re-introduces *some* contention with etcd's WAL fsync — the exact thing the etcd disk was meant to escape. Two facts keep Option A safe: (1) SSD random IOPS are ~2–3 orders of magnitude above the HDD, so concurrent fsync + rebuild is tolerable where it was fatal on spinning rust; (2) the hot-tier footprint is small so rebuild bursts are short. **Mitigation:** keep `concurrent-replica-rebuild-per-node-limit: 1`, and watch `etcd_disk_wal_fsync_duration_seconds` (Prometheus) during the first SSD rebuild — if p99 fsync climbs materially, stop and reassess before proceeding to the next node. --- ## 4. Recommended implementation — Option A, phased, one node at a time > Every step below is a *proposal*. Nothing here has been executed. Apply per node, > lowest-quorum-risk node first, verifying Longhorn/etcd health between nodes. ### Phase 0 — Pre-flight (read-only, do before touching anything) 1. Re-verify SSD free space: on the Proxmox host `zpool list rpool` and `pvesm status` (confirm ≳ 100 GiB `AVAIL` remains after the etcd zvols). 2. Collect HDD SMART: `smartctl -a` on both `Storage`-pool members + `zpool status -x`. If a member is degraded, note the added urgency but the mirror + 3× replication mean the plan is unchanged. 3. Confirm Longhorn is fully healthy (all volumes `robustness: healthy`) and no rebuild is in flight. Do not start during a rebuild. 4. Snapshot the CNPG DBs (CNPG can take an on-demand base backup) and note that gitea / Prometheus / Loki data is reproducible or non-critical — belt-and-braces before any volume move, since there is no Longhorn backup target. ### Phase 1 — Add an SSD Longhorn disk per node (tofu, mirrors the etcd-disk pattern) The VMs are bpg/OpenTofu-managed (`infra/tofu/main.tf`, `resource "proxmox_virtual_environment_vm" "node"`, `for_each = local.nodes`). Disks are `disk {}` blocks; `virtio0` (OS) and `virtio1` (Longhorn HDD) are on `var.datastore` (`Storage`), `virtio2` (etcd) is on `var.etcd_ssd_datastore` (`local-zfs`). Add a **fourth disk on the SSD datastore**, kept last so bpg does not reorder existing disks. 1. New variables in `infra/tofu/variables.tf`: ```hcl variable "longhorn_ssd_datastore" { description = "SSD-backed datastore for the tagged Longhorn SSD disk (local-zfs = rpool/data, Micron 5100 SATA SSD mirror w/ PLP)." type = string default = "local-zfs" } variable "longhorn_ssd_disk_gib" { description = "Per-node SSD Longhorn disk, GiB. Hot tier is ~17 GiB actual / 38 GiB provisioned; 48 GiB gives headroom without crowding rpool." type = number default = 48 } ``` 2. New `disk {}` block in the VM resource in `main.tf`, appended **after** the `virtio2` etcd block (ascending interface → `virtio3` → `/dev/vdd`): ```hcl # Tagged SSD Longhorn disk (virtio3 -> /dev/vdd) for the hot tier (Prometheus, CNPG # DBs, Loki, gitea). See docs/plans/longhorn-ssd-tiering.md. Kept last so bpg won't # reorder virtio0/1/2. discard=on so ext4/Longhorn TRIM reclaims thin zvol space. disk { datastore_id = var.longhorn_ssd_datastore interface = "virtio3" size = var.longhorn_ssd_disk_gib discard = "on" cache = "none" iothread = false } ``` 3. `tofu plan` **must** show an in-place *disk add* (update) for each node — **not** a replacement. If it shows replace, stop (never destroy a running etcd node). 4. Apply **one node at a time** with `-target` on the name-keyed resource address, e.g. `-target='proxmox_virtual_environment_vm.node["k3s-dev-3"]'`. A disk add is applied online by bpg, but treat the node as maintenance-affected regardless. ### Phase 2 — Format + mount the SSD disk (ansible `node_baseline`, mirrors etcd) `roles/node_baseline` already formats/mounts the HDD Longhorn disk (`/dev/vdb` → `/var/lib/longhorn`) and the etcd disk (`/dev/vdc`). Add the same idempotent, data-safe pattern for `/dev/vdd`: 1. New defaults in `roles/node_baseline/defaults/main.yml`: ```yaml node_baseline_longhorn_ssd_device: /dev/vdd node_baseline_longhorn_ssd_mount: /var/lib/longhorn-ssd node_baseline_longhorn_ssd_fs_label: longhorn-ssd ``` 2. New tasks in `roles/node_baseline/tasks/main.yml` (copy the HDD Longhorn filesystem+mount pair; `community.general.filesystem` never reformats an existing fs, so re-runs are safe): ```yaml - name: Create ext4 filesystem on the SSD Longhorn disk (data-safe; skips if present) community.general.filesystem: fstype: ext4 dev: "{{ node_baseline_longhorn_ssd_device }}" opts: "-L {{ node_baseline_longhorn_ssd_fs_label }}" - name: Mount the SSD Longhorn disk ansible.posix.mount: path: "{{ node_baseline_longhorn_ssd_mount }}" src: "LABEL={{ node_baseline_longhorn_ssd_fs_label }}" fstype: ext4 opts: "defaults,nofail,x-systemd.device-timeout=5s" state: mounted ``` Use `LABEL=` (not `/dev/vdd`) so the fstab entry survives device renames, and `nofail` so a missing SSD Longhorn disk does not block boot (unlike etcd, an absent Longhorn extra-disk should not wedge the node). ### Phase 3 — Register + tag the disks in Longhorn Per node (via the Longhorn `nodes.longhorn.io` CR or UI): 1. **Tag the existing HDD disk** `default-disk` with `hdd` and set the new `longhorn-ssd` StorageClass's counterpart default class to select it (so cold volumes stay put and nothing hot accidentally lands on HDD once the SSD class exists). 2. **Add the SSD disk** at `/var/lib/longhorn-ssd` as a new Longhorn disk on each node, `allowScheduling: true`, tag `ssd`. ```yaml # patch each nodes.longhorn.io/ .spec.disks disk-ssd: path: /var/lib/longhorn-ssd allowScheduling: true tags: ["ssd"] storageReserved: 0 ``` 3. Confirm each node now reports two disks with correct `storageMaximum` (~48 GiB for the SSD one) and `Schedulable` condition true. ### Phase 4 — Publish the `longhorn-ssd` StorageClass New GitOps manifest (wherever the Longhorn StorageClasses are declared in `platform/` — mirror the existing `longhorn` class and add the selector): ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: longhorn-ssd provisioner: driver.longhorn.io allowVolumeExpansion: true parameters: numberOfReplicas: "3" staleReplicaTimeout: "30" fsType: ext4 dataLocality: disabled diskSelector: "ssd" # only schedules replicas on ssd-tagged disks disableRevisionCounter: "true" unmapMarkSnapChainRemoved: "ignored" ``` Optionally add a matching `diskSelector: "hdd"` to the existing default `longhorn` class so newly-created cold volumes stay on HDD. ### Phase 5 — Migrate the hot volumes (no Longhorn backup target → in-cluster methods) Because no backup target exists (§1.4), use one of these, **one volume at a time**, respecting `concurrent-replica-rebuild-per-node-limit: 1`: - **CNPG DBs (keycloak / nextcloud / semaphore) — cleanest via the operator.** Change the CNPG `Cluster` `.spec.storage.storageClass` to `longhorn-ssd` and let CNPG do a rolling replica re-creation (it provisions a fresh instance on the new class, streams from the primary, then promotes) — no manual data copy, no downtime beyond a switchover. Do one CNPG cluster at a time; verify replication caught up before the next. - **Prometheus / Loki / gitea — per-volume `diskSelector` rebuild (in place, no PVC swap).** Patch the existing Longhorn volume to select the SSD disk and let Longhorn rebuild the replicas onto it, then drop the HDD replicas: ``` kubectl -n longhorn-system patch volumes.longhorn.io --type merge \ -p '{"spec":{"diskSelector":["ssd"]}}' ``` Then trigger a controlled rebuild by deleting **one** HDD replica at a time (Longhorn reschedules onto an `ssd`-tagged disk; `replica-auto-balance: best-effort` helps). Watch the volume return to `robustness: healthy` before touching the next replica or the next volume. This avoids recreating PVCs and keeps the app running. - Alternative if the in-place rebuild misbehaves: scale the app to 0, CSI-**clone** the PVC onto `longhorn-ssd` (`dataSource` = existing PVC), repoint the workload to the clone, scale back up. More disruptive; keep as fallback. ### Phase 6 — Validate 1. All hot volumes `robustness: healthy`, replicas only on `ssd`-tagged disks (`kubectl -n longhorn-system get replicas.longhorn.io -o wide`). 2. SSD Longhorn disk usage ≈ 17 GiB/node; HDD disk down to the cold ~9 GiB/node. 3. Deliberately reboot **one** node and confirm the rebuild storm is materially smaller (fewer volumes, SSD-fast) and Proxmox IO-delay stays low. Confirm etcd fsync p99 did not regress during the SSD rebuild (§3.4). --- ## 5. Rebuild-storm mitigation (built into the plan) - **One node at a time** for every phase (tofu `-target`, ansible `--limit`, one CNPG cluster / one replica at a time) — etcd quorum is 3-node, never take two down. - **`concurrent-replica-rebuild-per-node-limit: 1`** already set — a rebooted node rebuilds serially, capping IO-delay. - **Migrate during a quiet window**, never while another rebuild is in flight. - **Lower `storage-over-provisioning-percentage`** *for the small SSD disk sizing* — 200 % on a 48 GiB disk would let Longhorn schedule ~96 GiB of nominal claims onto ~17 GiB of real data; keep the hot set well under `storageMaximum`, and rely on `storage-minimal-available-percentage: 25` as the floor. (The setting is global — consider dropping it to ~120–150 % cluster-wide, or simply keep the SSD disk sized so 200 % of the hot claims still fits.) - **Verify etcd fsync latency** during the first SSD rebuild before proceeding (§3.4). --- ## 6. Risks and rollback | Risk | Likelihood | Mitigation / rollback | |---|---|---| | SSD rebuild I/O re-starves etcd on shared `rpool` | Low (SSD IOPS) | rebuild-limit 1; watch `etcd_disk_wal_fsync_duration`; stop after node 1 if p99 regresses | | SSD pool fills (host root 430 G + etcd + Longhorn on 888 G) | Low (hot set ~51 GB) | keep SSD disk small (48 GiB), `storage-minimal-available-percentage: 25`; re-check `zpool list` between nodes | | tofu shows *replace* instead of disk-add | Medium if module drift | inspect `tofu plan` per node; abort on replace — never destroy a running etcd node | | Volume-migration data loss (no backup target) | Low | CNPG native rolling; per-volume rebuild keeps HDD replica until SSD healthy; CNPG base backup + reproducible data taken in Phase 0 | | Data disk device rename (`/dev/vdd` shifts) | Low | mount by `LABEL=longhorn-ssd`, not device path | | HDD actually failing mid-migration | Unknown (SMART unread) | 3× replication + mirror; Phase 0 SMART check; hot data lands on SSD first, reducing exposure | **Rollback:** every phase is reversible. To back out: migrate the hot volumes back to the HDD `default-disk` (reverse the `diskSelector` patch / point CNPG back at `longhorn`), remove the `disk-ssd` from each Longhorn node, delete the `longhorn-ssd` StorageClass, then remove the `virtio3` disk block from tofu and re-apply one node at a time. The SSD zvols are thin and independent of the HDD data, so removal is clean. --- ## 7. Recommendation **Adopt Option A (hot-tier move), phased, one node at a time.** The recorded "SSD doesn't fit Longhorn" blocker was based on the 384 GiB *provisioned* figure; the **actual working set is ~79 GB cluster-wide (~26 GB/node)** because all bulk media lives on NFS and every Longhorn volume is a small config/DB volume. The hot tier (Prometheus + the three CNPG Postgres DBs + Loki + gitea) is **~51 GB actual / 114 GiB provisioned** across three nodes and fits the ~323 GiB SSD `AVAIL` with wide headroom, while capturing the single largest and churniest rebuild contributor (Prometheus, 44 % of the working set) and the latency-sensitive databases. A **full move is feasible on space** and would fully eliminate HDD Longhorn rebuilds, but it re-couples all rebuild I/O with etcd + host root on the only SSD pool, partially undoing the just-completed etcd isolation — hold it as an optional Phase 2 gated on the hot-tier move proving insufficient. **Option C (wait for mini-PCs) is not a solution** tonight — no committed date, and the tagging + `longhorn-ssd` class built here carry over to that migration anyway. **Do this first, before any apply:** re-verify live SSD free space (`zpool list rpool`, `pvesm status`) and read HDD SMART (`smartctl -a`, `zpool status -x`). If the Seagate/`Storage` pool shows a degraded member, the urgency rises and the hot-tier move should be scheduled sooner — but the mirror + 3× Longhorn replication mean the plan is not gated on the disk's health. > This document is a plan only. No storage, tofu, Longhorn, StorageClass, or live > infrastructure has been modified.