Investigation + phased implementation plan to move the churn-heavy, latency-sensitive Longhorn volumes (Prometheus, the CNPG Postgres DBs, Loki, gitea) off the slow/failing HDD 'Storage' pool onto a tagged SSD-backed Longhorn disk on local-zfs, ending the reboot rebuild storm. Key finding: the recorded 'SSD does not fit Longhorn' blocker was based on the 384 GiB provisioned figure; actual working set is ~79 GB cluster-wide (~26 GB/node) since all bulk media is on NFS, not Longhorn. Hot tier is ~51 GB actual / 114 GiB provisioned and fits the ~323 GiB SSD AVAIL with headroom. Read-only investigation; no live storage touched. Co-Authored-By: Claude Opus 4.8 <[email protected]>
25 KiB
Longhorn SSD tiering for the k3s dev cluster
Status: PROPOSAL — awaiting user approval. No storage, tofu, Longhorn, or live
infrastructure has been changed. This is a read-only investigation + reviewable plan.
Author: investigation agent (read-only), 2026-08-31
Problem: All 18 Longhorn data volumes live on the slow/failing Seagate-class HDD
pool (Storage). Every node reboot marks that node's replicas stale and triggers a
full 18-volume rebuild that spikes Proxmox IO-delay to 30 %+ and runs for a long time,
making every upgrade / resize / reboot expensive and risky. etcd was already isolated
onto a dedicated per-node SSD disk (virtio2, local-zfs) for exactly this reason
(docs/plans/ssd-etcd-isolation.md). The recorded blocker was that the SSD "does not
fit" the 3 × 128 GiB Longhorn data disks, so Longhorn stayed on HDD "interim until the
all-SSD mini-PCs land."
Verdict (short): The "doesn't fit" blocker was computed against the provisioned
128 GiB disks. The actual working set is ~26 GB per node (~79 GB across all three
nodes) — bulk media is on NFS, not Longhorn, so every Longhorn volume is a small
config/DB volume. A hot-tier move onto SSD is viable and recommended; a full move
is also feasible on space but re-couples rebuild I/O with etcd/host on the same SSD
pool. Recommendation: Phase 1 hot-tier move (Prometheus + the CNPG DBs + Loki +
gitea) onto a tagged longhorn-ssd disk, one node at a time.
1. Measured facts (all collected read-only, 2026-08-31)
1.1 The Longhorn working set — provisioned vs actual
kubectl -n longhorn-system get volumes.longhorn.io — capacity (spec.size) vs
actual consumed (status.actualSize, the real on-disk footprint of one replica,
snapshots included). All 18 volumes are numberOfReplicas: 3, so each node holds one
replica of each and a node's on-disk footprint ≈ the actualSize column below.
| Volume / PVC | Namespace | App class | Provisioned | Actual (per replica) | Tier |
|---|---|---|---|---|---|
| prometheus-…-db-0 | monitoring | Prometheus TSDB | 8 GiB | 11.59 GB | HOT |
| unifi-rootdisk | unifi | KubeVirt VM disk (longhorn-vm) |
40 GiB | 6.20 GB | cold |
| nextcloud-data | nextcloud | file data | 20 GiB | 1.44 GB | cold |
| nextcloud-db-1 | nextcloud-db | CNPG Postgres | 5 GiB | 1.41 GB | HOT |
| storage-loki-0 | monitoring | Loki chunks/index | 5 GiB | 1.24 GB | HOT |
| keycloak-db-1 | keycloak | CNPG Postgres | 5 GiB | 0.93 GB | HOT |
| gitea-data | gitea | git repos + SQLite/meta | 10 GiB | 0.90 GB | HOT |
| semaphore-db-1 | semaphore-db | CNPG Postgres | 5 GiB | 0.79 GB | HOT |
| crafty-data | crafty | Minecraft world | 20 GiB | 0.54 GB | cold |
| plex-config | plex | Plex metadata/db | 8 GiB | 0.23 GB | cold |
| seerr-config | seerr | config | 5 GiB | 0.17 GB | cold |
| sonarr-config | sonarr | config db | 5 GiB | 0.15 GB | cold |
| qbittorrent-config | qbittorrent | config | 5 GiB | 0.15 GB | cold |
| radarr-config | radarr | config db | 5 GiB | 0.15 GB | cold |
| pihole-config | pihole | config | 2 GiB | 0.10 GB | cold |
| jackett-config | jackett | config | 2 GiB | 0.10 GB | cold |
| seerr/act-runner-data | gitea-actions | CI cache | 1 GiB | 0.05 GB | cold |
| semaphore-config | semaphore | config | 1 GiB | 0.05 GB | cold |
| TOTALS (per node / one replica) | 152 GiB | ~26.2 GB |
- Per node working set: ~26.2 GB actual vs 152 GiB provisioned (× 3 replicas = 456 GiB nominal, but Longhorn/ZFS are thin — nominal is not what lands on disk).
- Whole-cluster working set: ~79 GB across the three nodes.
- Bulk media is not on Longhorn. The StorageClass survey (§1.3) shows Plex media
libraries and the *arr download shares use the
bulkNFS class (nfs.csi.k8s.io), not Longhorn. So the "bulk stays on HDD while small/hot moves to SSD" framing mostly dissolves: there is no large Longhorn volume. The biggest isunifi-rootdiskat 6.2 GB actual, and it is a cold KubeVirt root disk.
1.2 Longhorn per-node disk (the current single HDD data disk)
kubectl -n longhorn-system get nodes.longhorn.io (all three nodes identical):
| Field | Value |
|---|---|
| Disk | default-disk, path /var/lib/longhorn, tags [], allowScheduling: true |
storageMaximum |
124.9 GiB (the ext4 fs on the 128 GiB virtio1 HDD disk) |
storageAvailable |
100.4 GiB free |
storageScheduled |
152.0 GiB — over-scheduled vs the 124.9 GiB max |
The 152 GiB scheduled > 124.9 GiB max is Longhorn honouring
storage-over-provisioning-percentage: 200. Actual usage is only ~24.5 GiB
(124.9 − 100.4), matching §1.1.
1.3 StorageClasses in play
kubectl get storageclass:
| Class | Provisioner | Notes |
|---|---|---|
longhorn |
driver.longhorn.io |
default class, 3 replicas, ext4 — 17 of 18 PVCs |
longhorn-vm |
driver.longhorn.io |
3 replicas, migratable: true — unifi-rootdisk only |
longhorn-scratch |
driver.longhorn.io |
1 replica, best-effort locality (ephemeral) |
longhorn-static |
driver.longhorn.io |
static-provisioning class |
bulk |
nfs.csi.k8s.io |
NFS — bulk media (unresolved REPLACE-ME server/share) |
1.4 Longhorn settings relevant to a migration / rebuild storm
kubectl -n longhorn-system get settings.longhorn.io …:
| Setting | Value | Relevance |
|---|---|---|
concurrent-replica-rebuild-per-node-limit |
1 | already throttled — one rebuild at a time per node (set during etcd work) |
storage-over-provisioning-percentage |
200 | dangerous on a small SSD disk — must lower for the SSD disk sizing |
storage-minimal-available-percentage |
25 | Longhorn stops scheduling when a disk is <25 % free — a safety floor |
replica-auto-balance |
best-effort | will help spread replicas after a new disk appears |
default-replica-count |
3 | — |
Longhorn version: v1.12.1. No backup target is configured
(backuptargets.longhorn.io/default: URL empty, available: false) — so any
migration path that relies on Longhorn backup → restore-to-new-SC is not
available until an S3/NFS backup target is set up. This constrains the migration
mechanism (§4) to in-cluster options: CSI clone, per-volume diskSelector rebuild, or
app-level redeploy.
1.5 The SSD pool (local-zfs = rpool/data)
Measured read-only on 2026-08-29 during the etcd-isolation work
(docs/plans/ssd-etcd-isolation.md §1.1) and unchanged in substance since — the only
addition was the three 16 GiB etcd disks on 2026-08-30, which are thin and hold ~83 MB
of etcd data each (~250 MB total actual):
| Property | Value |
|---|---|
| Pool | rpool, mirror of 2 × Micron 5100/52x0 960 GB enterprise SATA SSD w/ PLP |
| Health | 97 % lifetime remaining, 0 reallocated sectors — excellent |
zpool size / free |
888 G / ~350 G free (60 % capacity) |
ZFS AVAIL to a new dataset/zvol |
~323 GiB |
| Proxmox host root on it | rpool/ROOT/pve-1 = 430 G |
| Storage entry | local-zfs, content images,rootdir, sparse 1 (thin) |
| Already carries | 3 × 16 GiB etcd zvols (thin, ~250 MB actual) |
Re-verify at execution time. Before applying, re-run
zpool list rpoolandpvesm statuson the Proxmox host to confirm current free space (this figure is dated 2026-08-29). The vault-gated ansible path (proxmox_hostsgroup) or a Proxmox UI check both work.
1.6 Is the HDD failing — data-loss urgency?
The problem statement calls the Seagate HDD "slow/failing." SMART was not
re-collected in this investigation (the Proxmox host query is vault-gated in the
dev container). The etcd-isolation doc characterised Storage as 2 × WDC WUH721818
18 TB (healthy) and Storage2 as 2 × Seagate ST8000VN004 — the Longhorn virtio1
disks are on Storage (the WD mirror), a ZFS mirror, so a single-disk SMART fault
is not immediate data loss. Action item for the executor: collect
smartctl -a for both Storage-pool members and zpool status -x before scheduling;
if a member is degraded, that raises urgency and argues for moving the hot data off
sooner. Treat "failing" as unconfirmed until SMART is read — but the mirror + 3-way
Longhorn replication means the plan is not gated on it.
2. Capacity verdict — does it fit?
| Scenario | Per-node SSD footprint (actual) | ×3 across nodes | Provisioned to size the disk | Fits in ~323 GiB SSD AVAIL? |
|---|---|---|---|---|
| Hot tier (Prometheus + 3 CNPG DBs + Loki + gitea) | ~16.9 GB | ~50.6 GB | 38 GiB/node → suggest a 48 GiB SSD disk/node | Yes, comfortably (~145 GiB of thin zvols max, ~51 GB actual) |
| Full move (all 18 Longhorn volumes) | ~26.2 GB | ~79 GB | 152 GiB/node nominal, but thin | Yes on space (~79 GB actual) — the "doesn't fit" blocker was a provisioned-number artefact |
| All-SSD mini-PC path (no move now) | — | — | — | N/A — hardware-blocked, no committed date |
The recorded blocker is overturned by the actual numbers. 3 × 128 GiB = 384 GiB
provisioned does not fit next to the 430 GiB host root on an 888 GiB pool — that is
true and is why the earlier doc said "NO for Longhorn." But the real data is ~79 GB,
and both ZFS (sparse 1) and Longhorn are thin, so what actually lands on the SSD is
the working set, not the nominal disk size. Both a hot-tier move and a full move fit on
free space today.
The reason to still prefer the hot tier over a full move is not space — it is
I/O coupling (see §3.4 and §6): the only SSD pool is rpool, which now also carries
the Proxmox host root and the freshly-isolated etcd disks. A full move puts every
Longhorn rebuild back onto the same pool as etcd, partially undoing the isolation just
completed. A hot-tier move keeps the SSD footprint and the rebuild-I/O burst small while
capturing the single largest and churniest contributor (Prometheus = 11.6 GB = 44 % of
the whole working set) and all the latency-sensitive Postgres DBs.
3. Options evaluated
3.1 Option A — Hot-tier move onto a tagged SSD Longhorn disk (RECOMMENDED)
Add a second, SSD-backed Longhorn data disk per node (a new virtio3 zvol on
local-zfs), tag it ssd, tag the existing HDD disk hdd, publish a longhorn-ssd
StorageClass with diskSelector: ["ssd"], and migrate the hot volumes onto it. Cold
config volumes stay on the HDD default-disk.
- Hot set:
prometheus-…-db-0(monitoring),storage-loki-0(monitoring),keycloak-db-1,nextcloud-db-1,semaphore-db-1(the three CNPG Postgres DBs),gitea-data. ~16.9 GB/node actual, 38 GiB provisioned. - Why these: Prometheus is the biggest and churniest (TSDB compaction + snapshot bloat — 11.6 GB actual on an 8 GiB nominal volume). Postgres DBs are fsync-latency sensitive and small. Loki and gitea are moderate-churn. Moving these removes the bulk of rebuild bytes and the latency-sensitive I/O from the HDD.
- Footprint: small and bounded → the I/O-coupling risk on
rpoolis minimal. - Downside: the ~12 cold config volumes (~9 GB/node) still rebuild on the HDD on a reboot — but they are tiny and low-churn, so the storm shrinks from 18 volumes / ~26 GB to ~12 volumes / ~9 GB, and the SSD ones rebuild fast.
3.2 Option B — Full move (all 18 Longhorn volumes to SSD)
Size the SSD disk per node for the full working set (e.g. 96 GiB thin, lower over-provisioning) and repoint the default StorageClass / migrate every volume.
- Pro: eliminates HDD Longhorn rebuild entirely — reboots no longer touch the HDD for storage; SSD rebuild is fast and low-IO-delay.
- Con: re-couples all Longhorn rebuild I/O with etcd + host root on
rpool, partially undoingdocs/plans/ssd-etcd-isolation.md. SSD IOPS make this far less harmful than the HDD case, but it is a real regression of the isolation just done. Also consumes more of the small SSD pool's headroom. - Verdict: feasible, but hold it as a Phase 2 only if the hot-tier move proves insufficient. Not the opening move.
3.3 Option C — Do nothing; wait for the all-SSD mini-PCs
Zero work, zero risk, but leaves the cluster fragile to every node bounce
indefinitely — the mini-PC migration has no committed date. What that path requires:
new hardware (all-SSD mini-PCs), a fresh Proxmox/k3s build, and a full Longhorn data
migration anyway. Option A is a cheap, reversible bridge that the mini-PC path later
supersedes — it does not waste that work (the longhorn-ssd class and tagging carry
over).
3.4 The I/O-coupling caveat (applies to A and B)
rpool is the only SSD pool. It already carries the host root and the isolated
etcd disks. Putting Longhorn rebuild I/O back on it re-introduces some contention
with etcd's WAL fsync — the exact thing the etcd disk was meant to escape. Two facts
keep Option A safe: (1) SSD random IOPS are ~2–3 orders of magnitude above the HDD, so
concurrent fsync + rebuild is tolerable where it was fatal on spinning rust; (2) the
hot-tier footprint is small so rebuild bursts are short. Mitigation: keep
concurrent-replica-rebuild-per-node-limit: 1, and watch
etcd_disk_wal_fsync_duration_seconds (Prometheus) during the first SSD rebuild — if
p99 fsync climbs materially, stop and reassess before proceeding to the next node.
4. Recommended implementation — Option A, phased, one node at a time
Every step below is a proposal. Nothing here has been executed. Apply per node, lowest-quorum-risk node first, verifying Longhorn/etcd health between nodes.
Phase 0 — Pre-flight (read-only, do before touching anything)
- Re-verify SSD free space: on the Proxmox host
zpool list rpoolandpvesm status(confirm ≳ 100 GiBAVAILremains after the etcd zvols). - Collect HDD SMART:
smartctl -aon bothStorage-pool members +zpool status -x. If a member is degraded, note the added urgency but the mirror + 3× replication mean the plan is unchanged. - Confirm Longhorn is fully healthy (all volumes
robustness: healthy) and no rebuild is in flight. Do not start during a rebuild. - Snapshot the CNPG DBs (CNPG can take an on-demand base backup) and note that gitea / Prometheus / Loki data is reproducible or non-critical — belt-and-braces before any volume move, since there is no Longhorn backup target.
Phase 1 — Add an SSD Longhorn disk per node (tofu, mirrors the etcd-disk pattern)
The VMs are bpg/OpenTofu-managed (infra/tofu/main.tf,
resource "proxmox_virtual_environment_vm" "node", for_each = local.nodes). Disks
are disk {} blocks; virtio0 (OS) and virtio1 (Longhorn HDD) are on var.datastore
(Storage), virtio2 (etcd) is on var.etcd_ssd_datastore (local-zfs). Add a
fourth disk on the SSD datastore, kept last so bpg does not reorder existing disks.
- New variables in
infra/tofu/variables.tf:variable "longhorn_ssd_datastore" { description = "SSD-backed datastore for the tagged Longhorn SSD disk (local-zfs = rpool/data, Micron 5100 SATA SSD mirror w/ PLP)." type = string default = "local-zfs" } variable "longhorn_ssd_disk_gib" { description = "Per-node SSD Longhorn disk, GiB. Hot tier is ~17 GiB actual / 38 GiB provisioned; 48 GiB gives headroom without crowding rpool." type = number default = 48 } - New
disk {}block in the VM resource inmain.tf, appended after thevirtio2etcd block (ascending interface →virtio3→/dev/vdd):# Tagged SSD Longhorn disk (virtio3 -> /dev/vdd) for the hot tier (Prometheus, CNPG # DBs, Loki, gitea). See docs/plans/longhorn-ssd-tiering.md. Kept last so bpg won't # reorder virtio0/1/2. discard=on so ext4/Longhorn TRIM reclaims thin zvol space. disk { datastore_id = var.longhorn_ssd_datastore interface = "virtio3" size = var.longhorn_ssd_disk_gib discard = "on" cache = "none" iothread = false } tofu planmust show an in-place disk add (update) for each node — not a replacement. If it shows replace, stop (never destroy a running etcd node).- Apply one node at a time with
-targeton the name-keyed resource address, e.g.-target='proxmox_virtual_environment_vm.node["k3s-dev-3"]'. A disk add is applied online by bpg, but treat the node as maintenance-affected regardless.
Phase 2 — Format + mount the SSD disk (ansible node_baseline, mirrors etcd)
roles/node_baseline already formats/mounts the HDD Longhorn disk (/dev/vdb →
/var/lib/longhorn) and the etcd disk (/dev/vdc). Add the same idempotent,
data-safe pattern for /dev/vdd:
- New defaults in
roles/node_baseline/defaults/main.yml:node_baseline_longhorn_ssd_device: /dev/vdd node_baseline_longhorn_ssd_mount: /var/lib/longhorn-ssd node_baseline_longhorn_ssd_fs_label: longhorn-ssd - New tasks in
roles/node_baseline/tasks/main.yml(copy the HDD Longhorn filesystem+mount pair;community.general.filesystemnever reformats an existing fs, so re-runs are safe):Use- name: Create ext4 filesystem on the SSD Longhorn disk (data-safe; skips if present) community.general.filesystem: fstype: ext4 dev: "{{ node_baseline_longhorn_ssd_device }}" opts: "-L {{ node_baseline_longhorn_ssd_fs_label }}" - name: Mount the SSD Longhorn disk ansible.posix.mount: path: "{{ node_baseline_longhorn_ssd_mount }}" src: "LABEL={{ node_baseline_longhorn_ssd_fs_label }}" fstype: ext4 opts: "defaults,nofail,x-systemd.device-timeout=5s" state: mountedLABEL=(not/dev/vdd) so the fstab entry survives device renames, andnofailso a missing SSD Longhorn disk does not block boot (unlike etcd, an absent Longhorn extra-disk should not wedge the node).
Phase 3 — Register + tag the disks in Longhorn
Per node (via the Longhorn nodes.longhorn.io CR or UI):
- Tag the existing HDD disk
default-diskwithhddand set the newlonghorn-ssdStorageClass's counterpart default class to select it (so cold volumes stay put and nothing hot accidentally lands on HDD once the SSD class exists). - Add the SSD disk at
/var/lib/longhorn-ssdas a new Longhorn disk on each node,allowScheduling: true, tagssd.# patch each nodes.longhorn.io/<node> .spec.disks disk-ssd: path: /var/lib/longhorn-ssd allowScheduling: true tags: ["ssd"] storageReserved: 0 - Confirm each node now reports two disks with correct
storageMaximum(~48 GiB for the SSD one) andSchedulablecondition true.
Phase 4 — Publish the longhorn-ssd StorageClass
New GitOps manifest (wherever the Longhorn StorageClasses are declared in
platform/ — mirror the existing longhorn class and add the selector):
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: longhorn-ssd
provisioner: driver.longhorn.io
allowVolumeExpansion: true
parameters:
numberOfReplicas: "3"
staleReplicaTimeout: "30"
fsType: ext4
dataLocality: disabled
diskSelector: "ssd" # only schedules replicas on ssd-tagged disks
disableRevisionCounter: "true"
unmapMarkSnapChainRemoved: "ignored"
Optionally add a matching diskSelector: "hdd" to the existing default longhorn
class so newly-created cold volumes stay on HDD.
Phase 5 — Migrate the hot volumes (no Longhorn backup target → in-cluster methods)
Because no backup target exists (§1.4), use one of these, one volume at a time,
respecting concurrent-replica-rebuild-per-node-limit: 1:
- CNPG DBs (keycloak / nextcloud / semaphore) — cleanest via the operator. Change
the CNPG
Cluster.spec.storage.storageClasstolonghorn-ssdand let CNPG do a rolling replica re-creation (it provisions a fresh instance on the new class, streams from the primary, then promotes) — no manual data copy, no downtime beyond a switchover. Do one CNPG cluster at a time; verify replication caught up before the next. - Prometheus / Loki / gitea — per-volume
diskSelectorrebuild (in place, no PVC swap). Patch the existing Longhorn volume to select the SSD disk and let Longhorn rebuild the replicas onto it, then drop the HDD replicas:Then trigger a controlled rebuild by deleting one HDD replica at a time (Longhorn reschedules onto ankubectl -n longhorn-system patch volumes.longhorn.io <vol> --type merge \ -p '{"spec":{"diskSelector":["ssd"]}}'ssd-tagged disk;replica-auto-balance: best-efforthelps). Watch the volume return torobustness: healthybefore touching the next replica or the next volume. This avoids recreating PVCs and keeps the app running.- Alternative if the in-place rebuild misbehaves: scale the app to 0, CSI-clone
the PVC onto
longhorn-ssd(dataSource= existing PVC), repoint the workload to the clone, scale back up. More disruptive; keep as fallback.
- Alternative if the in-place rebuild misbehaves: scale the app to 0, CSI-clone
the PVC onto
Phase 6 — Validate
- All hot volumes
robustness: healthy, replicas only onssd-tagged disks (kubectl -n longhorn-system get replicas.longhorn.io -o wide). - SSD Longhorn disk usage ≈ 17 GiB/node; HDD disk down to the cold ~9 GiB/node.
- Deliberately reboot one node and confirm the rebuild storm is materially smaller (fewer volumes, SSD-fast) and Proxmox IO-delay stays low. Confirm etcd fsync p99 did not regress during the SSD rebuild (§3.4).
5. Rebuild-storm mitigation (built into the plan)
- One node at a time for every phase (tofu
-target, ansible--limit, one CNPG cluster / one replica at a time) — etcd quorum is 3-node, never take two down. concurrent-replica-rebuild-per-node-limit: 1already set — a rebooted node rebuilds serially, capping IO-delay.- Migrate during a quiet window, never while another rebuild is in flight.
- Lower
storage-over-provisioning-percentagefor the small SSD disk sizing — 200 % on a 48 GiB disk would let Longhorn schedule ~96 GiB of nominal claims onto ~17 GiB of real data; keep the hot set well understorageMaximum, and rely onstorage-minimal-available-percentage: 25as the floor. (The setting is global — consider dropping it to ~120–150 % cluster-wide, or simply keep the SSD disk sized so 200 % of the hot claims still fits.) - Verify etcd fsync latency during the first SSD rebuild before proceeding (§3.4).
6. Risks and rollback
| Risk | Likelihood | Mitigation / rollback |
|---|---|---|
SSD rebuild I/O re-starves etcd on shared rpool |
Low (SSD IOPS) | rebuild-limit 1; watch etcd_disk_wal_fsync_duration; stop after node 1 if p99 regresses |
| SSD pool fills (host root 430 G + etcd + Longhorn on 888 G) | Low (hot set ~51 GB) | keep SSD disk small (48 GiB), storage-minimal-available-percentage: 25; re-check zpool list between nodes |
| tofu shows replace instead of disk-add | Medium if module drift | inspect tofu plan per node; abort on replace — never destroy a running etcd node |
| Volume-migration data loss (no backup target) | Low | CNPG native rolling; per-volume rebuild keeps HDD replica until SSD healthy; CNPG base backup + reproducible data taken in Phase 0 |
Data disk device rename (/dev/vdd shifts) |
Low | mount by LABEL=longhorn-ssd, not device path |
| HDD actually failing mid-migration | Unknown (SMART unread) | 3× replication + mirror; Phase 0 SMART check; hot data lands on SSD first, reducing exposure |
Rollback: every phase is reversible. To back out: migrate the hot volumes back to
the HDD default-disk (reverse the diskSelector patch / point CNPG back at
longhorn), remove the disk-ssd from each Longhorn node, delete the longhorn-ssd
StorageClass, then remove the virtio3 disk block from tofu and re-apply one node at a
time. The SSD zvols are thin and independent of the HDD data, so removal is clean.
7. Recommendation
Adopt Option A (hot-tier move), phased, one node at a time. The recorded "SSD
doesn't fit Longhorn" blocker was based on the 384 GiB provisioned figure; the
actual working set is ~79 GB cluster-wide (~26 GB/node) because all bulk media
lives on NFS and every Longhorn volume is a small config/DB volume. The hot tier
(Prometheus + the three CNPG Postgres DBs + Loki + gitea) is ~51 GB actual /
114 GiB provisioned across three nodes and fits the ~323 GiB SSD AVAIL with wide
headroom, while capturing the single largest and churniest rebuild contributor
(Prometheus, 44 % of the working set) and the latency-sensitive databases.
A full move is feasible on space and would fully eliminate HDD Longhorn rebuilds,
but it re-couples all rebuild I/O with etcd + host root on the only SSD pool, partially
undoing the just-completed etcd isolation — hold it as an optional Phase 2 gated on the
hot-tier move proving insufficient. Option C (wait for mini-PCs) is not a solution
tonight — no committed date, and the tagging + longhorn-ssd class built here carry
over to that migration anyway.
Do this first, before any apply: re-verify live SSD free space (zpool list rpool, pvesm status) and read HDD SMART (smartctl -a, zpool status -x). If the
Seagate/Storage pool shows a degraded member, the urgency rises and the hot-tier move
should be scheduled sooner — but the mirror + 3× Longhorn replication mean the plan is
not gated on the disk's health.
This document is a plan only. No storage, tofu, Longhorn, StorageClass, or live infrastructure has been modified.