Files
home-network-k3s-impl/docs/plans/longhorn-ssd-tiering.md
T
jakman85andClaude Opus 4.8 a04251bbc8 docs: plan for Longhorn SSD tiering (hot-tier move off HDD)
Investigation + phased implementation plan to move the churn-heavy,
latency-sensitive Longhorn volumes (Prometheus, the CNPG Postgres DBs,
Loki, gitea) off the slow/failing HDD 'Storage' pool onto a tagged
SSD-backed Longhorn disk on local-zfs, ending the reboot rebuild storm.

Key finding: the recorded 'SSD does not fit Longhorn' blocker was based
on the 384 GiB provisioned figure; actual working set is ~79 GB
cluster-wide (~26 GB/node) since all bulk media is on NFS, not Longhorn.
Hot tier is ~51 GB actual / 114 GiB provisioned and fits the ~323 GiB
SSD AVAIL with headroom. Read-only investigation; no live storage
touched.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-31 08:27:21 -05:00

25 KiB
Raw Blame History

Longhorn SSD tiering for the k3s dev cluster

Status: PROPOSAL — awaiting user approval. No storage, tofu, Longhorn, or live infrastructure has been changed. This is a read-only investigation + reviewable plan. Author: investigation agent (read-only), 2026-08-31 Problem: All 18 Longhorn data volumes live on the slow/failing Seagate-class HDD pool (Storage). Every node reboot marks that node's replicas stale and triggers a full 18-volume rebuild that spikes Proxmox IO-delay to 30 %+ and runs for a long time, making every upgrade / resize / reboot expensive and risky. etcd was already isolated onto a dedicated per-node SSD disk (virtio2, local-zfs) for exactly this reason (docs/plans/ssd-etcd-isolation.md). The recorded blocker was that the SSD "does not fit" the 3 × 128 GiB Longhorn data disks, so Longhorn stayed on HDD "interim until the all-SSD mini-PCs land."

Verdict (short): The "doesn't fit" blocker was computed against the provisioned 128 GiB disks. The actual working set is ~26 GB per node (~79 GB across all three nodes) — bulk media is on NFS, not Longhorn, so every Longhorn volume is a small config/DB volume. A hot-tier move onto SSD is viable and recommended; a full move is also feasible on space but re-couples rebuild I/O with etcd/host on the same SSD pool. Recommendation: Phase 1 hot-tier move (Prometheus + the CNPG DBs + Loki + gitea) onto a tagged longhorn-ssd disk, one node at a time.


1. Measured facts (all collected read-only, 2026-08-31)

1.1 The Longhorn working set — provisioned vs actual

kubectl -n longhorn-system get volumes.longhorn.io — capacity (spec.size) vs actual consumed (status.actualSize, the real on-disk footprint of one replica, snapshots included). All 18 volumes are numberOfReplicas: 3, so each node holds one replica of each and a node's on-disk footprint ≈ the actualSize column below.

Volume / PVC Namespace App class Provisioned Actual (per replica) Tier
prometheus-…-db-0 monitoring Prometheus TSDB 8 GiB 11.59 GB HOT
unifi-rootdisk unifi KubeVirt VM disk (longhorn-vm) 40 GiB 6.20 GB cold
nextcloud-data nextcloud file data 20 GiB 1.44 GB cold
nextcloud-db-1 nextcloud-db CNPG Postgres 5 GiB 1.41 GB HOT
storage-loki-0 monitoring Loki chunks/index 5 GiB 1.24 GB HOT
keycloak-db-1 keycloak CNPG Postgres 5 GiB 0.93 GB HOT
gitea-data gitea git repos + SQLite/meta 10 GiB 0.90 GB HOT
semaphore-db-1 semaphore-db CNPG Postgres 5 GiB 0.79 GB HOT
crafty-data crafty Minecraft world 20 GiB 0.54 GB cold
plex-config plex Plex metadata/db 8 GiB 0.23 GB cold
seerr-config seerr config 5 GiB 0.17 GB cold
sonarr-config sonarr config db 5 GiB 0.15 GB cold
qbittorrent-config qbittorrent config 5 GiB 0.15 GB cold
radarr-config radarr config db 5 GiB 0.15 GB cold
pihole-config pihole config 2 GiB 0.10 GB cold
jackett-config jackett config 2 GiB 0.10 GB cold
seerr/act-runner-data gitea-actions CI cache 1 GiB 0.05 GB cold
semaphore-config semaphore config 1 GiB 0.05 GB cold
TOTALS (per node / one replica) 152 GiB ~26.2 GB
  • Per node working set: ~26.2 GB actual vs 152 GiB provisioned (× 3 replicas = 456 GiB nominal, but Longhorn/ZFS are thin — nominal is not what lands on disk).
  • Whole-cluster working set: ~79 GB across the three nodes.
  • Bulk media is not on Longhorn. The StorageClass survey (§1.3) shows Plex media libraries and the *arr download shares use the bulk NFS class (nfs.csi.k8s.io), not Longhorn. So the "bulk stays on HDD while small/hot moves to SSD" framing mostly dissolves: there is no large Longhorn volume. The biggest is unifi-rootdisk at 6.2 GB actual, and it is a cold KubeVirt root disk.

1.2 Longhorn per-node disk (the current single HDD data disk)

kubectl -n longhorn-system get nodes.longhorn.io (all three nodes identical):

Field Value
Disk default-disk, path /var/lib/longhorn, tags [], allowScheduling: true
storageMaximum 124.9 GiB (the ext4 fs on the 128 GiB virtio1 HDD disk)
storageAvailable 100.4 GiB free
storageScheduled 152.0 GiB — over-scheduled vs the 124.9 GiB max

The 152 GiB scheduled > 124.9 GiB max is Longhorn honouring storage-over-provisioning-percentage: 200. Actual usage is only ~24.5 GiB (124.9 − 100.4), matching §1.1.

1.3 StorageClasses in play

kubectl get storageclass:

Class Provisioner Notes
longhorn driver.longhorn.io default class, 3 replicas, ext4 — 17 of 18 PVCs
longhorn-vm driver.longhorn.io 3 replicas, migratable: true — unifi-rootdisk only
longhorn-scratch driver.longhorn.io 1 replica, best-effort locality (ephemeral)
longhorn-static driver.longhorn.io static-provisioning class
bulk nfs.csi.k8s.io NFS — bulk media (unresolved REPLACE-ME server/share)

1.4 Longhorn settings relevant to a migration / rebuild storm

kubectl -n longhorn-system get settings.longhorn.io …:

Setting Value Relevance
concurrent-replica-rebuild-per-node-limit 1 already throttled — one rebuild at a time per node (set during etcd work)
storage-over-provisioning-percentage 200 dangerous on a small SSD disk — must lower for the SSD disk sizing
storage-minimal-available-percentage 25 Longhorn stops scheduling when a disk is <25 % free — a safety floor
replica-auto-balance best-effort will help spread replicas after a new disk appears
default-replica-count 3 —

Longhorn version: v1.12.1. No backup target is configured (backuptargets.longhorn.io/default: URL empty, available: false) — so any migration path that relies on Longhorn backup → restore-to-new-SC is not available until an S3/NFS backup target is set up. This constrains the migration mechanism (§4) to in-cluster options: CSI clone, per-volume diskSelector rebuild, or app-level redeploy.

1.5 The SSD pool (local-zfs = rpool/data)

Measured read-only on 2026-08-29 during the etcd-isolation work (docs/plans/ssd-etcd-isolation.md §1.1) and unchanged in substance since — the only addition was the three 16 GiB etcd disks on 2026-08-30, which are thin and hold ~83 MB of etcd data each (~250 MB total actual):

Property Value
Pool rpool, mirror of 2 × Micron 5100/52x0 960 GB enterprise SATA SSD w/ PLP
Health 97 % lifetime remaining, 0 reallocated sectors — excellent
zpool size / free 888 G / ~350 G free (60 % capacity)
ZFS AVAIL to a new dataset/zvol ~323 GiB
Proxmox host root on it rpool/ROOT/pve-1 = 430 G
Storage entry local-zfs, content images,rootdir, sparse 1 (thin)
Already carries 3 × 16 GiB etcd zvols (thin, ~250 MB actual)

Re-verify at execution time. Before applying, re-run zpool list rpool and pvesm status on the Proxmox host to confirm current free space (this figure is dated 2026-08-29). The vault-gated ansible path (proxmox_hosts group) or a Proxmox UI check both work.

1.6 Is the HDD failing — data-loss urgency?

The problem statement calls the Seagate HDD "slow/failing." SMART was not re-collected in this investigation (the Proxmox host query is vault-gated in the dev container). The etcd-isolation doc characterised Storage as 2 × WDC WUH721818 18 TB (healthy) and Storage2 as 2 × Seagate ST8000VN004 — the Longhorn virtio1 disks are on Storage (the WD mirror), a ZFS mirror, so a single-disk SMART fault is not immediate data loss. Action item for the executor: collect smartctl -a for both Storage-pool members and zpool status -x before scheduling; if a member is degraded, that raises urgency and argues for moving the hot data off sooner. Treat "failing" as unconfirmed until SMART is read — but the mirror + 3-way Longhorn replication means the plan is not gated on it.


2. Capacity verdict — does it fit?

Scenario Per-node SSD footprint (actual) ×3 across nodes Provisioned to size the disk Fits in ~323 GiB SSD AVAIL?
Hot tier (Prometheus + 3 CNPG DBs + Loki + gitea) ~16.9 GB ~50.6 GB 38 GiB/node → suggest a 48 GiB SSD disk/node Yes, comfortably (~145 GiB of thin zvols max, ~51 GB actual)
Full move (all 18 Longhorn volumes) ~26.2 GB ~79 GB 152 GiB/node nominal, but thin Yes on space (~79 GB actual) — the "doesn't fit" blocker was a provisioned-number artefact
All-SSD mini-PC path (no move now) — — — N/A — hardware-blocked, no committed date

The recorded blocker is overturned by the actual numbers. 3 × 128 GiB = 384 GiB provisioned does not fit next to the 430 GiB host root on an 888 GiB pool — that is true and is why the earlier doc said "NO for Longhorn." But the real data is ~79 GB, and both ZFS (sparse 1) and Longhorn are thin, so what actually lands on the SSD is the working set, not the nominal disk size. Both a hot-tier move and a full move fit on free space today.

The reason to still prefer the hot tier over a full move is not space — it is I/O coupling (see §3.4 and §6): the only SSD pool is rpool, which now also carries the Proxmox host root and the freshly-isolated etcd disks. A full move puts every Longhorn rebuild back onto the same pool as etcd, partially undoing the isolation just completed. A hot-tier move keeps the SSD footprint and the rebuild-I/O burst small while capturing the single largest and churniest contributor (Prometheus = 11.6 GB = 44 % of the whole working set) and all the latency-sensitive Postgres DBs.


3. Options evaluated

Add a second, SSD-backed Longhorn data disk per node (a new virtio3 zvol on local-zfs), tag it ssd, tag the existing HDD disk hdd, publish a longhorn-ssd StorageClass with diskSelector: ["ssd"], and migrate the hot volumes onto it. Cold config volumes stay on the HDD default-disk.

  • Hot set: prometheus-…-db-0 (monitoring), storage-loki-0 (monitoring), keycloak-db-1, nextcloud-db-1, semaphore-db-1 (the three CNPG Postgres DBs), gitea-data. ~16.9 GB/node actual, 38 GiB provisioned.
  • Why these: Prometheus is the biggest and churniest (TSDB compaction + snapshot bloat — 11.6 GB actual on an 8 GiB nominal volume). Postgres DBs are fsync-latency sensitive and small. Loki and gitea are moderate-churn. Moving these removes the bulk of rebuild bytes and the latency-sensitive I/O from the HDD.
  • Footprint: small and bounded → the I/O-coupling risk on rpool is minimal.
  • Downside: the ~12 cold config volumes (~9 GB/node) still rebuild on the HDD on a reboot — but they are tiny and low-churn, so the storm shrinks from 18 volumes / ~26 GB to ~12 volumes / ~9 GB, and the SSD ones rebuild fast.

3.2 Option B — Full move (all 18 Longhorn volumes to SSD)

Size the SSD disk per node for the full working set (e.g. 96 GiB thin, lower over-provisioning) and repoint the default StorageClass / migrate every volume.

  • Pro: eliminates HDD Longhorn rebuild entirely — reboots no longer touch the HDD for storage; SSD rebuild is fast and low-IO-delay.
  • Con: re-couples all Longhorn rebuild I/O with etcd + host root on rpool, partially undoing docs/plans/ssd-etcd-isolation.md. SSD IOPS make this far less harmful than the HDD case, but it is a real regression of the isolation just done. Also consumes more of the small SSD pool's headroom.
  • Verdict: feasible, but hold it as a Phase 2 only if the hot-tier move proves insufficient. Not the opening move.

3.3 Option C — Do nothing; wait for the all-SSD mini-PCs

Zero work, zero risk, but leaves the cluster fragile to every node bounce indefinitely — the mini-PC migration has no committed date. What that path requires: new hardware (all-SSD mini-PCs), a fresh Proxmox/k3s build, and a full Longhorn data migration anyway. Option A is a cheap, reversible bridge that the mini-PC path later supersedes — it does not waste that work (the longhorn-ssd class and tagging carry over).

3.4 The I/O-coupling caveat (applies to A and B)

rpool is the only SSD pool. It already carries the host root and the isolated etcd disks. Putting Longhorn rebuild I/O back on it re-introduces some contention with etcd's WAL fsync — the exact thing the etcd disk was meant to escape. Two facts keep Option A safe: (1) SSD random IOPS are ~2–3 orders of magnitude above the HDD, so concurrent fsync + rebuild is tolerable where it was fatal on spinning rust; (2) the hot-tier footprint is small so rebuild bursts are short. Mitigation: keep concurrent-replica-rebuild-per-node-limit: 1, and watch etcd_disk_wal_fsync_duration_seconds (Prometheus) during the first SSD rebuild — if p99 fsync climbs materially, stop and reassess before proceeding to the next node.


Every step below is a proposal. Nothing here has been executed. Apply per node, lowest-quorum-risk node first, verifying Longhorn/etcd health between nodes.

Phase 0 — Pre-flight (read-only, do before touching anything)

  1. Re-verify SSD free space: on the Proxmox host zpool list rpool and pvesm status (confirm ≳ 100 GiB AVAIL remains after the etcd zvols).
  2. Collect HDD SMART: smartctl -a on both Storage-pool members + zpool status -x. If a member is degraded, note the added urgency but the mirror + 3× replication mean the plan is unchanged.
  3. Confirm Longhorn is fully healthy (all volumes robustness: healthy) and no rebuild is in flight. Do not start during a rebuild.
  4. Snapshot the CNPG DBs (CNPG can take an on-demand base backup) and note that gitea / Prometheus / Loki data is reproducible or non-critical — belt-and-braces before any volume move, since there is no Longhorn backup target.

Phase 1 — Add an SSD Longhorn disk per node (tofu, mirrors the etcd-disk pattern)

The VMs are bpg/OpenTofu-managed (infra/tofu/main.tf, resource "proxmox_virtual_environment_vm" "node", for_each = local.nodes). Disks are disk {} blocks; virtio0 (OS) and virtio1 (Longhorn HDD) are on var.datastore (Storage), virtio2 (etcd) is on var.etcd_ssd_datastore (local-zfs). Add a fourth disk on the SSD datastore, kept last so bpg does not reorder existing disks.

  1. New variables in infra/tofu/variables.tf:
    variable "longhorn_ssd_datastore" {
      description = "SSD-backed datastore for the tagged Longhorn SSD disk (local-zfs = rpool/data, Micron 5100 SATA SSD mirror w/ PLP)."
      type        = string
      default     = "local-zfs"
    }
    variable "longhorn_ssd_disk_gib" {
      description = "Per-node SSD Longhorn disk, GiB. Hot tier is ~17 GiB actual / 38 GiB provisioned; 48 GiB gives headroom without crowding rpool."
      type        = number
      default     = 48
    }
    
  2. New disk {} block in the VM resource in main.tf, appended after the virtio2 etcd block (ascending interface → virtio3 → /dev/vdd):
    # Tagged SSD Longhorn disk (virtio3 -> /dev/vdd) for the hot tier (Prometheus, CNPG
    # DBs, Loki, gitea). See docs/plans/longhorn-ssd-tiering.md. Kept last so bpg won't
    # reorder virtio0/1/2. discard=on so ext4/Longhorn TRIM reclaims thin zvol space.
    disk {
      datastore_id = var.longhorn_ssd_datastore
      interface    = "virtio3"
      size         = var.longhorn_ssd_disk_gib
      discard      = "on"
      cache        = "none"
      iothread     = false
    }
    
  3. tofu plan must show an in-place disk add (update) for each node — not a replacement. If it shows replace, stop (never destroy a running etcd node).
  4. Apply one node at a time with -target on the name-keyed resource address, e.g. -target='proxmox_virtual_environment_vm.node["k3s-dev-3"]'. A disk add is applied online by bpg, but treat the node as maintenance-affected regardless.

Phase 2 — Format + mount the SSD disk (ansible node_baseline, mirrors etcd)

roles/node_baseline already formats/mounts the HDD Longhorn disk (/dev/vdb → /var/lib/longhorn) and the etcd disk (/dev/vdc). Add the same idempotent, data-safe pattern for /dev/vdd:

  1. New defaults in roles/node_baseline/defaults/main.yml:
    node_baseline_longhorn_ssd_device: /dev/vdd
    node_baseline_longhorn_ssd_mount: /var/lib/longhorn-ssd
    node_baseline_longhorn_ssd_fs_label: longhorn-ssd
    
  2. New tasks in roles/node_baseline/tasks/main.yml (copy the HDD Longhorn filesystem+mount pair; community.general.filesystem never reformats an existing fs, so re-runs are safe):
    - name: Create ext4 filesystem on the SSD Longhorn disk (data-safe; skips if present)
      community.general.filesystem:
        fstype: ext4
        dev: "{{ node_baseline_longhorn_ssd_device }}"
        opts: "-L {{ node_baseline_longhorn_ssd_fs_label }}"
    - name: Mount the SSD Longhorn disk
      ansible.posix.mount:
        path: "{{ node_baseline_longhorn_ssd_mount }}"
        src: "LABEL={{ node_baseline_longhorn_ssd_fs_label }}"
        fstype: ext4
        opts: "defaults,nofail,x-systemd.device-timeout=5s"
        state: mounted
    
    Use LABEL= (not /dev/vdd) so the fstab entry survives device renames, and nofail so a missing SSD Longhorn disk does not block boot (unlike etcd, an absent Longhorn extra-disk should not wedge the node).

Phase 3 — Register + tag the disks in Longhorn

Per node (via the Longhorn nodes.longhorn.io CR or UI):

  1. Tag the existing HDD disk default-disk with hdd and set the new longhorn-ssd StorageClass's counterpart default class to select it (so cold volumes stay put and nothing hot accidentally lands on HDD once the SSD class exists).
  2. Add the SSD disk at /var/lib/longhorn-ssd as a new Longhorn disk on each node, allowScheduling: true, tag ssd.
    # patch each nodes.longhorn.io/<node> .spec.disks
    disk-ssd:
      path: /var/lib/longhorn-ssd
      allowScheduling: true
      tags: ["ssd"]
      storageReserved: 0
    
  3. Confirm each node now reports two disks with correct storageMaximum (~48 GiB for the SSD one) and Schedulable condition true.

Phase 4 — Publish the longhorn-ssd StorageClass

New GitOps manifest (wherever the Longhorn StorageClasses are declared in platform/ — mirror the existing longhorn class and add the selector):

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-ssd
provisioner: driver.longhorn.io
allowVolumeExpansion: true
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "30"
  fsType: ext4
  dataLocality: disabled
  diskSelector: "ssd"          # only schedules replicas on ssd-tagged disks
  disableRevisionCounter: "true"
  unmapMarkSnapChainRemoved: "ignored"

Optionally add a matching diskSelector: "hdd" to the existing default longhorn class so newly-created cold volumes stay on HDD.

Phase 5 — Migrate the hot volumes (no Longhorn backup target → in-cluster methods)

Because no backup target exists (§1.4), use one of these, one volume at a time, respecting concurrent-replica-rebuild-per-node-limit: 1:

  • CNPG DBs (keycloak / nextcloud / semaphore) — cleanest via the operator. Change the CNPG Cluster .spec.storage.storageClass to longhorn-ssd and let CNPG do a rolling replica re-creation (it provisions a fresh instance on the new class, streams from the primary, then promotes) — no manual data copy, no downtime beyond a switchover. Do one CNPG cluster at a time; verify replication caught up before the next.
  • Prometheus / Loki / gitea — per-volume diskSelector rebuild (in place, no PVC swap). Patch the existing Longhorn volume to select the SSD disk and let Longhorn rebuild the replicas onto it, then drop the HDD replicas:
    kubectl -n longhorn-system patch volumes.longhorn.io <vol> --type merge \
      -p '{"spec":{"diskSelector":["ssd"]}}'
    
    Then trigger a controlled rebuild by deleting one HDD replica at a time (Longhorn reschedules onto an ssd-tagged disk; replica-auto-balance: best-effort helps). Watch the volume return to robustness: healthy before touching the next replica or the next volume. This avoids recreating PVCs and keeps the app running.
    • Alternative if the in-place rebuild misbehaves: scale the app to 0, CSI-clone the PVC onto longhorn-ssd (dataSource = existing PVC), repoint the workload to the clone, scale back up. More disruptive; keep as fallback.

Phase 6 — Validate

  1. All hot volumes robustness: healthy, replicas only on ssd-tagged disks (kubectl -n longhorn-system get replicas.longhorn.io -o wide).
  2. SSD Longhorn disk usage ≈ 17 GiB/node; HDD disk down to the cold ~9 GiB/node.
  3. Deliberately reboot one node and confirm the rebuild storm is materially smaller (fewer volumes, SSD-fast) and Proxmox IO-delay stays low. Confirm etcd fsync p99 did not regress during the SSD rebuild (§3.4).

5. Rebuild-storm mitigation (built into the plan)

  • One node at a time for every phase (tofu -target, ansible --limit, one CNPG cluster / one replica at a time) — etcd quorum is 3-node, never take two down.
  • concurrent-replica-rebuild-per-node-limit: 1 already set — a rebooted node rebuilds serially, capping IO-delay.
  • Migrate during a quiet window, never while another rebuild is in flight.
  • Lower storage-over-provisioning-percentage for the small SSD disk sizing — 200 % on a 48 GiB disk would let Longhorn schedule ~96 GiB of nominal claims onto ~17 GiB of real data; keep the hot set well under storageMaximum, and rely on storage-minimal-available-percentage: 25 as the floor. (The setting is global — consider dropping it to ~120–150 % cluster-wide, or simply keep the SSD disk sized so 200 % of the hot claims still fits.)
  • Verify etcd fsync latency during the first SSD rebuild before proceeding (§3.4).

6. Risks and rollback

Risk Likelihood Mitigation / rollback
SSD rebuild I/O re-starves etcd on shared rpool Low (SSD IOPS) rebuild-limit 1; watch etcd_disk_wal_fsync_duration; stop after node 1 if p99 regresses
SSD pool fills (host root 430 G + etcd + Longhorn on 888 G) Low (hot set ~51 GB) keep SSD disk small (48 GiB), storage-minimal-available-percentage: 25; re-check zpool list between nodes
tofu shows replace instead of disk-add Medium if module drift inspect tofu plan per node; abort on replace — never destroy a running etcd node
Volume-migration data loss (no backup target) Low CNPG native rolling; per-volume rebuild keeps HDD replica until SSD healthy; CNPG base backup + reproducible data taken in Phase 0
Data disk device rename (/dev/vdd shifts) Low mount by LABEL=longhorn-ssd, not device path
HDD actually failing mid-migration Unknown (SMART unread) 3× replication + mirror; Phase 0 SMART check; hot data lands on SSD first, reducing exposure

Rollback: every phase is reversible. To back out: migrate the hot volumes back to the HDD default-disk (reverse the diskSelector patch / point CNPG back at longhorn), remove the disk-ssd from each Longhorn node, delete the longhorn-ssd StorageClass, then remove the virtio3 disk block from tofu and re-apply one node at a time. The SSD zvols are thin and independent of the HDD data, so removal is clean.


7. Recommendation

Adopt Option A (hot-tier move), phased, one node at a time. The recorded "SSD doesn't fit Longhorn" blocker was based on the 384 GiB provisioned figure; the actual working set is ~79 GB cluster-wide (~26 GB/node) because all bulk media lives on NFS and every Longhorn volume is a small config/DB volume. The hot tier (Prometheus + the three CNPG Postgres DBs + Loki + gitea) is ~51 GB actual / 114 GiB provisioned across three nodes and fits the ~323 GiB SSD AVAIL with wide headroom, while capturing the single largest and churniest rebuild contributor (Prometheus, 44 % of the working set) and the latency-sensitive databases.

A full move is feasible on space and would fully eliminate HDD Longhorn rebuilds, but it re-couples all rebuild I/O with etcd + host root on the only SSD pool, partially undoing the just-completed etcd isolation — hold it as an optional Phase 2 gated on the hot-tier move proving insufficient. Option C (wait for mini-PCs) is not a solution tonight — no committed date, and the tagging + longhorn-ssd class built here carry over to that migration anyway.

Do this first, before any apply: re-verify live SSD free space (zpool list rpool, pvesm status) and read HDD SMART (smartctl -a, zpool status -x). If the Seagate/Storage pool shows a degraded member, the urgency rises and the hot-tier move should be scheduled sooner — but the mirror + 3× Longhorn replication mean the plan is not gated on the disk's health.

This document is a plan only. No storage, tofu, Longhorn, StorageClass, or live infrastructure has been modified.