From a04251bbc8db8e2d548c31183afd812df0034995 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 31 Aug 2026 08:27:21 -0500 Subject: [PATCH] docs: plan for Longhorn SSD tiering (hot-tier move off HDD) Investigation + phased implementation plan to move the churn-heavy, latency-sensitive Longhorn volumes (Prometheus, the CNPG Postgres DBs, Loki, gitea) off the slow/failing HDD 'Storage' pool onto a tagged SSD-backed Longhorn disk on local-zfs, ending the reboot rebuild storm. Key finding: the recorded 'SSD does not fit Longhorn' blocker was based on the 384 GiB provisioned figure; actual working set is ~79 GB cluster-wide (~26 GB/node) since all bulk media is on NFS, not Longhorn. Hot tier is ~51 GB actual / 114 GiB provisioned and fits the ~323 GiB SSD AVAIL with headroom. Read-only investigation; no live storage touched. Co-Authored-By: Claude Opus 4.8 --- docs/plans/longhorn-ssd-tiering.md | 468 +++++++++++++++++++++++++++++ 1 file changed, 468 insertions(+) create mode 100644 docs/plans/longhorn-ssd-tiering.md diff --git a/docs/plans/longhorn-ssd-tiering.md b/docs/plans/longhorn-ssd-tiering.md new file mode 100644 index 0000000..dbb30d2 --- /dev/null +++ b/docs/plans/longhorn-ssd-tiering.md @@ -0,0 +1,468 @@ +# Longhorn SSD tiering for the k3s dev cluster + +**Status:** PROPOSAL — awaiting user approval. No storage, tofu, Longhorn, or live +infrastructure has been changed. This is a read-only investigation + reviewable plan. +**Author:** investigation agent (read-only), 2026-08-31 +**Problem:** All 18 Longhorn data volumes live on the slow/failing Seagate-class HDD +pool (`Storage`). Every node reboot marks that node's replicas stale and triggers a +full 18-volume rebuild that spikes Proxmox IO-delay to 30 %+ and runs for a long time, +making every upgrade / resize / reboot expensive and risky. etcd was already isolated +onto a dedicated per-node SSD disk (`virtio2`, `local-zfs`) for exactly this reason +(`docs/plans/ssd-etcd-isolation.md`). The recorded blocker was that the SSD "does not +fit" the 3 × 128 GiB Longhorn data disks, so Longhorn stayed on HDD "interim until the +all-SSD mini-PCs land." + +**Verdict (short):** The "doesn't fit" blocker was computed against the **provisioned** +128 GiB disks. The **actual working set is ~26 GB per node (~79 GB across all three +nodes)** — bulk media is on NFS, *not* Longhorn, so every Longhorn volume is a small +config/DB volume. **A hot-tier move onto SSD is viable and recommended**; a full move +is also feasible on space but re-couples rebuild I/O with etcd/host on the same SSD +pool. Recommendation: **Phase 1 hot-tier move (Prometheus + the CNPG DBs + Loki + +gitea) onto a tagged `longhorn-ssd` disk**, one node at a time. + +--- + +## 1. Measured facts (all collected read-only, 2026-08-31) + +### 1.1 The Longhorn working set — provisioned vs *actual* + +`kubectl -n longhorn-system get volumes.longhorn.io` — capacity (`spec.size`) vs +**actual consumed** (`status.actualSize`, the real on-disk footprint of one replica, +snapshots included). All 18 volumes are `numberOfReplicas: 3`, so each node holds one +replica of each and a node's on-disk footprint ≈ the actualSize column below. + +| Volume / PVC | Namespace | App class | Provisioned | **Actual (per replica)** | Tier | +|---|---|---|---:|---:|---| +| prometheus-…-db-0 | monitoring | Prometheus TSDB | 8 GiB | **11.59 GB** | HOT | +| unifi-rootdisk | unifi | KubeVirt VM disk (`longhorn-vm`) | 40 GiB | 6.20 GB | cold | +| nextcloud-data | nextcloud | file data | 20 GiB | 1.44 GB | cold | +| nextcloud-db-1 | nextcloud-db | CNPG Postgres | 5 GiB | 1.41 GB | HOT | +| storage-loki-0 | monitoring | Loki chunks/index | 5 GiB | 1.24 GB | HOT | +| keycloak-db-1 | keycloak | CNPG Postgres | 5 GiB | 0.93 GB | HOT | +| gitea-data | gitea | git repos + SQLite/meta | 10 GiB | 0.90 GB | HOT | +| semaphore-db-1 | semaphore-db | CNPG Postgres | 5 GiB | 0.79 GB | HOT | +| crafty-data | crafty | Minecraft world | 20 GiB | 0.54 GB | cold | +| plex-config | plex | Plex metadata/db | 8 GiB | 0.23 GB | cold | +| seerr-config | seerr | config | 5 GiB | 0.17 GB | cold | +| sonarr-config | sonarr | config db | 5 GiB | 0.15 GB | cold | +| qbittorrent-config | qbittorrent | config | 5 GiB | 0.15 GB | cold | +| radarr-config | radarr | config db | 5 GiB | 0.15 GB | cold | +| pihole-config | pihole | config | 2 GiB | 0.10 GB | cold | +| jackett-config | jackett | config | 2 GiB | 0.10 GB | cold | +| seerr/act-runner-data | gitea-actions | CI cache | 1 GiB | 0.05 GB | cold | +| semaphore-config | semaphore | config | 1 GiB | 0.05 GB | cold | +| **TOTALS (per node / one replica)** | | | **152 GiB** | **~26.2 GB** | | + +- **Per node working set: ~26.2 GB actual** vs **152 GiB provisioned** (× 3 replicas = + 456 GiB nominal, but Longhorn/ZFS are thin — nominal is not what lands on disk). +- **Whole-cluster working set: ~79 GB** across the three nodes. +- **Bulk media is not on Longhorn.** The StorageClass survey (§1.3) shows Plex media + libraries and the *arr download shares use the `bulk` NFS class + (`nfs.csi.k8s.io`), not Longhorn. So the "bulk stays on HDD while small/hot moves to + SSD" framing mostly dissolves: **there is no large Longhorn volume.** The biggest is + `unifi-rootdisk` at 6.2 GB actual, and it is a cold KubeVirt root disk. + +### 1.2 Longhorn per-node disk (the current single HDD data disk) + +`kubectl -n longhorn-system get nodes.longhorn.io` (all three nodes identical): + +| Field | Value | +|---|---| +| Disk | `default-disk`, path `/var/lib/longhorn`, tags `[]`, `allowScheduling: true` | +| `storageMaximum` | 124.9 GiB (the ext4 fs on the 128 GiB `virtio1` HDD disk) | +| `storageAvailable` | 100.4 GiB free | +| `storageScheduled` | **152.0 GiB** — over-scheduled vs the 124.9 GiB max | + +The 152 GiB *scheduled* > 124.9 GiB *max* is Longhorn honouring +`storage-over-provisioning-percentage: 200`. Actual usage is only ~24.5 GiB +(124.9 − 100.4), matching §1.1. + +### 1.3 StorageClasses in play + +`kubectl get storageclass`: + +| Class | Provisioner | Notes | +|---|---|---| +| `longhorn` | `driver.longhorn.io` | default class, 3 replicas, ext4 — **17 of 18 PVCs** | +| `longhorn-vm` | `driver.longhorn.io` | 3 replicas, `migratable: true` — `unifi-rootdisk` only | +| `longhorn-scratch` | `driver.longhorn.io` | 1 replica, best-effort locality (ephemeral) | +| `longhorn-static` | `driver.longhorn.io` | static-provisioning class | +| `bulk` | `nfs.csi.k8s.io` | NFS — bulk media (unresolved `REPLACE-ME` server/share) | + +### 1.4 Longhorn settings relevant to a migration / rebuild storm + +`kubectl -n longhorn-system get settings.longhorn.io …`: + +| Setting | Value | Relevance | +|---|---|---| +| `concurrent-replica-rebuild-per-node-limit` | **1** | already throttled — one rebuild at a time per node (set during etcd work) | +| `storage-over-provisioning-percentage` | 200 | dangerous on a *small* SSD disk — must lower for the SSD disk sizing | +| `storage-minimal-available-percentage` | 25 | Longhorn stops scheduling when a disk is <25 % free — a safety floor | +| `replica-auto-balance` | best-effort | will help spread replicas after a new disk appears | +| `default-replica-count` | 3 | — | + +Longhorn version: **v1.12.1**. **No backup target is configured** +(`backuptargets.longhorn.io/default`: URL empty, `available: false`) — so any +migration path that relies on Longhorn *backup → restore-to-new-SC* is **not +available** until an S3/NFS backup target is set up. This constrains the migration +mechanism (§4) to in-cluster options: CSI clone, per-volume `diskSelector` rebuild, or +app-level redeploy. + +### 1.5 The SSD pool (`local-zfs` = `rpool/data`) + +Measured read-only on 2026-08-29 during the etcd-isolation work +(`docs/plans/ssd-etcd-isolation.md` §1.1) and unchanged in substance since — the only +addition was the three 16 GiB etcd disks on 2026-08-30, which are thin and hold ~83 MB +of etcd data each (~250 MB total actual): + +| Property | Value | +|---|---| +| Pool | `rpool`, mirror of 2 × Micron 5100/52x0 960 GB **enterprise SATA SSD w/ PLP** | +| Health | 97 % lifetime remaining, 0 reallocated sectors — excellent | +| `zpool` size / free | 888 G / **~350 G free** (60 % capacity) | +| ZFS `AVAIL` to a new dataset/zvol | **~323 GiB** | +| Proxmox host root on it | `rpool/ROOT/pve-1` = 430 G | +| Storage entry | `local-zfs`, content `images,rootdir`, `sparse 1` (thin) | +| Already carries | 3 × 16 GiB etcd zvols (thin, ~250 MB actual) | + +> **Re-verify at execution time.** Before applying, re-run `zpool list rpool` and +> `pvesm status` on the Proxmox host to confirm current free space (this figure is +> dated 2026-08-29). The vault-gated ansible path (`proxmox_hosts` group) or a Proxmox +> UI check both work. + +### 1.6 Is the HDD *failing* — data-loss urgency? + +The problem statement calls the Seagate HDD "slow/failing." **SMART was not +re-collected in this investigation** (the Proxmox host query is vault-gated in the +dev container). The etcd-isolation doc characterised `Storage` as 2 × WDC WUH721818 +18 TB (healthy) and `Storage2` as 2 × Seagate ST8000VN004 — the Longhorn `virtio1` +disks are on `Storage` (the WD mirror), a ZFS **mirror**, so a single-disk SMART fault +is not immediate data loss. **Action item for the executor:** collect +`smartctl -a` for both `Storage`-pool members and `zpool status -x` before scheduling; +if a member is degraded, that raises urgency and argues for moving the *hot* data off +sooner. Treat "failing" as unconfirmed until SMART is read — but the mirror + 3-way +Longhorn replication means the plan is not gated on it. + +--- + +## 2. Capacity verdict — does it fit? + +| Scenario | Per-node SSD footprint (actual) | ×3 across nodes | Provisioned to size the disk | Fits in ~323 GiB SSD `AVAIL`? | +|---|---:|---:|---:|---| +| **Hot tier** (Prometheus + 3 CNPG DBs + Loki + gitea) | **~16.9 GB** | **~50.6 GB** | 38 GiB/node → suggest a **48 GiB** SSD disk/node | **Yes, comfortably** (~145 GiB of thin zvols max, ~51 GB actual) | +| **Full move** (all 18 Longhorn volumes) | ~26.2 GB | ~79 GB | 152 GiB/node nominal, but thin | **Yes on space** (~79 GB actual) — the "doesn't fit" blocker was a *provisioned*-number artefact | +| All-SSD mini-PC path (no move now) | — | — | — | N/A — hardware-blocked, no committed date | + +**The recorded blocker is overturned by the actual numbers.** 3 × 128 GiB = 384 GiB +*provisioned* does not fit next to the 430 GiB host root on an 888 GiB pool — that is +true and is why the earlier doc said "NO for Longhorn." But the **real data is ~79 GB**, +and both ZFS (`sparse 1`) and Longhorn are thin, so what actually lands on the SSD is +the working set, not the nominal disk size. Both a hot-tier move and a full move fit on +free space today. + +The reason to still prefer the **hot tier** over a full move is **not space** — it is +**I/O coupling** (see §3.4 and §6): the only SSD pool is `rpool`, which now also carries +the Proxmox host root *and* the freshly-isolated etcd disks. A full move puts every +Longhorn rebuild back onto the same pool as etcd, partially undoing the isolation just +completed. A hot-tier move keeps the SSD footprint and the rebuild-I/O burst small while +capturing the single largest and churniest contributor (Prometheus = 11.6 GB = 44 % of +the whole working set) and all the latency-sensitive Postgres DBs. + +--- + +## 3. Options evaluated + +### 3.1 Option A — Hot-tier move onto a tagged SSD Longhorn disk (RECOMMENDED) + +Add a second, SSD-backed Longhorn data disk per node (a new `virtio3` zvol on +`local-zfs`), tag it `ssd`, tag the existing HDD disk `hdd`, publish a `longhorn-ssd` +StorageClass with `diskSelector: ["ssd"]`, and migrate the hot volumes onto it. Cold +config volumes stay on the HDD `default-disk`. + +- **Hot set:** `prometheus-…-db-0` (monitoring), `storage-loki-0` (monitoring), + `keycloak-db-1`, `nextcloud-db-1`, `semaphore-db-1` (the three CNPG Postgres DBs), + `gitea-data`. ~16.9 GB/node actual, 38 GiB provisioned. +- **Why these:** Prometheus is the biggest and churniest (TSDB compaction + snapshot + bloat — 11.6 GB actual on an 8 GiB nominal volume). Postgres DBs are fsync-latency + sensitive and small. Loki and gitea are moderate-churn. Moving these removes the bulk + of rebuild bytes *and* the latency-sensitive I/O from the HDD. +- **Footprint:** small and bounded → the I/O-coupling risk on `rpool` is minimal. +- **Downside:** the ~12 cold config volumes (~9 GB/node) still rebuild on the HDD on a + reboot — but they are tiny and low-churn, so the storm shrinks from 18 volumes / + ~26 GB to ~12 volumes / ~9 GB, and the SSD ones rebuild fast. + +### 3.2 Option B — Full move (all 18 Longhorn volumes to SSD) + +Size the SSD disk per node for the full working set (e.g. 96 GiB thin, lower +over-provisioning) and repoint the default StorageClass / migrate every volume. + +- **Pro:** eliminates HDD Longhorn rebuild entirely — reboots no longer touch the HDD + for storage; SSD rebuild is fast and low-IO-delay. +- **Con:** re-couples *all* Longhorn rebuild I/O with etcd + host root on `rpool`, + partially undoing `docs/plans/ssd-etcd-isolation.md`. SSD IOPS make this far less + harmful than the HDD case, but it is a real regression of the isolation just done. + Also consumes more of the small SSD pool's headroom. +- **Verdict:** feasible, but hold it as a *Phase 2* only if the hot-tier move proves + insufficient. Not the opening move. + +### 3.3 Option C — Do nothing; wait for the all-SSD mini-PCs + +Zero work, zero risk, but leaves the cluster fragile to every node bounce +indefinitely — the mini-PC migration has no committed date. What that path *requires*: +new hardware (all-SSD mini-PCs), a fresh Proxmox/k3s build, and a full Longhorn data +migration anyway. Option A is a cheap, reversible bridge that the mini-PC path later +supersedes — it does not waste that work (the `longhorn-ssd` class and tagging carry +over). + +### 3.4 The I/O-coupling caveat (applies to A and B) + +`rpool` is the **only** SSD pool. It already carries the host root and the isolated +etcd disks. Putting Longhorn rebuild I/O back on it re-introduces *some* contention +with etcd's WAL fsync — the exact thing the etcd disk was meant to escape. Two facts +keep Option A safe: (1) SSD random IOPS are ~2–3 orders of magnitude above the HDD, so +concurrent fsync + rebuild is tolerable where it was fatal on spinning rust; (2) the +hot-tier footprint is small so rebuild bursts are short. **Mitigation:** keep +`concurrent-replica-rebuild-per-node-limit: 1`, and watch +`etcd_disk_wal_fsync_duration_seconds` (Prometheus) during the first SSD rebuild — if +p99 fsync climbs materially, stop and reassess before proceeding to the next node. + +--- + +## 4. Recommended implementation — Option A, phased, one node at a time + +> Every step below is a *proposal*. Nothing here has been executed. Apply per node, +> lowest-quorum-risk node first, verifying Longhorn/etcd health between nodes. + +### Phase 0 — Pre-flight (read-only, do before touching anything) + +1. Re-verify SSD free space: on the Proxmox host `zpool list rpool` and `pvesm status` + (confirm ≳ 100 GiB `AVAIL` remains after the etcd zvols). +2. Collect HDD SMART: `smartctl -a` on both `Storage`-pool members + `zpool status -x`. + If a member is degraded, note the added urgency but the mirror + 3× replication mean + the plan is unchanged. +3. Confirm Longhorn is fully healthy (all volumes `robustness: healthy`) and no rebuild + is in flight. Do not start during a rebuild. +4. Snapshot the CNPG DBs (CNPG can take an on-demand base backup) and note that gitea / + Prometheus / Loki data is reproducible or non-critical — belt-and-braces before any + volume move, since there is no Longhorn backup target. + +### Phase 1 — Add an SSD Longhorn disk per node (tofu, mirrors the etcd-disk pattern) + +The VMs are bpg/OpenTofu-managed (`infra/tofu/main.tf`, +`resource "proxmox_virtual_environment_vm" "node"`, `for_each = local.nodes`). Disks +are `disk {}` blocks; `virtio0` (OS) and `virtio1` (Longhorn HDD) are on `var.datastore` +(`Storage`), `virtio2` (etcd) is on `var.etcd_ssd_datastore` (`local-zfs`). Add a +**fourth disk on the SSD datastore**, kept last so bpg does not reorder existing disks. + +1. New variables in `infra/tofu/variables.tf`: + ```hcl + variable "longhorn_ssd_datastore" { + description = "SSD-backed datastore for the tagged Longhorn SSD disk (local-zfs = rpool/data, Micron 5100 SATA SSD mirror w/ PLP)." + type = string + default = "local-zfs" + } + variable "longhorn_ssd_disk_gib" { + description = "Per-node SSD Longhorn disk, GiB. Hot tier is ~17 GiB actual / 38 GiB provisioned; 48 GiB gives headroom without crowding rpool." + type = number + default = 48 + } + ``` +2. New `disk {}` block in the VM resource in `main.tf`, appended **after** the + `virtio2` etcd block (ascending interface → `virtio3` → `/dev/vdd`): + ```hcl + # Tagged SSD Longhorn disk (virtio3 -> /dev/vdd) for the hot tier (Prometheus, CNPG + # DBs, Loki, gitea). See docs/plans/longhorn-ssd-tiering.md. Kept last so bpg won't + # reorder virtio0/1/2. discard=on so ext4/Longhorn TRIM reclaims thin zvol space. + disk { + datastore_id = var.longhorn_ssd_datastore + interface = "virtio3" + size = var.longhorn_ssd_disk_gib + discard = "on" + cache = "none" + iothread = false + } + ``` +3. `tofu plan` **must** show an in-place *disk add* (update) for each node — **not** a + replacement. If it shows replace, stop (never destroy a running etcd node). +4. Apply **one node at a time** with `-target` on the name-keyed resource address, + e.g. `-target='proxmox_virtual_environment_vm.node["k3s-dev-3"]'`. A disk add is + applied online by bpg, but treat the node as maintenance-affected regardless. + +### Phase 2 — Format + mount the SSD disk (ansible `node_baseline`, mirrors etcd) + +`roles/node_baseline` already formats/mounts the HDD Longhorn disk (`/dev/vdb` → +`/var/lib/longhorn`) and the etcd disk (`/dev/vdc`). Add the same idempotent, +data-safe pattern for `/dev/vdd`: + +1. New defaults in `roles/node_baseline/defaults/main.yml`: + ```yaml + node_baseline_longhorn_ssd_device: /dev/vdd + node_baseline_longhorn_ssd_mount: /var/lib/longhorn-ssd + node_baseline_longhorn_ssd_fs_label: longhorn-ssd + ``` +2. New tasks in `roles/node_baseline/tasks/main.yml` (copy the HDD Longhorn + filesystem+mount pair; `community.general.filesystem` never reformats an existing + fs, so re-runs are safe): + ```yaml + - name: Create ext4 filesystem on the SSD Longhorn disk (data-safe; skips if present) + community.general.filesystem: + fstype: ext4 + dev: "{{ node_baseline_longhorn_ssd_device }}" + opts: "-L {{ node_baseline_longhorn_ssd_fs_label }}" + - name: Mount the SSD Longhorn disk + ansible.posix.mount: + path: "{{ node_baseline_longhorn_ssd_mount }}" + src: "LABEL={{ node_baseline_longhorn_ssd_fs_label }}" + fstype: ext4 + opts: "defaults,nofail,x-systemd.device-timeout=5s" + state: mounted + ``` + Use `LABEL=` (not `/dev/vdd`) so the fstab entry survives device renames, and + `nofail` so a missing SSD Longhorn disk does not block boot (unlike etcd, an absent + Longhorn extra-disk should not wedge the node). + +### Phase 3 — Register + tag the disks in Longhorn + +Per node (via the Longhorn `nodes.longhorn.io` CR or UI): + +1. **Tag the existing HDD disk** `default-disk` with `hdd` and set the new + `longhorn-ssd` StorageClass's counterpart default class to select it (so cold + volumes stay put and nothing hot accidentally lands on HDD once the SSD class + exists). +2. **Add the SSD disk** at `/var/lib/longhorn-ssd` as a new Longhorn disk on each node, + `allowScheduling: true`, tag `ssd`. + ```yaml + # patch each nodes.longhorn.io/ .spec.disks + disk-ssd: + path: /var/lib/longhorn-ssd + allowScheduling: true + tags: ["ssd"] + storageReserved: 0 + ``` +3. Confirm each node now reports two disks with correct `storageMaximum` (~48 GiB for + the SSD one) and `Schedulable` condition true. + +### Phase 4 — Publish the `longhorn-ssd` StorageClass + +New GitOps manifest (wherever the Longhorn StorageClasses are declared in +`platform/` — mirror the existing `longhorn` class and add the selector): + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: longhorn-ssd +provisioner: driver.longhorn.io +allowVolumeExpansion: true +parameters: + numberOfReplicas: "3" + staleReplicaTimeout: "30" + fsType: ext4 + dataLocality: disabled + diskSelector: "ssd" # only schedules replicas on ssd-tagged disks + disableRevisionCounter: "true" + unmapMarkSnapChainRemoved: "ignored" +``` + +Optionally add a matching `diskSelector: "hdd"` to the existing default `longhorn` +class so newly-created cold volumes stay on HDD. + +### Phase 5 — Migrate the hot volumes (no Longhorn backup target → in-cluster methods) + +Because no backup target exists (§1.4), use one of these, **one volume at a time**, +respecting `concurrent-replica-rebuild-per-node-limit: 1`: + +- **CNPG DBs (keycloak / nextcloud / semaphore) — cleanest via the operator.** Change + the CNPG `Cluster` `.spec.storage.storageClass` to `longhorn-ssd` and let CNPG do a + rolling replica re-creation (it provisions a fresh instance on the new class, streams + from the primary, then promotes) — no manual data copy, no downtime beyond a + switchover. Do one CNPG cluster at a time; verify replication caught up before the + next. +- **Prometheus / Loki / gitea — per-volume `diskSelector` rebuild (in place, no PVC + swap).** Patch the existing Longhorn volume to select the SSD disk and let Longhorn + rebuild the replicas onto it, then drop the HDD replicas: + ``` + kubectl -n longhorn-system patch volumes.longhorn.io --type merge \ + -p '{"spec":{"diskSelector":["ssd"]}}' + ``` + Then trigger a controlled rebuild by deleting **one** HDD replica at a time (Longhorn + reschedules onto an `ssd`-tagged disk; `replica-auto-balance: best-effort` helps). + Watch the volume return to `robustness: healthy` before touching the next replica or + the next volume. This avoids recreating PVCs and keeps the app running. + - Alternative if the in-place rebuild misbehaves: scale the app to 0, CSI-**clone** + the PVC onto `longhorn-ssd` (`dataSource` = existing PVC), repoint the workload to + the clone, scale back up. More disruptive; keep as fallback. + +### Phase 6 — Validate + +1. All hot volumes `robustness: healthy`, replicas only on `ssd`-tagged disks + (`kubectl -n longhorn-system get replicas.longhorn.io -o wide`). +2. SSD Longhorn disk usage ≈ 17 GiB/node; HDD disk down to the cold ~9 GiB/node. +3. Deliberately reboot **one** node and confirm the rebuild storm is materially smaller + (fewer volumes, SSD-fast) and Proxmox IO-delay stays low. Confirm etcd fsync p99 did + not regress during the SSD rebuild (§3.4). + +--- + +## 5. Rebuild-storm mitigation (built into the plan) + +- **One node at a time** for every phase (tofu `-target`, ansible `--limit`, one CNPG + cluster / one replica at a time) — etcd quorum is 3-node, never take two down. +- **`concurrent-replica-rebuild-per-node-limit: 1`** already set — a rebooted node + rebuilds serially, capping IO-delay. +- **Migrate during a quiet window**, never while another rebuild is in flight. +- **Lower `storage-over-provisioning-percentage`** *for the small SSD disk sizing* — + 200 % on a 48 GiB disk would let Longhorn schedule ~96 GiB of nominal claims onto + ~17 GiB of real data; keep the hot set well under `storageMaximum`, and rely on + `storage-minimal-available-percentage: 25` as the floor. (The setting is global — + consider dropping it to ~120–150 % cluster-wide, or simply keep the SSD disk sized so + 200 % of the hot claims still fits.) +- **Verify etcd fsync latency** during the first SSD rebuild before proceeding (§3.4). + +--- + +## 6. Risks and rollback + +| Risk | Likelihood | Mitigation / rollback | +|---|---|---| +| SSD rebuild I/O re-starves etcd on shared `rpool` | Low (SSD IOPS) | rebuild-limit 1; watch `etcd_disk_wal_fsync_duration`; stop after node 1 if p99 regresses | +| SSD pool fills (host root 430 G + etcd + Longhorn on 888 G) | Low (hot set ~51 GB) | keep SSD disk small (48 GiB), `storage-minimal-available-percentage: 25`; re-check `zpool list` between nodes | +| tofu shows *replace* instead of disk-add | Medium if module drift | inspect `tofu plan` per node; abort on replace — never destroy a running etcd node | +| Volume-migration data loss (no backup target) | Low | CNPG native rolling; per-volume rebuild keeps HDD replica until SSD healthy; CNPG base backup + reproducible data taken in Phase 0 | +| Data disk device rename (`/dev/vdd` shifts) | Low | mount by `LABEL=longhorn-ssd`, not device path | +| HDD actually failing mid-migration | Unknown (SMART unread) | 3× replication + mirror; Phase 0 SMART check; hot data lands on SSD first, reducing exposure | + +**Rollback:** every phase is reversible. To back out: migrate the hot volumes back to +the HDD `default-disk` (reverse the `diskSelector` patch / point CNPG back at +`longhorn`), remove the `disk-ssd` from each Longhorn node, delete the `longhorn-ssd` +StorageClass, then remove the `virtio3` disk block from tofu and re-apply one node at a +time. The SSD zvols are thin and independent of the HDD data, so removal is clean. + +--- + +## 7. Recommendation + +**Adopt Option A (hot-tier move), phased, one node at a time.** The recorded "SSD +doesn't fit Longhorn" blocker was based on the 384 GiB *provisioned* figure; the +**actual working set is ~79 GB cluster-wide (~26 GB/node)** because all bulk media +lives on NFS and every Longhorn volume is a small config/DB volume. The hot tier +(Prometheus + the three CNPG Postgres DBs + Loki + gitea) is **~51 GB actual / +114 GiB provisioned** across three nodes and fits the ~323 GiB SSD `AVAIL` with wide +headroom, while capturing the single largest and churniest rebuild contributor +(Prometheus, 44 % of the working set) and the latency-sensitive databases. + +A **full move is feasible on space** and would fully eliminate HDD Longhorn rebuilds, +but it re-couples all rebuild I/O with etcd + host root on the only SSD pool, partially +undoing the just-completed etcd isolation — hold it as an optional Phase 2 gated on the +hot-tier move proving insufficient. **Option C (wait for mini-PCs) is not a solution** +tonight — no committed date, and the tagging + `longhorn-ssd` class built here carry +over to that migration anyway. + +**Do this first, before any apply:** re-verify live SSD free space (`zpool list +rpool`, `pvesm status`) and read HDD SMART (`smartctl -a`, `zpool status -x`). If the +Seagate/`Storage` pool shows a degraded member, the urgency rises and the hot-tier move +should be scheduled sooner — but the mirror + 3× Longhorn replication mean the plan is +not gated on the disk's health. + +> This document is a plan only. No storage, tofu, Longhorn, StorageClass, or live +> infrastructure has been modified.