docs: plan for Longhorn SSD tiering (hot-tier move off HDD)
Investigation + phased implementation plan to move the churn-heavy, latency-sensitive Longhorn volumes (Prometheus, the CNPG Postgres DBs, Loki, gitea) off the slow/failing HDD 'Storage' pool onto a tagged SSD-backed Longhorn disk on local-zfs, ending the reboot rebuild storm. Key finding: the recorded 'SSD does not fit Longhorn' blocker was based on the 384 GiB provisioned figure; actual working set is ~79 GB cluster-wide (~26 GB/node) since all bulk media is on NFS, not Longhorn. Hot tier is ~51 GB actual / 114 GiB provisioned and fits the ~323 GiB SSD AVAIL with headroom. Read-only investigation; no live storage touched. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed
1 parent
d60e5531b1
commit
a04251bbc8
1 file changed
+468
@@ -0,0 +1,468 @@
|
||||
# Longhorn SSD tiering for the k3s dev cluster
|
||||
|
||||
**Status:** PROPOSAL — awaiting user approval. No storage, tofu, Longhorn, or live
|
||||
infrastructure has been changed. This is a read-only investigation + reviewable plan.
|
||||
**Author:** investigation agent (read-only), 2026-08-31
|
||||
**Problem:** All 18 Longhorn data volumes live on the slow/failing Seagate-class HDD
|
||||
pool (`Storage`). Every node reboot marks that node's replicas stale and triggers a
|
||||
full 18-volume rebuild that spikes Proxmox IO-delay to 30 %+ and runs for a long time,
|
||||
making every upgrade / resize / reboot expensive and risky. etcd was already isolated
|
||||
onto a dedicated per-node SSD disk (`virtio2`, `local-zfs`) for exactly this reason
|
||||
(`docs/plans/ssd-etcd-isolation.md`). The recorded blocker was that the SSD "does not
|
||||
fit" the 3 × 128 GiB Longhorn data disks, so Longhorn stayed on HDD "interim until the
|
||||
all-SSD mini-PCs land."
|
||||
|
||||
**Verdict (short):** The "doesn't fit" blocker was computed against the **provisioned**
|
||||
128 GiB disks. The **actual working set is ~26 GB per node (~79 GB across all three
|
||||
nodes)** — bulk media is on NFS, *not* Longhorn, so every Longhorn volume is a small
|
||||
config/DB volume. **A hot-tier move onto SSD is viable and recommended**; a full move
|
||||
is also feasible on space but re-couples rebuild I/O with etcd/host on the same SSD
|
||||
pool. Recommendation: **Phase 1 hot-tier move (Prometheus + the CNPG DBs + Loki +
|
||||
gitea) onto a tagged `longhorn-ssd` disk**, one node at a time.
|
||||
|
||||
---
|
||||
|
||||
## 1. Measured facts (all collected read-only, 2026-08-31)
|
||||
|
||||
### 1.1 The Longhorn working set — provisioned vs *actual*
|
||||
|
||||
`kubectl -n longhorn-system get volumes.longhorn.io` — capacity (`spec.size`) vs
|
||||
**actual consumed** (`status.actualSize`, the real on-disk footprint of one replica,
|
||||
snapshots included). All 18 volumes are `numberOfReplicas: 3`, so each node holds one
|
||||
replica of each and a node's on-disk footprint ≈ the actualSize column below.
|
||||
|
||||
| Volume / PVC | Namespace | App class | Provisioned | **Actual (per replica)** | Tier |
|
||||
|---|---|---|---:|---:|---|
|
||||
| prometheus-…-db-0 | monitoring | Prometheus TSDB | 8 GiB | **11.59 GB** | HOT |
|
||||
| unifi-rootdisk | unifi | KubeVirt VM disk (`longhorn-vm`) | 40 GiB | 6.20 GB | cold |
|
||||
| nextcloud-data | nextcloud | file data | 20 GiB | 1.44 GB | cold |
|
||||
| nextcloud-db-1 | nextcloud-db | CNPG Postgres | 5 GiB | 1.41 GB | HOT |
|
||||
| storage-loki-0 | monitoring | Loki chunks/index | 5 GiB | 1.24 GB | HOT |
|
||||
| keycloak-db-1 | keycloak | CNPG Postgres | 5 GiB | 0.93 GB | HOT |
|
||||
| gitea-data | gitea | git repos + SQLite/meta | 10 GiB | 0.90 GB | HOT |
|
||||
| semaphore-db-1 | semaphore-db | CNPG Postgres | 5 GiB | 0.79 GB | HOT |
|
||||
| crafty-data | crafty | Minecraft world | 20 GiB | 0.54 GB | cold |
|
||||
| plex-config | plex | Plex metadata/db | 8 GiB | 0.23 GB | cold |
|
||||
| seerr-config | seerr | config | 5 GiB | 0.17 GB | cold |
|
||||
| sonarr-config | sonarr | config db | 5 GiB | 0.15 GB | cold |
|
||||
| qbittorrent-config | qbittorrent | config | 5 GiB | 0.15 GB | cold |
|
||||
| radarr-config | radarr | config db | 5 GiB | 0.15 GB | cold |
|
||||
| pihole-config | pihole | config | 2 GiB | 0.10 GB | cold |
|
||||
| jackett-config | jackett | config | 2 GiB | 0.10 GB | cold |
|
||||
| seerr/act-runner-data | gitea-actions | CI cache | 1 GiB | 0.05 GB | cold |
|
||||
| semaphore-config | semaphore | config | 1 GiB | 0.05 GB | cold |
|
||||
| **TOTALS (per node / one replica)** | | | **152 GiB** | **~26.2 GB** | |
|
||||
|
||||
- **Per node working set: ~26.2 GB actual** vs **152 GiB provisioned** (× 3 replicas =
|
||||
456 GiB nominal, but Longhorn/ZFS are thin — nominal is not what lands on disk).
|
||||
- **Whole-cluster working set: ~79 GB** across the three nodes.
|
||||
- **Bulk media is not on Longhorn.** The StorageClass survey (§1.3) shows Plex media
|
||||
libraries and the *arr download shares use the `bulk` NFS class
|
||||
(`nfs.csi.k8s.io`), not Longhorn. So the "bulk stays on HDD while small/hot moves to
|
||||
SSD" framing mostly dissolves: **there is no large Longhorn volume.** The biggest is
|
||||
`unifi-rootdisk` at 6.2 GB actual, and it is a cold KubeVirt root disk.
|
||||
|
||||
### 1.2 Longhorn per-node disk (the current single HDD data disk)
|
||||
|
||||
`kubectl -n longhorn-system get nodes.longhorn.io` (all three nodes identical):
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Disk | `default-disk`, path `/var/lib/longhorn`, tags `[]`, `allowScheduling: true` |
|
||||
| `storageMaximum` | 124.9 GiB (the ext4 fs on the 128 GiB `virtio1` HDD disk) |
|
||||
| `storageAvailable` | 100.4 GiB free |
|
||||
| `storageScheduled` | **152.0 GiB** — over-scheduled vs the 124.9 GiB max |
|
||||
|
||||
The 152 GiB *scheduled* > 124.9 GiB *max* is Longhorn honouring
|
||||
`storage-over-provisioning-percentage: 200`. Actual usage is only ~24.5 GiB
|
||||
(124.9 − 100.4), matching §1.1.
|
||||
|
||||
### 1.3 StorageClasses in play
|
||||
|
||||
`kubectl get storageclass`:
|
||||
|
||||
| Class | Provisioner | Notes |
|
||||
|---|---|---|
|
||||
| `longhorn` | `driver.longhorn.io` | default class, 3 replicas, ext4 — **17 of 18 PVCs** |
|
||||
| `longhorn-vm` | `driver.longhorn.io` | 3 replicas, `migratable: true` — `unifi-rootdisk` only |
|
||||
| `longhorn-scratch` | `driver.longhorn.io` | 1 replica, best-effort locality (ephemeral) |
|
||||
| `longhorn-static` | `driver.longhorn.io` | static-provisioning class |
|
||||
| `bulk` | `nfs.csi.k8s.io` | NFS — bulk media (unresolved `REPLACE-ME` server/share) |
|
||||
|
||||
### 1.4 Longhorn settings relevant to a migration / rebuild storm
|
||||
|
||||
`kubectl -n longhorn-system get settings.longhorn.io …`:
|
||||
|
||||
| Setting | Value | Relevance |
|
||||
|---|---|---|
|
||||
| `concurrent-replica-rebuild-per-node-limit` | **1** | already throttled — one rebuild at a time per node (set during etcd work) |
|
||||
| `storage-over-provisioning-percentage` | 200 | dangerous on a *small* SSD disk — must lower for the SSD disk sizing |
|
||||
| `storage-minimal-available-percentage` | 25 | Longhorn stops scheduling when a disk is <25 % free — a safety floor |
|
||||
| `replica-auto-balance` | best-effort | will help spread replicas after a new disk appears |
|
||||
| `default-replica-count` | 3 | — |
|
||||
|
||||
Longhorn version: **v1.12.1**. **No backup target is configured**
|
||||
(`backuptargets.longhorn.io/default`: URL empty, `available: false`) — so any
|
||||
migration path that relies on Longhorn *backup → restore-to-new-SC* is **not
|
||||
available** until an S3/NFS backup target is set up. This constrains the migration
|
||||
mechanism (§4) to in-cluster options: CSI clone, per-volume `diskSelector` rebuild, or
|
||||
app-level redeploy.
|
||||
|
||||
### 1.5 The SSD pool (`local-zfs` = `rpool/data`)
|
||||
|
||||
Measured read-only on 2026-08-29 during the etcd-isolation work
|
||||
(`docs/plans/ssd-etcd-isolation.md` §1.1) and unchanged in substance since — the only
|
||||
addition was the three 16 GiB etcd disks on 2026-08-30, which are thin and hold ~83 MB
|
||||
of etcd data each (~250 MB total actual):
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Pool | `rpool`, mirror of 2 × Micron 5100/52x0 960 GB **enterprise SATA SSD w/ PLP** |
|
||||
| Health | 97 % lifetime remaining, 0 reallocated sectors — excellent |
|
||||
| `zpool` size / free | 888 G / **~350 G free** (60 % capacity) |
|
||||
| ZFS `AVAIL` to a new dataset/zvol | **~323 GiB** |
|
||||
| Proxmox host root on it | `rpool/ROOT/pve-1` = 430 G |
|
||||
| Storage entry | `local-zfs`, content `images,rootdir`, `sparse 1` (thin) |
|
||||
| Already carries | 3 × 16 GiB etcd zvols (thin, ~250 MB actual) |
|
||||
|
||||
> **Re-verify at execution time.** Before applying, re-run `zpool list rpool` and
|
||||
> `pvesm status` on the Proxmox host to confirm current free space (this figure is
|
||||
> dated 2026-08-29). The vault-gated ansible path (`proxmox_hosts` group) or a Proxmox
|
||||
> UI check both work.
|
||||
|
||||
### 1.6 Is the HDD *failing* — data-loss urgency?
|
||||
|
||||
The problem statement calls the Seagate HDD "slow/failing." **SMART was not
|
||||
re-collected in this investigation** (the Proxmox host query is vault-gated in the
|
||||
dev container). The etcd-isolation doc characterised `Storage` as 2 × WDC WUH721818
|
||||
18 TB (healthy) and `Storage2` as 2 × Seagate ST8000VN004 — the Longhorn `virtio1`
|
||||
disks are on `Storage` (the WD mirror), a ZFS **mirror**, so a single-disk SMART fault
|
||||
is not immediate data loss. **Action item for the executor:** collect
|
||||
`smartctl -a` for both `Storage`-pool members and `zpool status -x` before scheduling;
|
||||
if a member is degraded, that raises urgency and argues for moving the *hot* data off
|
||||
sooner. Treat "failing" as unconfirmed until SMART is read — but the mirror + 3-way
|
||||
Longhorn replication means the plan is not gated on it.
|
||||
|
||||
---
|
||||
|
||||
## 2. Capacity verdict — does it fit?
|
||||
|
||||
| Scenario | Per-node SSD footprint (actual) | ×3 across nodes | Provisioned to size the disk | Fits in ~323 GiB SSD `AVAIL`? |
|
||||
|---|---:|---:|---:|---|
|
||||
| **Hot tier** (Prometheus + 3 CNPG DBs + Loki + gitea) | **~16.9 GB** | **~50.6 GB** | 38 GiB/node → suggest a **48 GiB** SSD disk/node | **Yes, comfortably** (~145 GiB of thin zvols max, ~51 GB actual) |
|
||||
| **Full move** (all 18 Longhorn volumes) | ~26.2 GB | ~79 GB | 152 GiB/node nominal, but thin | **Yes on space** (~79 GB actual) — the "doesn't fit" blocker was a *provisioned*-number artefact |
|
||||
| All-SSD mini-PC path (no move now) | — | — | — | N/A — hardware-blocked, no committed date |
|
||||
|
||||
**The recorded blocker is overturned by the actual numbers.** 3 × 128 GiB = 384 GiB
|
||||
*provisioned* does not fit next to the 430 GiB host root on an 888 GiB pool — that is
|
||||
true and is why the earlier doc said "NO for Longhorn." But the **real data is ~79 GB**,
|
||||
and both ZFS (`sparse 1`) and Longhorn are thin, so what actually lands on the SSD is
|
||||
the working set, not the nominal disk size. Both a hot-tier move and a full move fit on
|
||||
free space today.
|
||||
|
||||
The reason to still prefer the **hot tier** over a full move is **not space** — it is
|
||||
**I/O coupling** (see §3.4 and §6): the only SSD pool is `rpool`, which now also carries
|
||||
the Proxmox host root *and* the freshly-isolated etcd disks. A full move puts every
|
||||
Longhorn rebuild back onto the same pool as etcd, partially undoing the isolation just
|
||||
completed. A hot-tier move keeps the SSD footprint and the rebuild-I/O burst small while
|
||||
capturing the single largest and churniest contributor (Prometheus = 11.6 GB = 44 % of
|
||||
the whole working set) and all the latency-sensitive Postgres DBs.
|
||||
|
||||
---
|
||||
|
||||
## 3. Options evaluated
|
||||
|
||||
### 3.1 Option A — Hot-tier move onto a tagged SSD Longhorn disk (RECOMMENDED)
|
||||
|
||||
Add a second, SSD-backed Longhorn data disk per node (a new `virtio3` zvol on
|
||||
`local-zfs`), tag it `ssd`, tag the existing HDD disk `hdd`, publish a `longhorn-ssd`
|
||||
StorageClass with `diskSelector: ["ssd"]`, and migrate the hot volumes onto it. Cold
|
||||
config volumes stay on the HDD `default-disk`.
|
||||
|
||||
- **Hot set:** `prometheus-…-db-0` (monitoring), `storage-loki-0` (monitoring),
|
||||
`keycloak-db-1`, `nextcloud-db-1`, `semaphore-db-1` (the three CNPG Postgres DBs),
|
||||
`gitea-data`. ~16.9 GB/node actual, 38 GiB provisioned.
|
||||
- **Why these:** Prometheus is the biggest and churniest (TSDB compaction + snapshot
|
||||
bloat — 11.6 GB actual on an 8 GiB nominal volume). Postgres DBs are fsync-latency
|
||||
sensitive and small. Loki and gitea are moderate-churn. Moving these removes the bulk
|
||||
of rebuild bytes *and* the latency-sensitive I/O from the HDD.
|
||||
- **Footprint:** small and bounded → the I/O-coupling risk on `rpool` is minimal.
|
||||
- **Downside:** the ~12 cold config volumes (~9 GB/node) still rebuild on the HDD on a
|
||||
reboot — but they are tiny and low-churn, so the storm shrinks from 18 volumes /
|
||||
~26 GB to ~12 volumes / ~9 GB, and the SSD ones rebuild fast.
|
||||
|
||||
### 3.2 Option B — Full move (all 18 Longhorn volumes to SSD)
|
||||
|
||||
Size the SSD disk per node for the full working set (e.g. 96 GiB thin, lower
|
||||
over-provisioning) and repoint the default StorageClass / migrate every volume.
|
||||
|
||||
- **Pro:** eliminates HDD Longhorn rebuild entirely — reboots no longer touch the HDD
|
||||
for storage; SSD rebuild is fast and low-IO-delay.
|
||||
- **Con:** re-couples *all* Longhorn rebuild I/O with etcd + host root on `rpool`,
|
||||
partially undoing `docs/plans/ssd-etcd-isolation.md`. SSD IOPS make this far less
|
||||
harmful than the HDD case, but it is a real regression of the isolation just done.
|
||||
Also consumes more of the small SSD pool's headroom.
|
||||
- **Verdict:** feasible, but hold it as a *Phase 2* only if the hot-tier move proves
|
||||
insufficient. Not the opening move.
|
||||
|
||||
### 3.3 Option C — Do nothing; wait for the all-SSD mini-PCs
|
||||
|
||||
Zero work, zero risk, but leaves the cluster fragile to every node bounce
|
||||
indefinitely — the mini-PC migration has no committed date. What that path *requires*:
|
||||
new hardware (all-SSD mini-PCs), a fresh Proxmox/k3s build, and a full Longhorn data
|
||||
migration anyway. Option A is a cheap, reversible bridge that the mini-PC path later
|
||||
supersedes — it does not waste that work (the `longhorn-ssd` class and tagging carry
|
||||
over).
|
||||
|
||||
### 3.4 The I/O-coupling caveat (applies to A and B)
|
||||
|
||||
`rpool` is the **only** SSD pool. It already carries the host root and the isolated
|
||||
etcd disks. Putting Longhorn rebuild I/O back on it re-introduces *some* contention
|
||||
with etcd's WAL fsync — the exact thing the etcd disk was meant to escape. Two facts
|
||||
keep Option A safe: (1) SSD random IOPS are ~2–3 orders of magnitude above the HDD, so
|
||||
concurrent fsync + rebuild is tolerable where it was fatal on spinning rust; (2) the
|
||||
hot-tier footprint is small so rebuild bursts are short. **Mitigation:** keep
|
||||
`concurrent-replica-rebuild-per-node-limit: 1`, and watch
|
||||
`etcd_disk_wal_fsync_duration_seconds` (Prometheus) during the first SSD rebuild — if
|
||||
p99 fsync climbs materially, stop and reassess before proceeding to the next node.
|
||||
|
||||
---
|
||||
|
||||
## 4. Recommended implementation — Option A, phased, one node at a time
|
||||
|
||||
> Every step below is a *proposal*. Nothing here has been executed. Apply per node,
|
||||
> lowest-quorum-risk node first, verifying Longhorn/etcd health between nodes.
|
||||
|
||||
### Phase 0 — Pre-flight (read-only, do before touching anything)
|
||||
|
||||
1. Re-verify SSD free space: on the Proxmox host `zpool list rpool` and `pvesm status`
|
||||
(confirm ≳ 100 GiB `AVAIL` remains after the etcd zvols).
|
||||
2. Collect HDD SMART: `smartctl -a` on both `Storage`-pool members + `zpool status -x`.
|
||||
If a member is degraded, note the added urgency but the mirror + 3× replication mean
|
||||
the plan is unchanged.
|
||||
3. Confirm Longhorn is fully healthy (all volumes `robustness: healthy`) and no rebuild
|
||||
is in flight. Do not start during a rebuild.
|
||||
4. Snapshot the CNPG DBs (CNPG can take an on-demand base backup) and note that gitea /
|
||||
Prometheus / Loki data is reproducible or non-critical — belt-and-braces before any
|
||||
volume move, since there is no Longhorn backup target.
|
||||
|
||||
### Phase 1 — Add an SSD Longhorn disk per node (tofu, mirrors the etcd-disk pattern)
|
||||
|
||||
The VMs are bpg/OpenTofu-managed (`infra/tofu/main.tf`,
|
||||
`resource "proxmox_virtual_environment_vm" "node"`, `for_each = local.nodes`). Disks
|
||||
are `disk {}` blocks; `virtio0` (OS) and `virtio1` (Longhorn HDD) are on `var.datastore`
|
||||
(`Storage`), `virtio2` (etcd) is on `var.etcd_ssd_datastore` (`local-zfs`). Add a
|
||||
**fourth disk on the SSD datastore**, kept last so bpg does not reorder existing disks.
|
||||
|
||||
1. New variables in `infra/tofu/variables.tf`:
|
||||
```hcl
|
||||
variable "longhorn_ssd_datastore" {
|
||||
description = "SSD-backed datastore for the tagged Longhorn SSD disk (local-zfs = rpool/data, Micron 5100 SATA SSD mirror w/ PLP)."
|
||||
type = string
|
||||
default = "local-zfs"
|
||||
}
|
||||
variable "longhorn_ssd_disk_gib" {
|
||||
description = "Per-node SSD Longhorn disk, GiB. Hot tier is ~17 GiB actual / 38 GiB provisioned; 48 GiB gives headroom without crowding rpool."
|
||||
type = number
|
||||
default = 48
|
||||
}
|
||||
```
|
||||
2. New `disk {}` block in the VM resource in `main.tf`, appended **after** the
|
||||
`virtio2` etcd block (ascending interface → `virtio3` → `/dev/vdd`):
|
||||
```hcl
|
||||
# Tagged SSD Longhorn disk (virtio3 -> /dev/vdd) for the hot tier (Prometheus, CNPG
|
||||
# DBs, Loki, gitea). See docs/plans/longhorn-ssd-tiering.md. Kept last so bpg won't
|
||||
# reorder virtio0/1/2. discard=on so ext4/Longhorn TRIM reclaims thin zvol space.
|
||||
disk {
|
||||
datastore_id = var.longhorn_ssd_datastore
|
||||
interface = "virtio3"
|
||||
size = var.longhorn_ssd_disk_gib
|
||||
discard = "on"
|
||||
cache = "none"
|
||||
iothread = false
|
||||
}
|
||||
```
|
||||
3. `tofu plan` **must** show an in-place *disk add* (update) for each node — **not** a
|
||||
replacement. If it shows replace, stop (never destroy a running etcd node).
|
||||
4. Apply **one node at a time** with `-target` on the name-keyed resource address,
|
||||
e.g. `-target='proxmox_virtual_environment_vm.node["k3s-dev-3"]'`. A disk add is
|
||||
applied online by bpg, but treat the node as maintenance-affected regardless.
|
||||
|
||||
### Phase 2 — Format + mount the SSD disk (ansible `node_baseline`, mirrors etcd)
|
||||
|
||||
`roles/node_baseline` already formats/mounts the HDD Longhorn disk (`/dev/vdb` →
|
||||
`/var/lib/longhorn`) and the etcd disk (`/dev/vdc`). Add the same idempotent,
|
||||
data-safe pattern for `/dev/vdd`:
|
||||
|
||||
1. New defaults in `roles/node_baseline/defaults/main.yml`:
|
||||
```yaml
|
||||
node_baseline_longhorn_ssd_device: /dev/vdd
|
||||
node_baseline_longhorn_ssd_mount: /var/lib/longhorn-ssd
|
||||
node_baseline_longhorn_ssd_fs_label: longhorn-ssd
|
||||
```
|
||||
2. New tasks in `roles/node_baseline/tasks/main.yml` (copy the HDD Longhorn
|
||||
filesystem+mount pair; `community.general.filesystem` never reformats an existing
|
||||
fs, so re-runs are safe):
|
||||
```yaml
|
||||
- name: Create ext4 filesystem on the SSD Longhorn disk (data-safe; skips if present)
|
||||
community.general.filesystem:
|
||||
fstype: ext4
|
||||
dev: "{{ node_baseline_longhorn_ssd_device }}"
|
||||
opts: "-L {{ node_baseline_longhorn_ssd_fs_label }}"
|
||||
- name: Mount the SSD Longhorn disk
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_longhorn_ssd_mount }}"
|
||||
src: "LABEL={{ node_baseline_longhorn_ssd_fs_label }}"
|
||||
fstype: ext4
|
||||
opts: "defaults,nofail,x-systemd.device-timeout=5s"
|
||||
state: mounted
|
||||
```
|
||||
Use `LABEL=` (not `/dev/vdd`) so the fstab entry survives device renames, and
|
||||
`nofail` so a missing SSD Longhorn disk does not block boot (unlike etcd, an absent
|
||||
Longhorn extra-disk should not wedge the node).
|
||||
|
||||
### Phase 3 — Register + tag the disks in Longhorn
|
||||
|
||||
Per node (via the Longhorn `nodes.longhorn.io` CR or UI):
|
||||
|
||||
1. **Tag the existing HDD disk** `default-disk` with `hdd` and set the new
|
||||
`longhorn-ssd` StorageClass's counterpart default class to select it (so cold
|
||||
volumes stay put and nothing hot accidentally lands on HDD once the SSD class
|
||||
exists).
|
||||
2. **Add the SSD disk** at `/var/lib/longhorn-ssd` as a new Longhorn disk on each node,
|
||||
`allowScheduling: true`, tag `ssd`.
|
||||
```yaml
|
||||
# patch each nodes.longhorn.io/<node> .spec.disks
|
||||
disk-ssd:
|
||||
path: /var/lib/longhorn-ssd
|
||||
allowScheduling: true
|
||||
tags: ["ssd"]
|
||||
storageReserved: 0
|
||||
```
|
||||
3. Confirm each node now reports two disks with correct `storageMaximum` (~48 GiB for
|
||||
the SSD one) and `Schedulable` condition true.
|
||||
|
||||
### Phase 4 — Publish the `longhorn-ssd` StorageClass
|
||||
|
||||
New GitOps manifest (wherever the Longhorn StorageClasses are declared in
|
||||
`platform/` — mirror the existing `longhorn` class and add the selector):
|
||||
|
||||
```yaml
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: longhorn-ssd
|
||||
provisioner: driver.longhorn.io
|
||||
allowVolumeExpansion: true
|
||||
parameters:
|
||||
numberOfReplicas: "3"
|
||||
staleReplicaTimeout: "30"
|
||||
fsType: ext4
|
||||
dataLocality: disabled
|
||||
diskSelector: "ssd" # only schedules replicas on ssd-tagged disks
|
||||
disableRevisionCounter: "true"
|
||||
unmapMarkSnapChainRemoved: "ignored"
|
||||
```
|
||||
|
||||
Optionally add a matching `diskSelector: "hdd"` to the existing default `longhorn`
|
||||
class so newly-created cold volumes stay on HDD.
|
||||
|
||||
### Phase 5 — Migrate the hot volumes (no Longhorn backup target → in-cluster methods)
|
||||
|
||||
Because no backup target exists (§1.4), use one of these, **one volume at a time**,
|
||||
respecting `concurrent-replica-rebuild-per-node-limit: 1`:
|
||||
|
||||
- **CNPG DBs (keycloak / nextcloud / semaphore) — cleanest via the operator.** Change
|
||||
the CNPG `Cluster` `.spec.storage.storageClass` to `longhorn-ssd` and let CNPG do a
|
||||
rolling replica re-creation (it provisions a fresh instance on the new class, streams
|
||||
from the primary, then promotes) — no manual data copy, no downtime beyond a
|
||||
switchover. Do one CNPG cluster at a time; verify replication caught up before the
|
||||
next.
|
||||
- **Prometheus / Loki / gitea — per-volume `diskSelector` rebuild (in place, no PVC
|
||||
swap).** Patch the existing Longhorn volume to select the SSD disk and let Longhorn
|
||||
rebuild the replicas onto it, then drop the HDD replicas:
|
||||
```
|
||||
kubectl -n longhorn-system patch volumes.longhorn.io <vol> --type merge \
|
||||
-p '{"spec":{"diskSelector":["ssd"]}}'
|
||||
```
|
||||
Then trigger a controlled rebuild by deleting **one** HDD replica at a time (Longhorn
|
||||
reschedules onto an `ssd`-tagged disk; `replica-auto-balance: best-effort` helps).
|
||||
Watch the volume return to `robustness: healthy` before touching the next replica or
|
||||
the next volume. This avoids recreating PVCs and keeps the app running.
|
||||
- Alternative if the in-place rebuild misbehaves: scale the app to 0, CSI-**clone**
|
||||
the PVC onto `longhorn-ssd` (`dataSource` = existing PVC), repoint the workload to
|
||||
the clone, scale back up. More disruptive; keep as fallback.
|
||||
|
||||
### Phase 6 — Validate
|
||||
|
||||
1. All hot volumes `robustness: healthy`, replicas only on `ssd`-tagged disks
|
||||
(`kubectl -n longhorn-system get replicas.longhorn.io -o wide`).
|
||||
2. SSD Longhorn disk usage ≈ 17 GiB/node; HDD disk down to the cold ~9 GiB/node.
|
||||
3. Deliberately reboot **one** node and confirm the rebuild storm is materially smaller
|
||||
(fewer volumes, SSD-fast) and Proxmox IO-delay stays low. Confirm etcd fsync p99 did
|
||||
not regress during the SSD rebuild (§3.4).
|
||||
|
||||
---
|
||||
|
||||
## 5. Rebuild-storm mitigation (built into the plan)
|
||||
|
||||
- **One node at a time** for every phase (tofu `-target`, ansible `--limit`, one CNPG
|
||||
cluster / one replica at a time) — etcd quorum is 3-node, never take two down.
|
||||
- **`concurrent-replica-rebuild-per-node-limit: 1`** already set — a rebooted node
|
||||
rebuilds serially, capping IO-delay.
|
||||
- **Migrate during a quiet window**, never while another rebuild is in flight.
|
||||
- **Lower `storage-over-provisioning-percentage`** *for the small SSD disk sizing* —
|
||||
200 % on a 48 GiB disk would let Longhorn schedule ~96 GiB of nominal claims onto
|
||||
~17 GiB of real data; keep the hot set well under `storageMaximum`, and rely on
|
||||
`storage-minimal-available-percentage: 25` as the floor. (The setting is global —
|
||||
consider dropping it to ~120–150 % cluster-wide, or simply keep the SSD disk sized so
|
||||
200 % of the hot claims still fits.)
|
||||
- **Verify etcd fsync latency** during the first SSD rebuild before proceeding (§3.4).
|
||||
|
||||
---
|
||||
|
||||
## 6. Risks and rollback
|
||||
|
||||
| Risk | Likelihood | Mitigation / rollback |
|
||||
|---|---|---|
|
||||
| SSD rebuild I/O re-starves etcd on shared `rpool` | Low (SSD IOPS) | rebuild-limit 1; watch `etcd_disk_wal_fsync_duration`; stop after node 1 if p99 regresses |
|
||||
| SSD pool fills (host root 430 G + etcd + Longhorn on 888 G) | Low (hot set ~51 GB) | keep SSD disk small (48 GiB), `storage-minimal-available-percentage: 25`; re-check `zpool list` between nodes |
|
||||
| tofu shows *replace* instead of disk-add | Medium if module drift | inspect `tofu plan` per node; abort on replace — never destroy a running etcd node |
|
||||
| Volume-migration data loss (no backup target) | Low | CNPG native rolling; per-volume rebuild keeps HDD replica until SSD healthy; CNPG base backup + reproducible data taken in Phase 0 |
|
||||
| Data disk device rename (`/dev/vdd` shifts) | Low | mount by `LABEL=longhorn-ssd`, not device path |
|
||||
| HDD actually failing mid-migration | Unknown (SMART unread) | 3× replication + mirror; Phase 0 SMART check; hot data lands on SSD first, reducing exposure |
|
||||
|
||||
**Rollback:** every phase is reversible. To back out: migrate the hot volumes back to
|
||||
the HDD `default-disk` (reverse the `diskSelector` patch / point CNPG back at
|
||||
`longhorn`), remove the `disk-ssd` from each Longhorn node, delete the `longhorn-ssd`
|
||||
StorageClass, then remove the `virtio3` disk block from tofu and re-apply one node at a
|
||||
time. The SSD zvols are thin and independent of the HDD data, so removal is clean.
|
||||
|
||||
---
|
||||
|
||||
## 7. Recommendation
|
||||
|
||||
**Adopt Option A (hot-tier move), phased, one node at a time.** The recorded "SSD
|
||||
doesn't fit Longhorn" blocker was based on the 384 GiB *provisioned* figure; the
|
||||
**actual working set is ~79 GB cluster-wide (~26 GB/node)** because all bulk media
|
||||
lives on NFS and every Longhorn volume is a small config/DB volume. The hot tier
|
||||
(Prometheus + the three CNPG Postgres DBs + Loki + gitea) is **~51 GB actual /
|
||||
114 GiB provisioned** across three nodes and fits the ~323 GiB SSD `AVAIL` with wide
|
||||
headroom, while capturing the single largest and churniest rebuild contributor
|
||||
(Prometheus, 44 % of the working set) and the latency-sensitive databases.
|
||||
|
||||
A **full move is feasible on space** and would fully eliminate HDD Longhorn rebuilds,
|
||||
but it re-couples all rebuild I/O with etcd + host root on the only SSD pool, partially
|
||||
undoing the just-completed etcd isolation — hold it as an optional Phase 2 gated on the
|
||||
hot-tier move proving insufficient. **Option C (wait for mini-PCs) is not a solution**
|
||||
tonight — no committed date, and the tagging + `longhorn-ssd` class built here carry
|
||||
over to that migration anyway.
|
||||
|
||||
**Do this first, before any apply:** re-verify live SSD free space (`zpool list
|
||||
rpool`, `pvesm status`) and read HDD SMART (`smartctl -a`, `zpool status -x`). If the
|
||||
Seagate/`Storage` pool shows a degraded member, the urgency rises and the hot-tier move
|
||||
should be scheduled sooner — but the mirror + 3× Longhorn replication mean the plan is
|
||||
not gated on the disk's health.
|
||||
|
||||
> This document is a plan only. No storage, tofu, Longhorn, StorageClass, or live
|
||||
> infrastructure has been modified.
|
||||
Reference in new issue
Block a user