jakman85
863af01062
infra/PROD-PROVISIONING.md §2 recorded the etcd/Longhorn shared-partition risk as "accepted to be MEASURED, not assumed", with an exit condition. The measurement arrived on 2026-09-07 in production, 2 d 7 h after bring-up, as an outage. §2 now carries "THE MEASUREMENT CAME BACK": the journal evidence, the three-node IO PSI table (11.36 / 0.75 / 1.29), and the honest verdict that the section's own premise — "every prod node is SSD or NVMe, where rebuild I/O costs latency rather than collapsing it" — was right about node 2 and wrong about node 1. A 2011 C400 with 61,440 reallocated sectors and no power-loss protection is not the "SSD" that reasoning assumed, and §0 had already named the three-way drive asymmetry as the fault line. The exit condition is OPEN and the second bullet (dedicated device) applies; that remains the operator's. Also records what the three software mitigations in this branch do and do not buy, including the point that no cgroup or scheduler control arbitrates the ext4 journal both tenants share — and the gap that etcd's own fsync histograms, which the exit condition literally asks for, are not collected because kubeEtcd is disabled for k3s. PSI is the signal that exists. DECISIONS-LOG gains the full entry: the four I/O mechanisms found INERT on these nodes (IOSchedulingClass, IOSchedulingPriority, io.latency, bare IOWeight), the finding that concurrentReplicaRebuildPerNodeLimit was already set and already held during the outage, the third instance of config that looked applied and was not (orphanAutoDeletion), and the three things still requiring the operator. infra/ansible/README.md: CORRECTED a stale claim that k3s-prod-2/-3 "stay commented out" of the inventory. They have been live in [k3s_prod_secondary] since 2026-09-03. That sentence is exactly the kind used to reason about blast radius — `-e k3s_hosts=k3s_prod` targets THREE hosts, not one — so it is replaced with the --list-hosts command that settles it. The new etcd-io-priority play is added to the run order, the layer table and the prod command list. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit a00a27a2567ea6a805c8c196b75df2c92a7c63a4)
320 lines
15 KiB
Markdown
320 lines
15 KiB
Markdown
# Decommission Proxmox `192.168.2.102` → TrueNAS SCALE on the N5 Air
|
|
|
|
**Status:** plan. **Drafted** 2026-09-06, **revised** 2026-09-06 after the operator settled the
|
|
target architecture.
|
|
|
|
| | |
|
|
|---|---|
|
|
| **N5 Air** | TrueNAS SCALE. **Storage only** — no VMs, no hypervisor. |
|
|
| **Compute** | k3s **prod** cluster (`192.168.2.10/.11/.12`). Plex, sonarr, radarr, jackett, qbittorrent, seerr and nextcloud are already modelled in `apps/wave-0/`. |
|
|
| **Dev k3s** | Does not move to the NAS. |
|
|
| **Proxmox host** | **Decommissioned.** |
|
|
| **Drives** | Physically relocated. None purchased. |
|
|
|
|
All figures measured on the live host, not inferred.
|
|
|
|
---
|
|
|
|
## Already done — 2026-09-06
|
|
|
|
**Reclaimed 3.56 TB.** `Storage/plex@pre-cleanup-20260828` destroyed after confirming no holds and
|
|
no clones. Async free drained in ~90 seconds.
|
|
|
|
| | before | after |
|
|
|---|---|---|
|
|
| `Storage` allocated | 12.3 T (75%) | **8.76 T (53%)** |
|
|
| `Storage` free | 4.04 T | **7.60 T** |
|
|
| `Storage/plex` used | 11.9 T | 8.33 T (0 B in snapshots) |
|
|
|
|
That free space is not a nicety — **it is what lets this whole migration happen with no network
|
|
transfer at all.** See §3.
|
|
|
|
---
|
|
|
|
## 1. The plan changed shape
|
|
|
|
The first draft carried the VMs across on their own zvols and put Proxmox on the NAS. With no
|
|
hypervisor on the far side, that inverts:
|
|
|
|
> **Every zvol is now dead weight, and every service's data has to be drained onto a plain dataset
|
|
> before the Proxmox box dies.**
|
|
|
|
Dead-weight zvols, once their contents are extracted:
|
|
|
|
| zvol | size | after extraction |
|
|
|---|---|---|
|
|
| `Storage:vm-101/102/105-disk-0,-disk-1` (dev k3s) | 585 G | delete |
|
|
| `Storage:vm-112-disk-0` (docker-services) | 65 G | delete |
|
|
| `Storage:vm-103-disk-0` (nextcloud boot) | 81 G | delete |
|
|
| `Storage2:vm-103-disk-0` (**nextcloud data**) | 4.06 T | extract, then delete |
|
|
|
|
The good news is the shape of the drain. Everything that must be rescued is either already on
|
|
`Storage`, or can be copied onto `Storage` **locally, inside the same chassis, onto the very drives
|
|
that are moving.** With 7.60 T free there is room. **Nothing crosses the network.**
|
|
|
|
---
|
|
|
|
## 2. Findings
|
|
|
|
### 2a. "The path lies" — two confirmed instances, assume a third
|
|
|
|
**Games — 422 GB, not on the Storage pool.** Challenged and re-verified; the test is decisive:
|
|
|
|
```
|
|
findmnt -T /mnt/Storage/Games → TARGET / SOURCE rpool/ROOT/pve-1
|
|
findmnt -T /mnt/Storage/plex → TARGET /mnt/Storage/plex SOURCE Storage/plex
|
|
```
|
|
|
|
`Games` is a plain directory on the Proxmox **root** filesystem that merely sits inside the Storage
|
|
mount tree. The pool accounting closes on it exactly:
|
|
|
|
```
|
|
rpool ALLOC 540 G = ROOT 431 G + data 105 G + var-lib-vz 3.75 G
|
|
ROOT 431 G = Games 422 G + /usr 3.5 G + /var 5 G + /boot 0.4 G
|
|
```
|
|
|
|
The two Microns are a **mirror**, so the pool is ~888 G usable, not ~1.9 T — 422 G is 48% of it, and
|
|
`zpool list` agrees at 60% capacity overall. Surprising, but the numbers do match.
|
|
|
|
**Gitea — 13 MB in an anonymous Docker volume, not where its compose file points.**
|
|
`/root/docker-compose.yml` in LXC 106 bind-mounts `/opt/gitea/data:/var/lib/data`. That path holds
|
|
**512 bytes**. The real Gitea state is in an unnamed Docker volume:
|
|
|
|
```
|
|
.../volumes/8a1d2616…/_data/git/repositories/ 12 M home-network.git, home-network-uds.git, jakman85/…
|
|
.../volumes/8a1d2616…/_data/data/gitea.db 11 M SQLite, mtime today 19:00 — live
|
|
```
|
|
|
|
Copying `/opt/gitea` would migrate an empty directory and silently lose all three repositories.
|
|
|
|
**Rule for the rest of the drain:** for every service, verify where the bytes actually are before
|
|
trusting the path in a compose file, an export list or a mount table. Two for two so far.
|
|
|
|
### 2b. Retiring pfSense is easier than rehoming it
|
|
|
|
`pfsense` is VM 100 and **is** `192.168.1.1`. Decommissioning the host takes the entire
|
|
`192.168.1.0/24` with it:
|
|
|
|
| | |
|
|
|---|---|
|
|
| `.1` pfSense · `.4` iDRAC · `.9` the Proxmox host itself | die with the box |
|
|
| `.5` nextcloud · `.6` plex · `.7` gitea · `.18` docker-services/Pi-hole | data drains, service moves to k3s |
|
|
| `.8` unifi · `.20` monitoring · `.21` auth · `.10` semaphore | rehome or retire (§2e) |
|
|
| `.49` kube-vip VIP · `.50/.51/.52` dev k3s | dev cluster is not moving; retire with the box |
|
|
|
|
**But pfSense exists only to contain that lab LAN.** Prod k3s is already on `192.168.2.0/24` —
|
|
pfSense's *WAN* side — and the NAS will be too. Once the lab is gone, nothing lives behind pfSense.
|
|
**Retire it; do not rehome it.** That is only clean because everything else is leaving `1.0/24`
|
|
anyway, which is exactly what this migration does.
|
|
|
|
**A documented blocker dissolves with it.** `infra/PROD-STANDUP.md` records that prod k3s cannot
|
|
reach the S3 backup endpoint at `192.168.1.9:9000` because prod sits on pfSense's WAN side and the
|
|
crossing rule does not exist. With the NAS on `192.168.2.0/24`, prod reaches its storage directly
|
|
and no pfSense rule is ever needed.
|
|
|
|
**Three things to close before pfSense goes:**
|
|
|
|
1. The SOHO router NATs inbound **443** to `192.168.2.102` for the legacy stack. It must be
|
|
repointed at prod's Gateway (`192.168.2.20`). `infra/PROD-PROVISIONING.md` §10a already flags
|
|
that the apex flip and the legacy cutover are the same event — **this migration is that event.**
|
|
2. **VLANs 5 and 123** (minecraft; the arr VM, which reaches Plex media as `10.123.1.10`) are
|
|
pfSense-side constructs. They disappear; the k3s apps need no VLAN.
|
|
3. The **OpenVPN split-tunnel egress gateway** currently runs on `k3s-dev-3` (ADR-0015). It is
|
|
already modelled for prod in `platform/vpn-egress/`, but confirm prod's gateway is live before
|
|
the dev node goes away.
|
|
|
|
### 2c. Nextcloud is the critical path
|
|
|
|
Its user data is **inside a filesystem, inside LVM, inside a 4 TB zvol, inside a VM**:
|
|
|
|
```
|
|
Storage2/vm-103-disk-0 → zd224 → PV /dev/zd224 → VG vg_storage → LV lv_storage (<4.00 T)
|
|
```
|
|
|
|
With no hypervisor on the far side this must be copied out **at the file level, while the VM is
|
|
still running** — it gates the decommission and it is the longest single task in the plan.
|
|
|
|
**One number is missing and I could not get it.** VM 103's QEMU guest agent is not running
|
|
(`qm agent 103 ping` → *"QEMU guest agent is not running"*) and its SSH does not accept the
|
|
`home_network` key. The zvol shows `referenced 3.94 T`, but it is attached **without `discard=on`**
|
|
and is thick-provisioned (`refreservation 4.06 T`), so deletions inside the guest were never
|
|
returned to ZFS — 3.94 T is an **upper bound**, possibly far above the truth.
|
|
|
|
**Run this inside the Nextcloud VM and the last open question closes:**
|
|
|
|
```bash
|
|
df -h / # or wherever lv_storage is mounted
|
|
lsblk -f /dev/mapper/vg_storage-lv_storage
|
|
```
|
|
|
|
Separately: `Storage/nextcloud` is 270 G, and **267 G of it is a directory called `config_backup`**
|
|
(`config` itself is 103 M). Almost certainly a stale dump — likely another easy reclaim, but confirm
|
|
before deleting.
|
|
|
|
### 2d. Storage2 — and the two oldest drives — can probably be retired
|
|
|
|
After the drain and the zvol deletions, `Storage` would hold roughly:
|
|
|
|
| | |
|
|
|---|---|
|
|
| plex | 8.33 T |
|
|
| nextcloud data | ≤ 3.94 T (real number pending §2c) |
|
|
| Games | 0.42 T |
|
|
| docker-volumes, other_containers, k8s-volumes, gitea | ~0.02 T |
|
|
| **total** | **≤ 12.7 T of 16.4 T (≤ 77%)** |
|
|
|
|
If the real Nextcloud figure comes in under ~2 T — which the missing-TRIM situation makes plausible
|
|
— everything fits on the 18 TB mirror at a comfortable ~65%, and **`Storage2` can be retired
|
|
entirely.** That would remove the two **6.0-year-old** Seagates, by a wide margin the most
|
|
failure-prone hardware in the estate, and free two bays. This is the single best safety outcome
|
|
available, and it costs nothing.
|
|
|
|
### 2e. Services that simply end
|
|
|
|
Decommissioning kills these; none has a home in the target architecture unless you give it one:
|
|
|
|
| guest | note |
|
|
|---|---|
|
|
| `docker-services` Pi-hole (`192.168.1.18`) | **LAN DNS.** Confirm what `192.168.2.0/24` uses for DNS before the box dies. |
|
|
| `unifi` (VM 121) | Network controller — needs a home or a cloud key. |
|
|
| `auth` (VM 122) | Superseded by Keycloak on prod k3s. |
|
|
| `monitoring` (VM 120) | Superseded by the k3s platform monitoring stack. |
|
|
| `semaphore` (LXC 110) | 432 MB resident of an 8 GB reservation; low value. |
|
|
| `minecraft` (LXC 108) | Operator call. Its 3.5 G subvol is on `Storage` and rides along. |
|
|
| **versitygw S3** on `192.168.1.9:9000` | **Velero + CNPG backup target.** Dies with the box. TrueNAS must take this role (MinIO app) or prod loses its backup destination — which `infra/PROD-STANDUP.md` already records as uncovered. |
|
|
|
|
### 2f. Still true from the first draft
|
|
|
|
- **No VM backups exist.** `/etc/pve/jobs.cfg` empty, no PBS, newest dump log April 2025.
|
|
- **`ata-WDC_…3FGHAB2T` has 36 ZFS read errors** on pristine SMART (0 reallocated / pending /
|
|
uncorrectable, `PASSED`) — points at cable or controller, not platter. Re-verify after the move.
|
|
- Drive ages: WD 18 TB pair **4.75 y**, Seagate 8 TB pair **6.0 y**.
|
|
|
|
---
|
|
|
|
## 3. TrueNAS SCALE specifics
|
|
|
|
**ZFS compatibility: verified, not assumed.** The only non-default features *active* on either pool
|
|
are `block_cloning`, `head_errlog`, `vdev_zaps_v2`, `zilsaxattr`, `zstd_compress`, `log_spacemap`,
|
|
`project_quota`, `spacemap_v2` — all OpenZFS **2.2** era or older. Nothing 2.3- or 2.4-only is
|
|
active, even though the pools were last written by `zfs-2.4.3-pve1`. Any TrueNAS SCALE from 24.10
|
|
onward imports them cleanly.
|
|
|
|
**Mountpoints mostly already match TrueNAS convention** — a lucky accident worth knowing:
|
|
|
|
| dataset | current mountpoint | TrueNAS wants | |
|
|
|---|---|---|---|
|
|
| `Storage/plex`, `/nextcloud`, `/docker-volumes`, `/other_containers`, `/k8s-volumes`, `/k3s-backups` | `/mnt/Storage/…` | `/mnt/Storage/…` | already correct |
|
|
| `Storage` (pool root) | `/Storage` | `/mnt/Storage` | fix on import |
|
|
| `Storage2` (pool root) | `/Storage2` | `/mnt/Storage2` | fix on import |
|
|
| `Storage/subvol-108-disk-0` | `/Storage/subvol-108-disk-0` | — | rename or delete |
|
|
|
|
**Clear the ZFS `sharenfs` properties.** `Storage/plex`, `Storage/nextcloud` and
|
|
`Storage/docker-volumes` carry exports as ZFS properties; `Storage/k8s-volumes` is in
|
|
`/etc/exports`. TrueNAS generates exports from its own middleware database and will fight
|
|
properties it did not set. Wipe them and recreate the shares in the UI.
|
|
|
|
**Fix the export targets while you are there.** Today's lists are stale: `/mnt/k3s` →
|
|
`192.168.1.25/.26/.27` and `k8s-volumes` → `192.168.1.40/.41/.42` are hosts that do not answer ARP.
|
|
The new exports target **`192.168.2.10/.11/.12`**.
|
|
|
|
---
|
|
|
|
## 4. Sequence
|
|
|
|
### Phase A — drain, locally, no downtime, no network
|
|
|
|
Everything below is a copy from somewhere else in the chassis onto `Storage`, which is moving.
|
|
|
|
1. **Games** — `rpool` → new dataset.
|
|
```bash
|
|
zfs create Storage/games
|
|
rsync -aHAX --info=progress2 /mnt/Storage/Games/ /mnt/Storage/games/
|
|
```
|
|
2. **Gitea** — copy the *Docker volume*, not `/opt/gitea` (§2a). Stop the container first so
|
|
`gitea.db` is quiesced.
|
|
```bash
|
|
pct exec 106 -- docker stop gitea
|
|
zfs create Storage/gitea
|
|
rsync -aHAX /rpool/data/subvol-106-disk-0/var/lib/docker/volumes/8a1d2616*/_data/ /mnt/Storage/gitea/
|
|
```
|
|
3. **Nextcloud** — the long one. Copy `lv_storage`'s contents to a new dataset at file level from
|
|
inside the running VM, over its existing NFS mount of the host. Verify §2c's size first so you
|
|
know whether it fits (it will, in 7.60 T free).
|
|
4. **Confirm `config_backup`** (267 G) is disposable; drop it if so.
|
|
5. **Verify each service's real data location before copying it** — the §2a rule.
|
|
|
|
### Phase B — prove the compute side before the storage moves
|
|
|
|
Bring the `apps/wave-0` manifests up on prod k3s while the old box is still alive, so the cutover is
|
|
a storage swap against known-good workloads rather than two migrations at once.
|
|
|
|
Wrinkle: prod cannot reach `192.168.1.9` today (§2b). Either add a temporary pfSense rule for the
|
|
test, or accept that app bring-up happens after the NAS is live on `192.168.2.0/24` — in which case
|
|
stage the app configs now and keep the window short.
|
|
|
|
### Phase C — cutover
|
|
|
|
1. Stop all guests; reverse `startup:` order, pfSense last.
|
|
2. Export cleanly — **never just cut power:**
|
|
```bash
|
|
zpool export Storage && zpool export Storage2
|
|
zpool status # both must be gone before power-off
|
|
```
|
|
3. **Label every drive** with pool and serial before it leaves the chassis
|
|
(`3FGJ6M9T`/`3FGHAB2T` = Storage; `WKD1S0TR`/`WKD1S10M` = Storage2).
|
|
4. Move the four HDDs — cold, upright. The 6-year-old Seagates are the fragile ones.
|
|
5. Install TrueNAS SCALE on the shipped 64 GB M.2 (Slot A). The five bays take the four HDDs with
|
|
one spare; the Micron SSDs are **not needed** in the target — leave them in the old chassis.
|
|
6. **Import read-only first**, verify, then import for real:
|
|
```bash
|
|
zpool import # both pools listed
|
|
zpool import -o readonly=on Storage
|
|
zfs list # verify datasets, sizes
|
|
zpool export Storage && zpool import Storage
|
|
```
|
|
|
|
### Phase D — repoint, reclaim, retire
|
|
|
|
1. Recreate NFS shares in the TrueNAS UI targeting `192.168.2.10/.11/.12`; clear the old `sharenfs`
|
|
properties.
|
|
2. Point the `apps/wave-0` PVs at the NAS; bring Plex, the arr stack and Nextcloud up on prod k3s.
|
|
3. Stand up MinIO on TrueNAS to replace versitygw, and re-enable Velero + CNPG backups (§2e).
|
|
4. **Only once everything is verified from k3s:** delete the dead zvols (§1) — ~731 G on `Storage`,
|
|
4.06 T on `Storage2`.
|
|
5. Re-evaluate §2d. If it fits, evacuate `Storage2`, retire the 6-year-old Seagates, keep two bays
|
|
free.
|
|
6. Repoint the SOHO 443 NAT, then power off the Proxmox host.
|
|
7. `zpool status` — confirm the 36 read errors did **not** follow `3FGHAB2T` into the new chassis.
|
|
Full scrub on the new hardware.
|
|
|
|
---
|
|
|
|
## 5. Rollback, and the one-way door
|
|
|
|
Until Phase D step 4, the old chassis is intact: `rpool`, the Proxmox install, both Microns and
|
|
every `/etc/pve` config untouched. Put the four HDDs back and it boots into the previous world.
|
|
|
|
> ### Do not run `zpool upgrade`
|
|
>
|
|
> `zpool status Storage2` already nags — *"Some supported and requested features are not enabled…
|
|
> action: Enable all features using 'zpool upgrade'"* — and TrueNAS will offer it in the UI after
|
|
> import. It is the only irreversible step here: run it and the pools can never go back to the
|
|
> Proxmox host. Leave it until the NAS has been serving k3s for weeks.
|
|
|
|
Note the asymmetry that makes this safe: TrueNAS's OpenZFS **reads** these pools happily, because
|
|
newer ZFS always reads older pools. The reverse is not true.
|
|
|
|
---
|
|
|
|
## 6. Open items
|
|
|
|
1. **The Nextcloud number** (§2c) — one `df -h` inside VM 103. It decides whether `Storage2` and the
|
|
two oldest drives can be retired.
|
|
2. **Is `config_backup` (267 G) disposable?**
|
|
3. **What serves DNS on `192.168.2.0/24`** once Pi-hole dies?
|
|
4. **UniFi controller** — rehome, cloud key, or drop?
|
|
5. **Minecraft** — keep or retire?
|
|
6. **Gitea** — its whole state is 3 repos and an 11 MB SQLite DB, and prod k3s already runs its own
|
|
Gitea. Is the legacy one worth migrating at all, or does it just end?
|