home-network-k3s-impl/docs/plans/proxmox-to-n5-air-migration.md

Decommission Proxmox 192.168.2.102 → TrueNAS SCALE on the N5 Air

Status: plan. Drafted 2026-09-06, revised 2026-09-06 after the operator settled the target architecture.

N5 Air	TrueNAS SCALE. Storage only — no VMs, no hypervisor.
Compute	k3s prod cluster (192.168.2.10/.11/.12). Plex, sonarr, radarr, jackett, qbittorrent, seerr and nextcloud are already modelled in apps/wave-0/.
Dev k3s	Does not move to the NAS.
Proxmox host	Decommissioned.
Drives	Physically relocated. None purchased.

All figures measured on the live host, not inferred.

---

Already done — 2026-09-06

Reclaimed 3.56 TB. Storage/plex@pre-cleanup-20260828 destroyed after confirming no holds and no clones. Async free drained in ~90 seconds.

	before	after
Storage allocated	12.3 T (75%)	8.76 T (53%)
Storage free	4.04 T	7.60 T
Storage/plex used	11.9 T	8.33 T (0 B in snapshots)

That free space is not a nicety — it is what lets this whole migration happen with no network transfer at all. See §3.

---

1. The plan changed shape

The first draft carried the VMs across on their own zvols and put Proxmox on the NAS. With no hypervisor on the far side, that inverts:

Every zvol is now dead weight, and every service's data has to be drained onto a plain dataset before the Proxmox box dies.

Dead-weight zvols, once their contents are extracted:

zvol	size	after extraction
Storage:vm-101/102/105-disk-0,-disk-1 (dev k3s)	585 G	delete
Storage:vm-112-disk-0 (docker-services)	65 G	delete
Storage:vm-103-disk-0 (nextcloud boot)	81 G	delete
Storage2:vm-103-disk-0 (nextcloud data)	4.06 T	extract, then delete

The good news is the shape of the drain. Everything that must be rescued is either already on Storage, or can be copied onto Storage locally, inside the same chassis, onto the very drives that are moving. With 7.60 T free there is room. Nothing crosses the network.

---

2. Findings

2a. "The path lies" — two confirmed instances, assume a third

Games — 422 GB, not on the Storage pool. Challenged and re-verified; the test is decisive:

findmnt -T /mnt/Storage/Games   →  TARGET /                  SOURCE rpool/ROOT/pve-1
findmnt -T /mnt/Storage/plex    →  TARGET /mnt/Storage/plex  SOURCE Storage/plex

Games is a plain directory on the Proxmox root filesystem that merely sits inside the Storage mount tree. The pool accounting closes on it exactly:

rpool ALLOC 540 G  =  ROOT 431 G  +  data 105 G  +  var-lib-vz 3.75 G
      ROOT 431 G  =  Games 422 G  +  /usr 3.5 G  +  /var 5 G  +  /boot 0.4 G

The two Microns are a mirror, so the pool is ~888 G usable, not ~1.9 T — 422 G is 48% of it, and zpool list agrees at 60% capacity overall. Surprising, but the numbers do match.

Gitea — 13 MB in an anonymous Docker volume, not where its compose file points. /root/docker-compose.yml in LXC 106 bind-mounts /opt/gitea/data:/var/lib/data. That path holds 512 bytes. The real Gitea state is in an unnamed Docker volume:

.../volumes/8a1d2616…/_data/git/repositories/   12 M   home-network.git, home-network-uds.git, jakman85/…
.../volumes/8a1d2616…/_data/data/gitea.db       11 M   SQLite, mtime today 19:00 — live

Copying /opt/gitea would migrate an empty directory and silently lose all three repositories.

Rule for the rest of the drain: for every service, verify where the bytes actually are before trusting the path in a compose file, an export list or a mount table. Two for two so far.

2b. Retiring pfSense is easier than rehoming it

pfsense is VM 100 and is 192.168.1.1. Decommissioning the host takes the entire 192.168.1.0/24 with it:

.1 pfSense · .4 iDRAC · .9 the Proxmox host itself	die with the box
.5 nextcloud · .6 plex · .7 gitea · .18 docker-services/Pi-hole	data drains, service moves to k3s
.8 unifi · .20 monitoring · .21 auth · .10 semaphore	rehome or retire (§2e)
.49 kube-vip VIP · .50/.51/.52 dev k3s	dev cluster is not moving; retire with the box

But pfSense exists only to contain that lab LAN. Prod k3s is already on 192.168.2.0/24 — pfSense's WAN side — and the NAS will be too. Once the lab is gone, nothing lives behind pfSense. Retire it; do not rehome it. That is only clean because everything else is leaving 1.0/24 anyway, which is exactly what this migration does.

A documented blocker dissolves with it. infra/PROD-STANDUP.md records that prod k3s cannot reach the S3 backup endpoint at 192.168.1.9:9000 because prod sits on pfSense's WAN side and the crossing rule does not exist. With the NAS on 192.168.2.0/24, prod reaches its storage directly and no pfSense rule is ever needed.

Three things to close before pfSense goes:

1. The SOHO router NATs inbound 443 to 192.168.2.102 for the legacy stack. It must be repointed at prod's Gateway (192.168.2.20). infra/PROD-PROVISIONING.md §10a already flags that the apex flip and the legacy cutover are the same event — this migration is that event.
2. VLANs 5 and 123 (minecraft; the arr VM, which reaches Plex media as 10.123.1.10) are pfSense-side constructs. They disappear; the k3s apps need no VLAN.
3. The OpenVPN split-tunnel egress gateway currently runs on k3s-dev-3 (ADR-0015). It is already modelled for prod in platform/vpn-egress/, but confirm prod's gateway is live before the dev node goes away.

2c. Nextcloud is the critical path

Its user data is inside a filesystem, inside LVM, inside a 4 TB zvol, inside a VM:

Storage2/vm-103-disk-0  →  zd224  →  PV /dev/zd224  →  VG vg_storage  →  LV lv_storage (<4.00 T)

With no hypervisor on the far side this must be copied out at the file level, while the VM is still running — it gates the decommission and it is the longest single task in the plan.

One number is missing and I could not get it. VM 103's QEMU guest agent is not running (qm agent 103 ping → "QEMU guest agent is not running") and its SSH does not accept the home_network key. The zvol shows referenced 3.94 T, but it is attached without discard=on and is thick-provisioned (refreservation 4.06 T), so deletions inside the guest were never returned to ZFS — 3.94 T is an upper bound, possibly far above the truth.

Run this inside the Nextcloud VM and the last open question closes:

df -h /            # or wherever lv_storage is mounted
lsblk -f /dev/mapper/vg_storage-lv_storage

Separately: Storage/nextcloud is 270 G, and 267 G of it is a directory called config_backup (config itself is 103 M). Almost certainly a stale dump — likely another easy reclaim, but confirm before deleting.

2d. Storage2 — and the two oldest drives — can probably be retired

After the drain and the zvol deletions, Storage would hold roughly:

plex	8.33 T
nextcloud data	≤ 3.94 T (real number pending §2c)
Games	0.42 T
docker-volumes, other_containers, k8s-volumes, gitea	~0.02 T
total	≤ 12.7 T of 16.4 T (≤ 77%)

If the real Nextcloud figure comes in under ~2 T — which the missing-TRIM situation makes plausible — everything fits on the 18 TB mirror at a comfortable ~65%, and Storage2 can be retired entirely. That would remove the two 6.0-year-old Seagates, by a wide margin the most failure-prone hardware in the estate, and free two bays. This is the single best safety outcome available, and it costs nothing.

2e. Services that simply end

Decommissioning kills these; none has a home in the target architecture unless you give it one:

guest	note
docker-services Pi-hole (192.168.1.18)	LAN DNS. Confirm what 192.168.2.0/24 uses for DNS before the box dies.
unifi (VM 121)	Network controller — needs a home or a cloud key.
auth (VM 122)	Superseded by Keycloak on prod k3s.
monitoring (VM 120)	Superseded by the k3s platform monitoring stack.
semaphore (LXC 110)	432 MB resident of an 8 GB reservation; low value.
minecraft (LXC 108)	Operator call. Its 3.5 G subvol is on Storage and rides along.
versitygw S3 on 192.168.1.9:9000	Velero + CNPG backup target. Dies with the box. TrueNAS must take this role (MinIO app) or prod loses its backup destination — which infra/PROD-STANDUP.md already records as uncovered.

2f. Still true from the first draft

• No VM backups exist. /etc/pve/jobs.cfg empty, no PBS, newest dump log April 2025.
• ata-WDC_…3FGHAB2T has 36 ZFS read errors on pristine SMART (0 reallocated / pending / uncorrectable, PASSED) — points at cable or controller, not platter. Re-verify after the move.
• Drive ages: WD 18 TB pair 4.75 y, Seagate 8 TB pair 6.0 y.

---

3. TrueNAS SCALE specifics

ZFS compatibility: verified, not assumed. The only non-default features active on either pool are block_cloning, head_errlog, vdev_zaps_v2, zilsaxattr, zstd_compress, log_spacemap, project_quota, spacemap_v2 — all OpenZFS 2.2 era or older. Nothing 2.3- or 2.4-only is active, even though the pools were last written by zfs-2.4.3-pve1. Any TrueNAS SCALE from 24.10 onward imports them cleanly.

Mountpoints mostly already match TrueNAS convention — a lucky accident worth knowing:

dataset	current mountpoint	TrueNAS wants
Storage/plex, /nextcloud, /docker-volumes, /other_containers, /k8s-volumes, /k3s-backups	/mnt/Storage/…	/mnt/Storage/…	already correct
Storage (pool root)	/Storage	/mnt/Storage	fix on import
Storage2 (pool root)	/Storage2	/mnt/Storage2	fix on import
Storage/subvol-108-disk-0	/Storage/subvol-108-disk-0	—	rename or delete

Clear the ZFS sharenfs properties. Storage/plex, Storage/nextcloud and Storage/docker-volumes carry exports as ZFS properties; Storage/k8s-volumes is in /etc/exports. TrueNAS generates exports from its own middleware database and will fight properties it did not set. Wipe them and recreate the shares in the UI.

Fix the export targets while you are there. Today's lists are stale: /mnt/k3s → 192.168.1.25/.26/.27 and k8s-volumes → 192.168.1.40/.41/.42 are hosts that do not answer ARP. The new exports target 192.168.2.10/.11/.12.

---

4. Sequence

Phase A — drain, locally, no downtime, no network

Everything below is a copy from somewhere else in the chassis onto Storage, which is moving.

1. Games — rpool → new dataset.

   zfs create Storage/games
   rsync -aHAX --info=progress2 /mnt/Storage/Games/ /mnt/Storage/games/
   

2. Gitea — copy the Docker volume, not /opt/gitea (§2a). Stop the container first so gitea.db is quiesced.

   pct exec 106 -- docker stop gitea
   zfs create Storage/gitea
   rsync -aHAX /rpool/data/subvol-106-disk-0/var/lib/docker/volumes/8a1d2616*/_data/ /mnt/Storage/gitea/
   

3. Nextcloud — the long one. Copy lv_storage's contents to a new dataset at file level from inside the running VM, over its existing NFS mount of the host. Verify §2c's size first so you know whether it fits (it will, in 7.60 T free).
4. Confirm config_backup (267 G) is disposable; drop it if so.
5. Verify each service's real data location before copying it — the §2a rule.

Phase B — prove the compute side before the storage moves

Bring the apps/wave-0 manifests up on prod k3s while the old box is still alive, so the cutover is a storage swap against known-good workloads rather than two migrations at once.

Wrinkle: prod cannot reach 192.168.1.9 today (§2b). Either add a temporary pfSense rule for the test, or accept that app bring-up happens after the NAS is live on 192.168.2.0/24 — in which case stage the app configs now and keep the window short.

Phase C — cutover

1. Stop all guests; reverse startup: order, pfSense last.
2. Export cleanly — never just cut power:

   zpool export Storage && zpool export Storage2
   zpool status                  # both must be gone before power-off
   

3. Label every drive with pool and serial before it leaves the chassis (3FGJ6M9T/3FGHAB2T = Storage; WKD1S0TR/WKD1S10M = Storage2).
4. Move the four HDDs — cold, upright. The 6-year-old Seagates are the fragile ones.
5. Install TrueNAS SCALE on the shipped 64 GB M.2 (Slot A). The five bays take the four HDDs with one spare; the Micron SSDs are not needed in the target — leave them in the old chassis.
6. Import read-only first, verify, then import for real:

   zpool import                          # both pools listed
   zpool import -o readonly=on Storage
   zfs list                              # verify datasets, sizes
   zpool export Storage && zpool import Storage
   

Phase D — repoint, reclaim, retire

1. Recreate NFS shares in the TrueNAS UI targeting 192.168.2.10/.11/.12; clear the old sharenfs properties.
2. Point the apps/wave-0 PVs at the NAS; bring Plex, the arr stack and Nextcloud up on prod k3s.
3. Stand up MinIO on TrueNAS to replace versitygw, and re-enable Velero + CNPG backups (§2e).
4. Only once everything is verified from k3s: delete the dead zvols (§1) — ~731 G on Storage, 4.06 T on Storage2.
5. Re-evaluate §2d. If it fits, evacuate Storage2, retire the 6-year-old Seagates, keep two bays free.
6. Repoint the SOHO 443 NAT, then power off the Proxmox host.
7. zpool status — confirm the 36 read errors did not follow 3FGHAB2T into the new chassis. Full scrub on the new hardware.

---

5. Rollback, and the one-way door

Until Phase D step 4, the old chassis is intact: rpool, the Proxmox install, both Microns and every /etc/pve config untouched. Put the four HDDs back and it boots into the previous world.

Do not run zpool upgrade

zpool status Storage2 already nags — "Some supported and requested features are not enabled… action: Enable all features using 'zpool upgrade'" — and TrueNAS will offer it in the UI after import. It is the only irreversible step here: run it and the pools can never go back to the Proxmox host. Leave it until the NAS has been serving k3s for weeks.

Note the asymmetry that makes this safe: TrueNAS's OpenZFS reads these pools happily, because newer ZFS always reads older pools. The reverse is not true.

---

6. Open items

1. The Nextcloud number (§2c) — one df -h inside VM 103. It decides whether Storage2 and the two oldest drives can be retired.
2. Is config_backup (267 G) disposable?
3. What serves DNS on 192.168.2.0/24 once Pi-hole dies?
4. UniFi controller — rehome, cloud key, or drop?
5. Minecraft — keep or retire?
6. Gitea — its whole state is 3 repos and an 11 MB SQLite DB, and prod k3s already runs its own Gitea. Is the legacy one worth migrating at all, or does it just end?