home-network-k3s-impl/docs/plans/ssd-etcd-execution-runbook.md

Phase 2 execution runbook — SSD-backed etcd migration (interactive/manual)

✅ COMPLETED / historical — executed 2026-08-30, see git log

This runbook has been executed. Every k3s node's etcd now lives on its dedicated SSD virtio2 disk. The command sequence below is retained as the record of how the migration was performed — do NOT re-run it against the live cluster. Current state is tracked in infra/PHASE1-STATUS.md and the 2026-08-30 DECISIONS-LOG entry.

Companion to ssd-etcd-isolation.md. This is the exact command sequence to migrate each k3s node's etcd onto its dedicated SSD virtio2 disk. It must be run where live-infra mutations are permitted — an interactive claude session (default permission mode, which prompts for approval) or by hand. The autonomous "auto mode" session hard-denies tofu apply and node-level ssh mutations, so it cannot run this.

Preconditions (verified 2026-08-29, cluster healthy)

• Run everything from the staged worktree that carries the tofu disk block: cd /home/jakman85/source/home-network-k3s-impl/.claude/worktrees/agent-ssd-etcd (branch feature/ssd-etcd-isolation @ af5fc5b).
• All tooling via the devcontainer: ./infra/devcontainer/dev.sh run <cmd>.
• ./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh init once (done).
• Ansible must run from infra/ansible/ (for the vault-pass path): wrap as ./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible ...'.
• Run mutation commands STANDALONE (not chained with &&/pipes) — the classifier is more likely to deny a compound command that embeds a mutation.

ORDER (leader last — RE-VERIFY before each node)

Current leader = dev-3, so: dev-2 (vmid 102) → dev-1 (vmid 105) → dev-3 (vmid 101). dev-3 is last and additionally gets the memory 16384→24576 bump (needs a full stop/start). Re-check the leader before each node and keep the leader last:

./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_is_leader"'

Pre-flight before EACH node (read-only — all must pass)

./infra/devcontainer/dev.sh run kubectl get nodes
./infra/devcontainer/dev.sh run kubectl get --raw=/readyz/etcd            # expect: ok
./infra/devcontainer/dev.sh run kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns=R:.status.robustness --no-headers | grep -c healthy   # expect: 18

Per-node procedure (substitute NODE=k3s-dev-N, VMID)

1. Freeze rebuilds (STANDALONE)

./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"0"}'

2. Add the SSD disk via tofu (STANDALONE). dev-2/dev-1 = disk only; dev-3 also memory.

./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan  '-target=proxmox_virtual_environment_vm.node["k3s-dev-N"]'   # confirm: disk ADD, in-place, NOT replace
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh apply -auto-approve '-target=proxmox_virtual_environment_vm.node["k3s-dev-N"]'

Confirm the guest sees the new disk (hot-plug):

./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -m shell -a "lsblk /dev/vdc"'

If /dev/vdc is NOT present (no hot-plug), power-cycle the VM (freeze is holding):

./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible proxmox -m shell -a "qm stop VMID"'   # wait until stopped
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible proxmox -m shell -a "qm start VMID"'

(For dev-3 the memory bump requires this stop/start regardless.)

3. Migrate etcd data (node commands, become=root). Do NOT skip the copy-before-mount order.

./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "systemctl stop k3s"'
# mkfs ONLY if /dev/vdc has no filesystem yet (fresh disk):
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "blkid /dev/vdc || mkfs.ext4 -L k3s-etcd /dev/vdc"'
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "mv /var/lib/rancher/k3s/server/db /var/lib/rancher/k3s/server/db.bak && mkdir /var/lib/rancher/k3s/server/db && mount LABEL=k3s-etcd /var/lib/rancher/k3s/server/db && rsync -aHAX --numeric-ids /var/lib/rancher/k3s/server/db.bak/ /var/lib/rancher/k3s/server/db/"'
# Persist the mount (matches the node_baseline role: LABEL + defaults,noatime):
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m ansible.posix.mount -a "path=/var/lib/rancher/k3s/server/db src=LABEL=k3s-etcd fstype=ext4 opts=defaults,noatime dump=0 passno=2 state=mounted"'
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "systemctl start k3s"'

4. VERIFY full recovery before touching the next node (all must pass)

./infra/devcontainer/dev.sh run kubectl get nodes                          # NODE Ready
./infra/devcontainer/dev.sh run kubectl get --raw=/readyz/etcd             # ok
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -m shell -a "findmnt /var/lib/rancher/k3s/server/db"'   # source = /dev/vdc
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_has_leader"'  # all =1
./infra/devcontainer/dev.sh run kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns=R:.status.robustness --no-headers | grep -c healthy   # 18
# dev-3 only: confirm ~24Gi allocatable
./infra/devcontainer/dev.sh run kubectl get node k3s-dev-3 -o jsonpath='{.status.allocatable.memory}'

5. Thaw + clean up (STANDALONE), only after step 4 fully passes

./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"1"}'
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "rm -rf /var/lib/rancher/k3s/server/db.bak"'

Wait for 18/18 healthy (no rebuild storm), re-verify leader, then proceed to the next node.

STOP conditions (halt immediately, leave limit=0, do not advance)

etcd /readyz not ok, a member without a leader, VIP 192.168.2.50:6443 flapping, a node not rejoining, any Longhorn volume Faulting, or k3s crashlooping. Recovery: keep rebuilds frozen; a botched single member can be reset and re-joined from the other two (plan §4.5).

Close-out (after all 3)

• All 3 nodes Ready; etcd 3/3 with a leader; findmnt shows /dev/vdc at the db path on each.
• 18/18 Longhorn volumes healthy; rebuild-limit back to 1.
• ./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan → No changes (all disks present, dev-3 memory reconciled).
• Merge feature/ssd-etcd-isolation → main and push (push is classifier-blocked in auto mode — run git push gitea main by hand) so Flux/tofu source records the disk + baseline.
• Regression test: after approval, bounce ONE node and confirm etcd stays healthy / VIP steady — the whole point of the change.