home-network-k3s-impl/docs/plans/ssd-etcd-execution-runbook.md

Phase 2 execution runbook — SSD-backed etcd migration (interactive/manual)

Companion to ssd-etcd-isolation.md. This is the exact command sequence to migrate each k3s node's etcd onto its dedicated SSD virtio2 disk. It must be run where live-infra mutations are permitted — an interactive claude session (default permission mode, which prompts for approval) or by hand. The autonomous "auto mode" session hard-denies tofu apply and node-level ssh mutations, so it cannot run this.

Preconditions (verified 2026-08-29, cluster healthy)

• Run everything from the staged worktree that carries the tofu disk block: cd /home/jakman85/source/home-network-k3s-impl/.claude/worktrees/agent-ssd-etcd (branch feature/ssd-etcd-isolation @ af5fc5b).
• All tooling via the devcontainer: ./infra/devcontainer/dev.sh run <cmd>.
• ./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh init once (done).
• Ansible must run from infra/ansible/ (for the vault-pass path): wrap as ./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible ...'.
• Run mutation commands STANDALONE (not chained with &&/pipes) — the classifier is more likely to deny a compound command that embeds a mutation.

ORDER (leader last — RE-VERIFY before each node)

Current leader = dev-3, so: dev-2 (vmid 102) → dev-1 (vmid 105) → dev-3 (vmid 101). dev-3 is last and additionally gets the memory 16384→24576 bump (needs a full stop/start). Re-check the leader before each node and keep the leader last:

./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_is_leader"'

Pre-flight before EACH node (read-only — all must pass)

./infra/devcontainer/dev.sh run kubectl get nodes
./infra/devcontainer/dev.sh run kubectl get --raw=/readyz/etcd            # expect: ok
./infra/devcontainer/dev.sh run kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns=R:.status.robustness --no-headers | grep -c healthy   # expect: 18

Per-node procedure (substitute NODE=k3s-dev-N, VMID)

1. Freeze rebuilds (STANDALONE)

./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"0"}'

2. Add the SSD disk via tofu (STANDALONE). dev-2/dev-1 = disk only; dev-3 also memory.

./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan  '-target=proxmox_virtual_environment_vm.node["k3s-dev-N"]'   # confirm: disk ADD, in-place, NOT replace
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh apply -auto-approve '-target=proxmox_virtual_environment_vm.node["k3s-dev-N"]'

Confirm the guest sees the new disk (hot-plug):

./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -m shell -a "lsblk /dev/vdc"'

If /dev/vdc is NOT present (no hot-plug), power-cycle the VM (freeze is holding):

./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible proxmox -m shell -a "qm stop VMID"'   # wait until stopped
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible proxmox -m shell -a "qm start VMID"'

(For dev-3 the memory bump requires this stop/start regardless.)

3. Migrate etcd data (node commands, become=root). Do NOT skip the copy-before-mount order.

./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "systemctl stop k3s"'
# mkfs ONLY if /dev/vdc has no filesystem yet (fresh disk):
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "blkid /dev/vdc || mkfs.ext4 -L k3s-etcd /dev/vdc"'
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "mv /var/lib/rancher/k3s/server/db /var/lib/rancher/k3s/server/db.bak && mkdir /var/lib/rancher/k3s/server/db && mount LABEL=k3s-etcd /var/lib/rancher/k3s/server/db && rsync -aHAX --numeric-ids /var/lib/rancher/k3s/server/db.bak/ /var/lib/rancher/k3s/server/db/"'
# Persist the mount (matches the node_baseline role: LABEL + defaults,noatime):
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m ansible.posix.mount -a "path=/var/lib/rancher/k3s/server/db src=LABEL=k3s-etcd fstype=ext4 opts=defaults,noatime dump=0 passno=2 state=mounted"'
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "systemctl start k3s"'

4. VERIFY full recovery before touching the next node (all must pass)

./infra/devcontainer/dev.sh run kubectl get nodes                          # NODE Ready
./infra/devcontainer/dev.sh run kubectl get --raw=/readyz/etcd             # ok
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -m shell -a "findmnt /var/lib/rancher/k3s/server/db"'   # source = /dev/vdc
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_has_leader"'  # all =1
./infra/devcontainer/dev.sh run kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns=R:.status.robustness --no-headers | grep -c healthy   # 18
# dev-3 only: confirm ~24Gi allocatable
./infra/devcontainer/dev.sh run kubectl get node k3s-dev-3 -o jsonpath='{.status.allocatable.memory}'

5. Thaw + clean up (STANDALONE), only after step 4 fully passes

./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"1"}'
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "rm -rf /var/lib/rancher/k3s/server/db.bak"'

Wait for 18/18 healthy (no rebuild storm), re-verify leader, then proceed to the next node.

STOP conditions (halt immediately, leave limit=0, do not advance)

etcd /readyz not ok, a member without a leader, VIP 192.168.2.50:6443 flapping, a node not rejoining, any Longhorn volume Faulting, or k3s crashlooping. Recovery: keep rebuilds frozen; a botched single member can be reset and re-joined from the other two (plan §4.5).

Close-out (after all 3)

• All 3 nodes Ready; etcd 3/3 with a leader; findmnt shows /dev/vdc at the db path on each.
• 18/18 Longhorn volumes healthy; rebuild-limit back to 1.
• ./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan → No changes (all disks present, dev-3 memory reconciled).
• Merge feature/ssd-etcd-isolation → main and push (push is classifier-blocked in auto mode — run git push gitea main by hand) so Flux/tofu source records the disk + baseline.
• Regression test: after approval, bounce ONE node and confirm etcd stays healthy / VIP steady — the whole point of the change.