docs: add Phase 2 SSD-etcd execution runbook

Co-Authored-By: Claude Opus 4.8 <[email protected]>

docs/plans/ssd-etcd-execution-runbook.md

@@ -0,0 +1,97 @@
+# Phase 2 execution runbook — SSD-backed etcd migration (interactive/manual)
+
+Companion to `ssd-etcd-isolation.md`. This is the **exact command sequence** to migrate
+each k3s node's etcd onto its dedicated SSD `virtio2` disk. It must be run where live-infra
+mutations are permitted — an **interactive `claude` session** (default permission mode, which
+prompts for approval) or **by hand**. The autonomous "auto mode" session hard-denies
+`tofu apply` and node-level ssh mutations, so it cannot run this.
+
+## Preconditions (verified 2026-08-29, cluster healthy)
+- Run everything from the staged worktree that carries the tofu disk block:
+  `cd /home/jakman85/source/home-network-k3s-impl/.claude/worktrees/agent-ssd-etcd`
+  (branch `feature/ssd-etcd-isolation` @ `af5fc5b`).
+- All tooling via the devcontainer: `./infra/devcontainer/dev.sh run <cmd>`.
+- `./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh init` once (done).
+- Ansible must run from `infra/ansible/` (for the vault-pass path): wrap as
+  `./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible ...'`.
+- **Run mutation commands STANDALONE** (not chained with `&&`/pipes) — the classifier is
+  more likely to deny a compound command that embeds a mutation.
+
+## ORDER (leader last — RE-VERIFY before each node)
+Current leader = **dev-3**, so: **dev-2 (vmid 102) → dev-1 (vmid 105) → dev-3 (vmid 101)**.
+dev-3 is last and additionally gets the memory 16384→24576 bump (needs a full stop/start).
+Re-check the leader before each node and keep the leader last:
+```
+./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_is_leader"'
+```
+
+## Pre-flight before EACH node (read-only — all must pass)
+```
+./infra/devcontainer/dev.sh run kubectl get nodes
+./infra/devcontainer/dev.sh run kubectl get --raw=/readyz/etcd            # expect: ok
+./infra/devcontainer/dev.sh run kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns=R:.status.robustness --no-headers | grep -c healthy   # expect: 18
+```
+
+## Per-node procedure (substitute NODE=k3s-dev-N, VMID)
+
+### 1. Freeze rebuilds (STANDALONE)
+```
+./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"0"}'
+```
+
+### 2. Add the SSD disk via tofu (STANDALONE). dev-2/dev-1 = disk only; dev-3 also memory.
+```
+./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan  '-target=proxmox_virtual_environment_vm.node["k3s-dev-N"]'   # confirm: disk ADD, in-place, NOT replace
+./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh apply -auto-approve '-target=proxmox_virtual_environment_vm.node["k3s-dev-N"]'
+```
+Confirm the guest sees the new disk (hot-plug):
+```
+./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -m shell -a "lsblk /dev/vdc"'
+```
+If `/dev/vdc` is NOT present (no hot-plug), power-cycle the VM (freeze is holding):
+```
+./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible proxmox -m shell -a "qm stop VMID"'   # wait until stopped
+./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible proxmox -m shell -a "qm start VMID"'
+```
+(For dev-3 the memory bump requires this stop/start regardless.)
+
+### 3. Migrate etcd data (node commands, become=root). Do NOT skip the copy-before-mount order.
+```
+./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "systemctl stop k3s"'
+# mkfs ONLY if /dev/vdc has no filesystem yet (fresh disk):
+./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "blkid /dev/vdc || mkfs.ext4 -L k3s-etcd /dev/vdc"'
+./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "mv /var/lib/rancher/k3s/server/db /var/lib/rancher/k3s/server/db.bak && mkdir /var/lib/rancher/k3s/server/db && mount LABEL=k3s-etcd /var/lib/rancher/k3s/server/db && rsync -aHAX --numeric-ids /var/lib/rancher/k3s/server/db.bak/ /var/lib/rancher/k3s/server/db/"'
+# Persist the mount (matches the node_baseline role: LABEL + defaults,noatime):
+./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m ansible.posix.mount -a "path=/var/lib/rancher/k3s/server/db src=LABEL=k3s-etcd fstype=ext4 opts=defaults,noatime dump=0 passno=2 state=mounted"'
+./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "systemctl start k3s"'
+```
+
+### 4. VERIFY full recovery before touching the next node (all must pass)
+```
+./infra/devcontainer/dev.sh run kubectl get nodes                          # NODE Ready
+./infra/devcontainer/dev.sh run kubectl get --raw=/readyz/etcd             # ok
+./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -m shell -a "findmnt /var/lib/rancher/k3s/server/db"'   # source = /dev/vdc
+./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_has_leader"'  # all =1
+./infra/devcontainer/dev.sh run kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns=R:.status.robustness --no-headers | grep -c healthy   # 18
+# dev-3 only: confirm ~24Gi allocatable
+./infra/devcontainer/dev.sh run kubectl get node k3s-dev-3 -o jsonpath='{.status.allocatable.memory}'
+```
+
+### 5. Thaw + clean up (STANDALONE), only after step 4 fully passes
+```
+./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"1"}'
+./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "rm -rf /var/lib/rancher/k3s/server/db.bak"'
+```
+Wait for 18/18 healthy (no rebuild storm), re-verify leader, then proceed to the next node.
+
+## STOP conditions (halt immediately, leave limit=0, do not advance)
+etcd `/readyz` not ok, a member without a leader, VIP `192.168.2.50:6443` flapping, a node
+not rejoining, any Longhorn volume Faulting, or k3s crashlooping. Recovery: keep rebuilds
+frozen; a botched single member can be reset and re-joined from the other two (plan §4.5).
+
+## Close-out (after all 3)
+- All 3 nodes Ready; etcd 3/3 with a leader; `findmnt` shows `/dev/vdc` at the db path on each.
+- 18/18 Longhorn volumes healthy; rebuild-limit back to **1**.
+- `./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan` → **No changes** (all disks present, dev-3 memory reconciled).
+- Merge `feature/ssd-etcd-isolation` → `main` and push (push is classifier-blocked in auto mode — run `git push gitea main` by hand) so Flux/tofu source records the disk + baseline.
+- Regression test: after approval, bounce ONE node and confirm etcd stays healthy / VIP steady — the whole point of the change.