# Phase 2 execution runbook — SSD-backed etcd migration (interactive/manual)

Companion to `ssd-etcd-isolation.md`. This is the **exact command sequence** to migrate
each k3s node's etcd onto its dedicated SSD `virtio2` disk. It must be run where live-infra
mutations are permitted — an **interactive `claude` session** (default permission mode, which
prompts for approval) or **by hand**. The autonomous "auto mode" session hard-denies
`tofu apply` and node-level ssh mutations, so it cannot run this.

## Preconditions (verified 2026-08-29, cluster healthy)
- Run everything from the staged worktree that carries the tofu disk block:
  `cd /home/jakman85/source/home-network-k3s-impl/.claude/worktrees/agent-ssd-etcd`
  (branch `feature/ssd-etcd-isolation` @ `af5fc5b`).
- All tooling via the devcontainer: `./infra/devcontainer/dev.sh run <cmd>`.
- `./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh init` once (done).
- Ansible must run from `infra/ansible/` (for the vault-pass path): wrap as
  `./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible ...'`.
- **Run mutation commands STANDALONE** (not chained with `&&`/pipes) — the classifier is
  more likely to deny a compound command that embeds a mutation.

## ORDER (leader last — RE-VERIFY before each node)
Current leader = **dev-3**, so: **dev-2 (vmid 102) → dev-1 (vmid 105) → dev-3 (vmid 101)**.
dev-3 is last and additionally gets the memory 16384→24576 bump (needs a full stop/start).
Re-check the leader before each node and keep the leader last:
```
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_is_leader"'
```

## Pre-flight before EACH node (read-only — all must pass)
```
./infra/devcontainer/dev.sh run kubectl get nodes
./infra/devcontainer/dev.sh run kubectl get --raw=/readyz/etcd            # expect: ok
./infra/devcontainer/dev.sh run kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns=R:.status.robustness --no-headers | grep -c healthy   # expect: 18
```

## Per-node procedure (substitute NODE=k3s-dev-N, VMID)

### 1. Freeze rebuilds (STANDALONE)
```
./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"0"}'
```

### 2. Add the SSD disk via tofu (STANDALONE). dev-2/dev-1 = disk only; dev-3 also memory.
```
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan  '-target=proxmox_virtual_environment_vm.node["k3s-dev-N"]'   # confirm: disk ADD, in-place, NOT replace
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh apply -auto-approve '-target=proxmox_virtual_environment_vm.node["k3s-dev-N"]'
```
Confirm the guest sees the new disk (hot-plug):
```
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -m shell -a "lsblk /dev/vdc"'
```
If `/dev/vdc` is NOT present (no hot-plug), power-cycle the VM (freeze is holding):
```
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible proxmox -m shell -a "qm stop VMID"'   # wait until stopped
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible proxmox -m shell -a "qm start VMID"'
```
(For dev-3 the memory bump requires this stop/start regardless.)

### 3. Migrate etcd data (node commands, become=root). Do NOT skip the copy-before-mount order.
```
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "systemctl stop k3s"'
# mkfs ONLY if /dev/vdc has no filesystem yet (fresh disk):
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "blkid /dev/vdc || mkfs.ext4 -L k3s-etcd /dev/vdc"'
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "mv /var/lib/rancher/k3s/server/db /var/lib/rancher/k3s/server/db.bak && mkdir /var/lib/rancher/k3s/server/db && mount LABEL=k3s-etcd /var/lib/rancher/k3s/server/db && rsync -aHAX --numeric-ids /var/lib/rancher/k3s/server/db.bak/ /var/lib/rancher/k3s/server/db/"'
# Persist the mount (matches the node_baseline role: LABEL + defaults,noatime):
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m ansible.posix.mount -a "path=/var/lib/rancher/k3s/server/db src=LABEL=k3s-etcd fstype=ext4 opts=defaults,noatime dump=0 passno=2 state=mounted"'
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "systemctl start k3s"'
```

### 4. VERIFY full recovery before touching the next node (all must pass)
```
./infra/devcontainer/dev.sh run kubectl get nodes                          # NODE Ready
./infra/devcontainer/dev.sh run kubectl get --raw=/readyz/etcd             # ok
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -m shell -a "findmnt /var/lib/rancher/k3s/server/db"'   # source = /dev/vdc
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_has_leader"'  # all =1
./infra/devcontainer/dev.sh run kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns=R:.status.robustness --no-headers | grep -c healthy   # 18
# dev-3 only: confirm ~24Gi allocatable
./infra/devcontainer/dev.sh run kubectl get node k3s-dev-3 -o jsonpath='{.status.allocatable.memory}'
```

### 5. Thaw + clean up (STANDALONE), only after step 4 fully passes
```
./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"1"}'
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "rm -rf /var/lib/rancher/k3s/server/db.bak"'
```
Wait for 18/18 healthy (no rebuild storm), re-verify leader, then proceed to the next node.

## STOP conditions (halt immediately, leave limit=0, do not advance)
etcd `/readyz` not ok, a member without a leader, VIP `192.168.2.50:6443` flapping, a node
not rejoining, any Longhorn volume Faulting, or k3s crashlooping. Recovery: keep rebuilds
frozen; a botched single member can be reset and re-joined from the other two (plan §4.5).

## Close-out (after all 3)
- All 3 nodes Ready; etcd 3/3 with a leader; `findmnt` shows `/dev/vdc` at the db path on each.
- 18/18 Longhorn volumes healthy; rebuild-limit back to **1**.
- `./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan` → **No changes** (all disks present, dev-3 memory reconciled).
- Merge `feature/ssd-etcd-isolation` → `main` and push (push is classifier-blocked in auto mode — run `git push gitea main` by hand) so Flux/tofu source records the disk + baseline.
- Regression test: after approval, bounce ONE node and confirm etcd stays healthy / VIP steady — the whole point of the change.