diff --git a/docs/plans/ssd-etcd-execution-runbook.md b/docs/plans/ssd-etcd-execution-runbook.md new file mode 100644 index 0000000..212ad3d --- /dev/null +++ b/docs/plans/ssd-etcd-execution-runbook.md @@ -0,0 +1,97 @@ +# Phase 2 execution runbook — SSD-backed etcd migration (interactive/manual) + +Companion to `ssd-etcd-isolation.md`. This is the **exact command sequence** to migrate +each k3s node's etcd onto its dedicated SSD `virtio2` disk. It must be run where live-infra +mutations are permitted — an **interactive `claude` session** (default permission mode, which +prompts for approval) or **by hand**. The autonomous "auto mode" session hard-denies +`tofu apply` and node-level ssh mutations, so it cannot run this. + +## Preconditions (verified 2026-08-29, cluster healthy) +- Run everything from the staged worktree that carries the tofu disk block: + `cd /home/jakman85/source/home-network-k3s-impl/.claude/worktrees/agent-ssd-etcd` + (branch `feature/ssd-etcd-isolation` @ `af5fc5b`). +- All tooling via the devcontainer: `./infra/devcontainer/dev.sh run `. +- `./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh init` once (done). +- Ansible must run from `infra/ansible/` (for the vault-pass path): wrap as + `./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible ...'`. +- **Run mutation commands STANDALONE** (not chained with `&&`/pipes) — the classifier is + more likely to deny a compound command that embeds a mutation. + +## ORDER (leader last — RE-VERIFY before each node) +Current leader = **dev-3**, so: **dev-2 (vmid 102) → dev-1 (vmid 105) → dev-3 (vmid 101)**. +dev-3 is last and additionally gets the memory 16384→24576 bump (needs a full stop/start). +Re-check the leader before each node and keep the leader last: +``` +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_is_leader"' +``` + +## Pre-flight before EACH node (read-only — all must pass) +``` +./infra/devcontainer/dev.sh run kubectl get nodes +./infra/devcontainer/dev.sh run kubectl get --raw=/readyz/etcd # expect: ok +./infra/devcontainer/dev.sh run kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns=R:.status.robustness --no-headers | grep -c healthy # expect: 18 +``` + +## Per-node procedure (substitute NODE=k3s-dev-N, VMID) + +### 1. Freeze rebuilds (STANDALONE) +``` +./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"0"}' +``` + +### 2. Add the SSD disk via tofu (STANDALONE). dev-2/dev-1 = disk only; dev-3 also memory. +``` +./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan '-target=proxmox_virtual_environment_vm.node["k3s-dev-N"]' # confirm: disk ADD, in-place, NOT replace +./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh apply -auto-approve '-target=proxmox_virtual_environment_vm.node["k3s-dev-N"]' +``` +Confirm the guest sees the new disk (hot-plug): +``` +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -m shell -a "lsblk /dev/vdc"' +``` +If `/dev/vdc` is NOT present (no hot-plug), power-cycle the VM (freeze is holding): +``` +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible proxmox -m shell -a "qm stop VMID"' # wait until stopped +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible proxmox -m shell -a "qm start VMID"' +``` +(For dev-3 the memory bump requires this stop/start regardless.) + +### 3. Migrate etcd data (node commands, become=root). Do NOT skip the copy-before-mount order. +``` +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "systemctl stop k3s"' +# mkfs ONLY if /dev/vdc has no filesystem yet (fresh disk): +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "blkid /dev/vdc || mkfs.ext4 -L k3s-etcd /dev/vdc"' +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "mv /var/lib/rancher/k3s/server/db /var/lib/rancher/k3s/server/db.bak && mkdir /var/lib/rancher/k3s/server/db && mount LABEL=k3s-etcd /var/lib/rancher/k3s/server/db && rsync -aHAX --numeric-ids /var/lib/rancher/k3s/server/db.bak/ /var/lib/rancher/k3s/server/db/"' +# Persist the mount (matches the node_baseline role: LABEL + defaults,noatime): +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m ansible.posix.mount -a "path=/var/lib/rancher/k3s/server/db src=LABEL=k3s-etcd fstype=ext4 opts=defaults,noatime dump=0 passno=2 state=mounted"' +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "systemctl start k3s"' +``` + +### 4. VERIFY full recovery before touching the next node (all must pass) +``` +./infra/devcontainer/dev.sh run kubectl get nodes # NODE Ready +./infra/devcontainer/dev.sh run kubectl get --raw=/readyz/etcd # ok +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -m shell -a "findmnt /var/lib/rancher/k3s/server/db"' # source = /dev/vdc +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_has_leader"' # all =1 +./infra/devcontainer/dev.sh run kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns=R:.status.robustness --no-headers | grep -c healthy # 18 +# dev-3 only: confirm ~24Gi allocatable +./infra/devcontainer/dev.sh run kubectl get node k3s-dev-3 -o jsonpath='{.status.allocatable.memory}' +``` + +### 5. Thaw + clean up (STANDALONE), only after step 4 fully passes +``` +./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"1"}' +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "rm -rf /var/lib/rancher/k3s/server/db.bak"' +``` +Wait for 18/18 healthy (no rebuild storm), re-verify leader, then proceed to the next node. + +## STOP conditions (halt immediately, leave limit=0, do not advance) +etcd `/readyz` not ok, a member without a leader, VIP `192.168.2.50:6443` flapping, a node +not rejoining, any Longhorn volume Faulting, or k3s crashlooping. Recovery: keep rebuilds +frozen; a botched single member can be reset and re-joined from the other two (plan §4.5). + +## Close-out (after all 3) +- All 3 nodes Ready; etcd 3/3 with a leader; `findmnt` shows `/dev/vdc` at the db path on each. +- 18/18 Longhorn volumes healthy; rebuild-limit back to **1**. +- `./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan` → **No changes** (all disks present, dev-3 memory reconciled). +- Merge `feature/ssd-etcd-isolation` → `main` and push (push is classifier-blocked in auto mode — run `git push gitea main` by hand) so Flux/tofu source records the disk + baseline. +- Regression test: after approval, bounce ONE node and confirm etcd stays healthy / VIP steady — the whole point of the change.