jakman85andClaude Opus 4.8
cbc2301b30
Text-only cleanup after the upgrade campaign. Updates stale version strings, prose, and comments to the verified current facts; no Kubernetes behavior or version pins changed except the kubectl toolchain bump (below). Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie, Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB, dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18, Grafana 13.2.0. - Version strings across infra/ and platform/ READMEs + comments - kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew) - Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh) - COMPLETED/historical banners on the three executed plan docs - Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges - Appended dated DECISIONS-LOG entry (2026-08-30) superseding the 2026-08-26 LIVE STATE line Left as revalidation items (not blind-edited): external-snapshotter v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1, conformance ImageVolume feature-gate assertions under 1.36. Co-Authored-By: Claude Opus 4.8 <[email protected]>
105 lines
7.1 KiB
Markdown
105 lines
7.1 KiB
Markdown
# Phase 2 execution runbook — SSD-backed etcd migration (interactive/manual)
|
|
|
|
> ## ✅ COMPLETED / historical — executed 2026-08-30, see git log
|
|
>
|
|
> **This runbook has been executed.** Every k3s node's etcd now lives on its dedicated SSD `virtio2`
|
|
> disk. The command sequence below is retained as the record of how the migration was performed — do
|
|
> NOT re-run it against the live cluster. Current state is tracked in `infra/PHASE1-STATUS.md` and the
|
|
> 2026-08-30 DECISIONS-LOG entry.
|
|
|
|
Companion to `ssd-etcd-isolation.md`. This is the **exact command sequence** to migrate
|
|
each k3s node's etcd onto its dedicated SSD `virtio2` disk. It must be run where live-infra
|
|
mutations are permitted — an **interactive `claude` session** (default permission mode, which
|
|
prompts for approval) or **by hand**. The autonomous "auto mode" session hard-denies
|
|
`tofu apply` and node-level ssh mutations, so it cannot run this.
|
|
|
|
## Preconditions (verified 2026-08-29, cluster healthy)
|
|
- Run everything from the staged worktree that carries the tofu disk block:
|
|
`cd /home/jakman85/source/home-network-k3s-impl/.claude/worktrees/agent-ssd-etcd`
|
|
(branch `feature/ssd-etcd-isolation` @ `af5fc5b`).
|
|
- All tooling via the devcontainer: `./infra/devcontainer/dev.sh run <cmd>`.
|
|
- `./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh init` once (done).
|
|
- Ansible must run from `infra/ansible/` (for the vault-pass path): wrap as
|
|
`./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible ...'`.
|
|
- **Run mutation commands STANDALONE** (not chained with `&&`/pipes) — the classifier is
|
|
more likely to deny a compound command that embeds a mutation.
|
|
|
|
## ORDER (leader last — RE-VERIFY before each node)
|
|
Current leader = **dev-3**, so: **dev-2 (vmid 102) → dev-1 (vmid 105) → dev-3 (vmid 101)**.
|
|
dev-3 is last and additionally gets the memory 16384→24576 bump (needs a full stop/start).
|
|
Re-check the leader before each node and keep the leader last:
|
|
```
|
|
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_is_leader"'
|
|
```
|
|
|
|
## Pre-flight before EACH node (read-only — all must pass)
|
|
```
|
|
./infra/devcontainer/dev.sh run kubectl get nodes
|
|
./infra/devcontainer/dev.sh run kubectl get --raw=/readyz/etcd # expect: ok
|
|
./infra/devcontainer/dev.sh run kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns=R:.status.robustness --no-headers | grep -c healthy # expect: 18
|
|
```
|
|
|
|
## Per-node procedure (substitute NODE=k3s-dev-N, VMID)
|
|
|
|
### 1. Freeze rebuilds (STANDALONE)
|
|
```
|
|
./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"0"}'
|
|
```
|
|
|
|
### 2. Add the SSD disk via tofu (STANDALONE). dev-2/dev-1 = disk only; dev-3 also memory.
|
|
```
|
|
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan '-target=proxmox_virtual_environment_vm.node["k3s-dev-N"]' # confirm: disk ADD, in-place, NOT replace
|
|
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh apply -auto-approve '-target=proxmox_virtual_environment_vm.node["k3s-dev-N"]'
|
|
```
|
|
Confirm the guest sees the new disk (hot-plug):
|
|
```
|
|
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -m shell -a "lsblk /dev/vdc"'
|
|
```
|
|
If `/dev/vdc` is NOT present (no hot-plug), power-cycle the VM (freeze is holding):
|
|
```
|
|
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible proxmox -m shell -a "qm stop VMID"' # wait until stopped
|
|
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible proxmox -m shell -a "qm start VMID"'
|
|
```
|
|
(For dev-3 the memory bump requires this stop/start regardless.)
|
|
|
|
### 3. Migrate etcd data (node commands, become=root). Do NOT skip the copy-before-mount order.
|
|
```
|
|
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "systemctl stop k3s"'
|
|
# mkfs ONLY if /dev/vdc has no filesystem yet (fresh disk):
|
|
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "blkid /dev/vdc || mkfs.ext4 -L k3s-etcd /dev/vdc"'
|
|
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "mv /var/lib/rancher/k3s/server/db /var/lib/rancher/k3s/server/db.bak && mkdir /var/lib/rancher/k3s/server/db && mount LABEL=k3s-etcd /var/lib/rancher/k3s/server/db && rsync -aHAX --numeric-ids /var/lib/rancher/k3s/server/db.bak/ /var/lib/rancher/k3s/server/db/"'
|
|
# Persist the mount (matches the node_baseline role: LABEL + defaults,noatime):
|
|
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m ansible.posix.mount -a "path=/var/lib/rancher/k3s/server/db src=LABEL=k3s-etcd fstype=ext4 opts=defaults,noatime dump=0 passno=2 state=mounted"'
|
|
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "systemctl start k3s"'
|
|
```
|
|
|
|
### 4. VERIFY full recovery before touching the next node (all must pass)
|
|
```
|
|
./infra/devcontainer/dev.sh run kubectl get nodes # NODE Ready
|
|
./infra/devcontainer/dev.sh run kubectl get --raw=/readyz/etcd # ok
|
|
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -m shell -a "findmnt /var/lib/rancher/k3s/server/db"' # source = /dev/vdc
|
|
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_has_leader"' # all =1
|
|
./infra/devcontainer/dev.sh run kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns=R:.status.robustness --no-headers | grep -c healthy # 18
|
|
# dev-3 only: confirm ~24Gi allocatable
|
|
./infra/devcontainer/dev.sh run kubectl get node k3s-dev-3 -o jsonpath='{.status.allocatable.memory}'
|
|
```
|
|
|
|
### 5. Thaw + clean up (STANDALONE), only after step 4 fully passes
|
|
```
|
|
./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"1"}'
|
|
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "rm -rf /var/lib/rancher/k3s/server/db.bak"'
|
|
```
|
|
Wait for 18/18 healthy (no rebuild storm), re-verify leader, then proceed to the next node.
|
|
|
|
## STOP conditions (halt immediately, leave limit=0, do not advance)
|
|
etcd `/readyz` not ok, a member without a leader, VIP `192.168.2.50:6443` flapping, a node
|
|
not rejoining, any Longhorn volume Faulting, or k3s crashlooping. Recovery: keep rebuilds
|
|
frozen; a botched single member can be reset and re-joined from the other two (plan §4.5).
|
|
|
|
## Close-out (after all 3)
|
|
- All 3 nodes Ready; etcd 3/3 with a leader; `findmnt` shows `/dev/vdc` at the db path on each.
|
|
- 18/18 Longhorn volumes healthy; rebuild-limit back to **1**.
|
|
- `./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan` → **No changes** (all disks present, dev-3 memory reconciled).
|
|
- Merge `feature/ssd-etcd-isolation` → `main` and push (push is classifier-blocked in auto mode — run `git push gitea main` by hand) so Flux/tofu source records the disk + baseline.
|
|
- Regression test: after approval, bounce ONE node and confirm etcd stays healthy / VIP steady — the whole point of the change.
|