jakman85andClaude Opus 5
df1d2ab44d
Records the 2026-09-07 I/O-starvation incident and the drive swap: the C400 was replaced with a Samsung 850 EVO, and under the same load class (an active 18-replica Longhorn rebuild, with node 1 as the write target) I/O pressure fell from full avg300 11.36 to 3.79 — from ~4x worse than its peers into node 3's band. Updates the three §0 cells that had become false (disk model, drive health, kernel — all three nodes now run 7.1.8, not 6.12) and adds §0c rather than rewriting the 'NOT hardware-symmetric' analysis, which is still the correct reasoning about the state it described. Two things the rebuild taught, recorded so the next one is cheaper: Longhorn refuses a replaced disk with DiskFilesystemChanged and needs disable -> remove -> re-add in that order, and the inventory's [k3s_prod_primary] decides BOTH --cluster-init and k3s_api_host, so rebuilding the founder without moving that designation would have founded a second cluster. The replacement does NOT close §2: etcd and Longhorn still share one SATA spindle. Co-Authored-By: Claude Opus 5 <[email protected]>
infra/ — node provisioning and the toolchain
OpenTofu (Proxmox VE) + cloud-init + Ansible bring up the 3-node k3s/Cilium dev cluster.
Spec: ../home-network-k3s/docs/02-environments-and-provisioning.md.
| Path | What |
|---|---|
tofu/ |
The "Create" stage: 3 cloud-init'd VMs on the Proxmox box (bpg/proxmox provider) |
cloud-init/ |
First-boot baseline shipped as a Proxmox user-data snippet |
ansible/ |
The "Baseline"/"Configure" stages: k3s HA, Cilium, kube-vip, the VPN egress gateway |
devcontainer/ |
The pinned toolchain (tofu, ansible, sops, age, flux, kubectl, helm and the whole CI gate toolchain) |
cluster.sh |
RETIRED — drove the old libvirt VMs; does not work against Proxmox |
PHASE1-STATUS.md |
Current state of the cluster and how to reconnect to it |
PHASE1-SCOPE.md |
Historical Phase-1 scoping doc (libvirt era — superseded) |
Where the cluster actually lives
The dev cluster is not on this workstation. It runs on a Proxmox VE host; the nodes sit on the Proxmox LAN and are reached across a pfSense NAT:
| Nodes | k3s-dev-1/2/3 at 192.168.1.50/.51/.52 — 6 vCPU, 24 GiB, Debian 13, k3s v1.36.4 |
| kube-API | kube-vip VIP 192.168.1.49:6443, reached from the workstation via NAT 192.168.2.50:6443 |
| Node SSH | pfSense NAT 192.168.2.102:2250/2251/2252 → node :22 (Ansible only) |
| Ingress VIP | 192.168.1.53 (Cilium L2, shared Gateway); .53–.60 reserved for LB VIPs |
The earlier libvirt/KVM cluster on 10.73.0.0/24 is retired. Any doc or command referencing
virsh, qemu:///system, or 10.73.0.x for the cluster is historical.