jakman85andClaude Opus 5
04569abd78
Two unplanned node events (prod-3 ungraceful power-down; prod-2 memory upgrade) produced one finding worth more than either incident. A FAILED `kubectl drain` STILL CORDONS, and Longhorn's `disable-scheduling-on-cordoned-node` is `true` — so a cordoned node gets no replica scheduling. Observed after prod-2's reboot: ALL 17 VOLUMES sat attached/degraded with prod-2 holding 2 of 18 replicas and ZERO rebuilds in flight. Nothing was driving recovery, every pod was Running, and no alert fired. `kubectl uncordon` started rebuilds immediately; converged 17/17 healthy in ~6 minutes (measured: 9→10→11→13→14→16→17 over 5 polls). So uncordon is not tidying up — it is what re-enables storage redundancy. A green pod list proves nothing about it. DRAIN CANNOT COMPLETE ON THIS CLUSTER, and the two usual flags do not fix it. `--ignore-daemonsets --delete-emptydir-data` clears the reported objections and then hangs: all three instance-manager PDBs report ALLOWED DISRUPTIONS = 0 and node-drain-policy is block-if-contains-last-replica. Requesting replica eviction does not rescue it either — with 3 nodes and 3 replicas there is no fourth node to move a replica to, so eviction can never complete. That is structural. Documented options in preference order: just reboot (proven twice here; the 600s replenishment interval means a quick reboot triggers no full rebuild); or scale down the workloads holding engines on that node so the PDBs free up. Changing node-drain-policy to always-allow is advised against at 3/3. Also recorded: check the kube-vip leader before rebooting a server — it holds the API VIP via a lease and is a DaemonSet, so a DRAIN does not move the VIP but a REBOOT does, and kubectl blips for seconds. Plus a post-maintenance checklist whose first item is verifying no cordon remains, and whose memory check exists because this hardware silently strands RAM on the wrong kernel (§0b). KNOWN, NOT FIXED: pod distribution does not rebalance — 52/10/50 after prod-2's cordon+reboot. For the many single-replica apps here a spread constraint would change nothing (one replica, one node). It self-corrects as pods restart. Recorded because it concentrates failure: all three CNPG databases currently run on prod-1, so losing that node takes Keycloak, Nextcloud and Semaphore together. The fix is database HA (instances: 2), a resource decision the operator owns. Header refreshed: phases A/B/C are done, 12/13 tiers, 112 pods, all three nodes at full 32 GB after node 2's DIMM upgrade. Refs R-STATE-01, R-STATE-02, R-PLAT-01. Co-Authored-By: Claude Opus 5 <[email protected]>
infra/ — node provisioning and the toolchain
OpenTofu (Proxmox VE) + cloud-init + Ansible bring up the 3-node k3s/Cilium dev cluster.
Spec: ../home-network-k3s/docs/02-environments-and-provisioning.md.
| Path | What |
|---|---|
tofu/ |
The "Create" stage: 3 cloud-init'd VMs on the Proxmox box (bpg/proxmox provider) |
cloud-init/ |
First-boot baseline shipped as a Proxmox user-data snippet |
ansible/ |
The "Baseline"/"Configure" stages: k3s HA, Cilium, kube-vip, the VPN egress gateway |
devcontainer/ |
The pinned toolchain (tofu, ansible, sops, age, flux, kubectl, helm and the whole CI gate toolchain) |
cluster.sh |
RETIRED — drove the old libvirt VMs; does not work against Proxmox |
PHASE1-STATUS.md |
Current state of the cluster and how to reconnect to it |
PHASE1-SCOPE.md |
Historical Phase-1 scoping doc (libvirt era — superseded) |
Where the cluster actually lives
The dev cluster is not on this workstation. It runs on a Proxmox VE host; the nodes sit on the Proxmox LAN and are reached across a pfSense NAT:
| Nodes | k3s-dev-1/2/3 at 192.168.1.50/.51/.52 — 6 vCPU, 24 GiB, Debian 13, k3s v1.36.4 |
| kube-API | kube-vip VIP 192.168.1.49:6443, reached from the workstation via NAT 192.168.2.50:6443 |
| Node SSH | pfSense NAT 192.168.2.102:2250/2251/2252 → node :22 (Ansible only) |
| Ingress VIP | 192.168.1.53 (Cilium L2, shared Gateway); .53–.60 reserved for LB VIPs |
The earlier libvirt/KVM cluster on 10.73.0.0/24 is retired. Any doc or command referencing
virsh, qemu:///system, or 10.73.0.x for the cluster is historical.