From 04569abd781944e6c836a8a83f572bbca75ad16a Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 6 Sep 2026 08:35:41 -0500 Subject: [PATCH] =?UTF-8?q?docs(prod):=20node-maintenance=20runbook=20?= =?UTF-8?q?=E2=80=94=20a=20failed=20drain=20leaves=20a=20cordon=20that=20s?= =?UTF-8?q?tops=20self-healing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two unplanned node events (prod-3 ungraceful power-down; prod-2 memory upgrade) produced one finding worth more than either incident. A FAILED `kubectl drain` STILL CORDONS, and Longhorn's `disable-scheduling-on-cordoned-node` is `true` — so a cordoned node gets no replica scheduling. Observed after prod-2's reboot: ALL 17 VOLUMES sat attached/degraded with prod-2 holding 2 of 18 replicas and ZERO rebuilds in flight. Nothing was driving recovery, every pod was Running, and no alert fired. `kubectl uncordon` started rebuilds immediately; converged 17/17 healthy in ~6 minutes (measured: 9→10→11→13→14→16→17 over 5 polls). So uncordon is not tidying up — it is what re-enables storage redundancy. A green pod list proves nothing about it. DRAIN CANNOT COMPLETE ON THIS CLUSTER, and the two usual flags do not fix it. `--ignore-daemonsets --delete-emptydir-data` clears the reported objections and then hangs: all three instance-manager PDBs report ALLOWED DISRUPTIONS = 0 and node-drain-policy is block-if-contains-last-replica. Requesting replica eviction does not rescue it either — with 3 nodes and 3 replicas there is no fourth node to move a replica to, so eviction can never complete. That is structural. Documented options in preference order: just reboot (proven twice here; the 600s replenishment interval means a quick reboot triggers no full rebuild); or scale down the workloads holding engines on that node so the PDBs free up. Changing node-drain-policy to always-allow is advised against at 3/3. Also recorded: check the kube-vip leader before rebooting a server — it holds the API VIP via a lease and is a DaemonSet, so a DRAIN does not move the VIP but a REBOOT does, and kubectl blips for seconds. Plus a post-maintenance checklist whose first item is verifying no cordon remains, and whose memory check exists because this hardware silently strands RAM on the wrong kernel (§0b). KNOWN, NOT FIXED: pod distribution does not rebalance — 52/10/50 after prod-2's cordon+reboot. For the many single-replica apps here a spread constraint would change nothing (one replica, one node). It self-corrects as pods restart. Recorded because it concentrates failure: all three CNPG databases currently run on prod-1, so losing that node takes Keycloak, Nextcloud and Semaphore together. The fix is database HA (instances: 2), a resource decision the operator owns. Header refreshed: phases A/B/C are done, 12/13 tiers, 112 pods, all three nodes at full 32 GB after node 2's DIMM upgrade. Refs R-STATE-01, R-STATE-02, R-PLAT-01. Co-Authored-By: Claude Opus 5 --- infra/PROD-STANDUP.md | 87 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 83 insertions(+), 4 deletions(-) diff --git a/infra/PROD-STANDUP.md b/infra/PROD-STANDUP.md index c896007..795d053 100644 --- a/infra/PROD-STANDUP.md +++ b/infra/PROD-STANDUP.md @@ -4,10 +4,22 @@ container. Nothing here needs a command invented on the spot; that is the point. If a step is not in this file it is not part of standup. -State as of **2026-09-05**: phases **A** and **B** are DONE on the current hardware. The cluster is -live (k3s v1.36.4 HA, 3 etcd members, kube-vip, Cilium 1.20.1, Longhorn) with `platform` reconciling -and **8/9 HelmReleases Ready, 76/76 pods Running**. Outstanding work is **phase C**, blocked on the -two items in [§Blockers](#blockers). +State as of **2026-09-06**: phases **A**, **B** and **C** are DONE. The cluster is live — k3s +v1.36.4 HA (3 etcd members), kube-vip, Cilium 1.20.1, Longhorn — with **12/13 Flux tiers Ready** +(`velero` deliberately suspended), **112 pods Running**, an issued Let's Encrypt wildcard, and +**all three nodes addressing their full 32 GB** (node 2's DIMM upgrade completed 2026-09-06 and is +confirmed at ~`32638256Ki`, matching the other two to within 4 KiB — the §0b memattr bug is not +biting on the backports kernel). + +Storage survived two unplanned node events (an ungraceful power-down of prod-3, then prod-2's memory +upgrade) and converged to **17/17 volumes healthy, replicas symmetric across all three nodes** — +though only after a leftover cordon was cleared; read +[§Node maintenance](#node-maintenance--draining-rebooting-and-the-cordon-that-stops-self-healing) +before doing node work, that cordon silently prevents Longhorn from rebuilding. + +What remains is in [§Blockers](#blockers), and every item there needs an operator decision: +no backup coverage (R2 credentials), prod still on `prod.home.` rather than the apex, and — found +2026-09-05 — **prod's UniFi VM has no disk** because Kyverno blocks CDI's generated importer pod. > **The one guard that matters.** Dev is LIVE and shares the `keithjackson.dev` Cloudflare zone with > prod. On 2026-09-04 prod deleted every one of dev's public DNS records. `bootstrap/standup.sh` @@ -345,6 +357,73 @@ curl -sS --resolve gitea.prod.home.keithjackson.dev:443:192.168.2.20 \ https://gitea.prod.home.keithjackson.dev/ ``` +## NODE MAINTENANCE — draining, rebooting, and the cordon that stops self-healing + +Two unplanned node events on 2026-09-05/06 (an ungraceful power-down of prod-3, then a memory upgrade +on prod-2) produced one finding worth more than either incident. + +### A failed `kubectl drain` still cordons — and on this cluster a cordon stops storage self-healing + +`kubectl drain k3s-prod-2` fails on this cluster with the usual two objections (emptyDir pods, +DaemonSet pods) — **but it cordons the node first, and the cordon persists after the failure.** +Longhorn's `disable-scheduling-on-cordoned-node` is **`true`**, so a cordoned node receives no replica +scheduling. Observed consequence: after prod-2's reboot, **all 17 volumes sat `attached/degraded` with +prod-2 holding 2 of 18 replicas and ZERO rebuilds in flight.** Nothing was driving recovery, every pod +was `Running`, and no alert fired. `kubectl uncordon` started the rebuilds immediately; they converged +17/17 healthy in ~6 minutes. + +**So: after any drain — successful or not — `kubectl uncordon ` is not tidying up, it is what +re-enables storage redundancy.** Check `kubectl get nodes -o custom-columns=N:.metadata.name,\ +U:.spec.unschedulable` before believing a green pod list. + +### `drain` cannot complete on this cluster, and the two usual flags do not fix it + +`--ignore-daemonsets --delete-emptydir-data` clears the two reported objections and then **hangs +indefinitely**: all three `instance-manager-*` PDBs report **`ALLOWED DISRUPTIONS = 0`** and +`node-drain-policy` is `block-if-contains-last-replica`. Longhorn is refusing to let an +instance-manager be evicted while it runs volume engines. That is data protection, not a bug. + +**Requesting replica eviction does not rescue it either:** with **3 nodes and 3 replicas** there is no +fourth node to move a replica to, so eviction can never complete. This is structural, not a setting. + +Options, in order of preference: + +1. **Just reboot it (cordon, reboot, uncordon).** Proven twice on this cluster: Longhorn resyncs on + return, and `replica-replenishment-wait-interval` is `600`s, so a reboot completed inside ~10 + minutes does not even trigger full rebuilds. This is the normal path for firmware, memory or kernel + work. +2. **If the node must be genuinely empty**, first scale to 0 the workloads holding engines on it — + that detaches them, the PDBs go to 1 allowed disruption, and the drain completes. Find them with: + `kubectl -n longhorn-system get engines.longhorn.io -o json` filtered on `.spec.nodeID`. +3. Changing `node-drain-policy` to `always-allow` is **not** advised at 3 nodes/3 replicas — it trades + the protection for exactly the data risk the default exists to prevent. + +### Check the kube-vip leader before rebooting a server + +`kube-vip` holds the API VIP (`192.168.2.9`) via a leader lease, and it is a **DaemonSet** — so +`--ignore-daemonsets` leaves it running and a *drain* does not move the VIP. A **reboot** does: the +lease fails over and `kubectl` blips for a few seconds. Harmless, but do not mistake it for a fault. +Identify the holder: `kubectl -n kube-system logs -l app.kubernetes.io/name=kube-vip-ds | grep -i leader`. + +### Post-maintenance checklist + +- [ ] `kubectl uncordon ` — **and confirm** `.spec.unschedulable` is clear on all three. +- [ ] Longhorn converged: `kubectl -n longhorn-system get volumes.longhorn.io` shows **0 degraded**. + Rebuilds are serial, so allow minutes, and watch rather than assume. +- [ ] Replica symmetry: each node runs the same count + (`get replicas.longhorn.io` grouped by `.spec.nodeID` / `.status.currentState`). +- [ ] All three nodes report full memory — prod's HP firmware publishes a corrupt EFI memattr table and + a wrong kernel silently strands RAM (§0b). `k3s-prod-*` should each show ~`32638256Ki`. +- [ ] Pods: none `Running`-but-not-`Ready`. + +> **KNOWN, NOT FIXED — pod distribution does not rebalance.** After prod-2's cordon+reboot the spread +> was **52 / 10 / 50**. Kubernetes does not move running pods back, and for the many SINGLE-REPLICA +> apps here a spread constraint would change nothing (one replica lives on one node either way). It +> self-corrects as pods restart. It is recorded because it concentrates failure: at the time of writing +> **all three CNPG databases run on prod-1**, so losing that node takes Keycloak, Nextcloud and +> Semaphore together. The real fix is database HA (`instances: 2`), which is a resource decision the +> operator owns — see §Blockers. + ## Blockers ### 1. Backups: endpoint DECIDED, but prod cannot reach it — **prod has no backup coverage**