docs(prod): node-maintenance runbook — a failed drain leaves a cordon that stops self-healing

Two unplanned node events (prod-3 ungraceful power-down; prod-2 memory upgrade)
produced one finding worth more than either incident.

A FAILED `kubectl drain` STILL CORDONS, and Longhorn's
`disable-scheduling-on-cordoned-node` is `true` — so a cordoned node gets no
replica scheduling. Observed after prod-2's reboot: ALL 17 VOLUMES sat
attached/degraded with prod-2 holding 2 of 18 replicas and ZERO rebuilds in
flight. Nothing was driving recovery, every pod was Running, and no alert fired.
`kubectl uncordon` started rebuilds immediately; converged 17/17 healthy in ~6
minutes (measured: 9→10→11→13→14→16→17 over 5 polls).

So uncordon is not tidying up — it is what re-enables storage redundancy. A green
pod list proves nothing about it.

DRAIN CANNOT COMPLETE ON THIS CLUSTER, and the two usual flags do not fix it.
`--ignore-daemonsets --delete-emptydir-data` clears the reported objections and
then hangs: all three instance-manager PDBs report ALLOWED DISRUPTIONS = 0 and
node-drain-policy is block-if-contains-last-replica. Requesting replica eviction
does not rescue it either — with 3 nodes and 3 replicas there is no fourth node to
move a replica to, so eviction can never complete. That is structural. Documented
options in preference order: just reboot (proven twice here; the 600s
replenishment interval means a quick reboot triggers no full rebuild); or scale
down the workloads holding engines on that node so the PDBs free up. Changing
node-drain-policy to always-allow is advised against at 3/3.

Also recorded: check the kube-vip leader before rebooting a server — it holds the
API VIP via a lease and is a DaemonSet, so a DRAIN does not move the VIP but a
REBOOT does, and kubectl blips for seconds. Plus a post-maintenance checklist
whose first item is verifying no cordon remains, and whose memory check exists
because this hardware silently strands RAM on the wrong kernel (§0b).

KNOWN, NOT FIXED: pod distribution does not rebalance — 52/10/50 after prod-2's
cordon+reboot. For the many single-replica apps here a spread constraint would
change nothing (one replica, one node). It self-corrects as pods restart. Recorded
because it concentrates failure: all three CNPG databases currently run on prod-1,
so losing that node takes Keycloak, Nextcloud and Semaphore together. The fix is
database HA (instances: 2), a resource decision the operator owns.

Header refreshed: phases A/B/C are done, 12/13 tiers, 112 pods, all three nodes at
full 32 GB after node 2's DIMM upgrade.

Refs R-STATE-01, R-STATE-02, R-PLAT-01.

Co-Authored-By: Claude Opus 5 <[email protected]>

infra/PROD-STANDUP.md

@@ -4,10 +4,22 @@
 container. Nothing here needs a command invented on the spot; that is the point. If a step is not in
 this file it is not part of standup.
 
-State as of **2026-09-05**: phases **A** and **B** are DONE on the current hardware. The cluster is
-live (k3s v1.36.4 HA, 3 etcd members, kube-vip, Cilium 1.20.1, Longhorn) with `platform` reconciling
-and **8/9 HelmReleases Ready, 76/76 pods Running**. Outstanding work is **phase C**, blocked on the
-two items in [§Blockers](#blockers).
+State as of **2026-09-06**: phases **A**, **B** and **C** are DONE. The cluster is live — k3s
+v1.36.4 HA (3 etcd members), kube-vip, Cilium 1.20.1, Longhorn — with **12/13 Flux tiers Ready**
+(`velero` deliberately suspended), **112 pods Running**, an issued Let's Encrypt wildcard, and
+**all three nodes addressing their full 32 GB** (node 2's DIMM upgrade completed 2026-09-06 and is
+confirmed at ~`32638256Ki`, matching the other two to within 4 KiB — the §0b memattr bug is not
+biting on the backports kernel).
+
+Storage survived two unplanned node events (an ungraceful power-down of prod-3, then prod-2's memory
+upgrade) and converged to **17/17 volumes healthy, replicas symmetric across all three nodes** —
+though only after a leftover cordon was cleared; read
+[§Node maintenance](#node-maintenance--draining-rebooting-and-the-cordon-that-stops-self-healing)
+before doing node work, that cordon silently prevents Longhorn from rebuilding.
+
+What remains is in [§Blockers](#blockers), and every item there needs an operator decision:
+no backup coverage (R2 credentials), prod still on `prod.home.` rather than the apex, and — found
+2026-09-05 — **prod's UniFi VM has no disk** because Kyverno blocks CDI's generated importer pod.
 
 > **The one guard that matters.** Dev is LIVE and shares the `keithjackson.dev` Cloudflare zone with
 > prod. On 2026-09-04 prod deleted every one of dev's public DNS records. `bootstrap/standup.sh`
@@ -345,6 +357,73 @@ curl -sS --resolve gitea.prod.home.keithjackson.dev:443:192.168.2.20 \
   https://gitea.prod.home.keithjackson.dev/
 ```
 
+## NODE MAINTENANCE — draining, rebooting, and the cordon that stops self-healing
+
+Two unplanned node events on 2026-09-05/06 (an ungraceful power-down of prod-3, then a memory upgrade
+on prod-2) produced one finding worth more than either incident.
+
+### A failed `kubectl drain` still cordons — and on this cluster a cordon stops storage self-healing
+
+`kubectl drain k3s-prod-2` fails on this cluster with the usual two objections (emptyDir pods,
+DaemonSet pods) — **but it cordons the node first, and the cordon persists after the failure.**
+Longhorn's `disable-scheduling-on-cordoned-node` is **`true`**, so a cordoned node receives no replica
+scheduling. Observed consequence: after prod-2's reboot, **all 17 volumes sat `attached/degraded` with
+prod-2 holding 2 of 18 replicas and ZERO rebuilds in flight.** Nothing was driving recovery, every pod
+was `Running`, and no alert fired. `kubectl uncordon` started the rebuilds immediately; they converged
+17/17 healthy in ~6 minutes.
+
+**So: after any drain — successful or not — `kubectl uncordon <node>` is not tidying up, it is what
+re-enables storage redundancy.** Check `kubectl get nodes -o custom-columns=N:.metadata.name,\
+U:.spec.unschedulable` before believing a green pod list.
+
+### `drain` cannot complete on this cluster, and the two usual flags do not fix it
+
+`--ignore-daemonsets --delete-emptydir-data` clears the two reported objections and then **hangs
+indefinitely**: all three `instance-manager-*` PDBs report **`ALLOWED DISRUPTIONS = 0`** and
+`node-drain-policy` is `block-if-contains-last-replica`. Longhorn is refusing to let an
+instance-manager be evicted while it runs volume engines. That is data protection, not a bug.
+
+**Requesting replica eviction does not rescue it either:** with **3 nodes and 3 replicas** there is no
+fourth node to move a replica to, so eviction can never complete. This is structural, not a setting.
+
+Options, in order of preference:
+
+1. **Just reboot it (cordon, reboot, uncordon).** Proven twice on this cluster: Longhorn resyncs on
+   return, and `replica-replenishment-wait-interval` is `600`s, so a reboot completed inside ~10
+   minutes does not even trigger full rebuilds. This is the normal path for firmware, memory or kernel
+   work.
+2. **If the node must be genuinely empty**, first scale to 0 the workloads holding engines on it —
+   that detaches them, the PDBs go to 1 allowed disruption, and the drain completes. Find them with:
+   `kubectl -n longhorn-system get engines.longhorn.io -o json` filtered on `.spec.nodeID`.
+3. Changing `node-drain-policy` to `always-allow` is **not** advised at 3 nodes/3 replicas — it trades
+   the protection for exactly the data risk the default exists to prevent.
+
+### Check the kube-vip leader before rebooting a server
+
+`kube-vip` holds the API VIP (`192.168.2.9`) via a leader lease, and it is a **DaemonSet** — so
+`--ignore-daemonsets` leaves it running and a *drain* does not move the VIP. A **reboot** does: the
+lease fails over and `kubectl` blips for a few seconds. Harmless, but do not mistake it for a fault.
+Identify the holder: `kubectl -n kube-system logs -l app.kubernetes.io/name=kube-vip-ds | grep -i leader`.
+
+### Post-maintenance checklist
+
+- [ ] `kubectl uncordon <node>` — **and confirm** `.spec.unschedulable` is clear on all three.
+- [ ] Longhorn converged: `kubectl -n longhorn-system get volumes.longhorn.io` shows **0 degraded**.
+      Rebuilds are serial, so allow minutes, and watch rather than assume.
+- [ ] Replica symmetry: each node runs the same count
+      (`get replicas.longhorn.io` grouped by `.spec.nodeID` / `.status.currentState`).
+- [ ] All three nodes report full memory — prod's HP firmware publishes a corrupt EFI memattr table and
+      a wrong kernel silently strands RAM (§0b). `k3s-prod-*` should each show ~`32638256Ki`.
+- [ ] Pods: none `Running`-but-not-`Ready`.
+
+> **KNOWN, NOT FIXED — pod distribution does not rebalance.** After prod-2's cordon+reboot the spread
+> was **52 / 10 / 50**. Kubernetes does not move running pods back, and for the many SINGLE-REPLICA
+> apps here a spread constraint would change nothing (one replica lives on one node either way). It
+> self-corrects as pods restart. It is recorded because it concentrates failure: at the time of writing
+> **all three CNPG databases run on prod-1**, so losing that node takes Keycloak, Nextcloud and
+> Semaphore together. The real fix is database HA (`instances: 2`), which is a resource decision the
+> operator owns — see §Blockers.
+
 ## Blockers
 
 ### 1. Backups: endpoint DECIDED, but prod cannot reach it — **prod has no backup coverage**