jakman85andClaude Opus 5
2f2e98aa19
All three nodes had to be powered off to move them onto a UPS, and there was NO procedure: infra/cluster.sh stop is virsh-based and only ever applied to the decommissioned dev cluster. The sequence was improvised live. Two defects came out of doing it. 1. GRACEFUL NODE SHUTDOWN WAS UNCONFIGURED ON EVERY PROD NODE. No shutdownGracePeriod anywhere in infra/ansible/, so `shutdown -h` stopped k3s, containerd went with it, and every pod was SIGKILLed where it stood -- 9 CNPG instances mid-checkpoint, the UniFi VM, 25 attached Longhorn volumes. roles/k3s_server now sets kubelet-arg shutdown-grace-period=90s and shutdown-grace-period-critical-pods=20s. The half that silently defeats the other half: this is enforced by LOGIND, not the kubelet. The kubelet holds an inhibitor DELAY lock and systemd waits at most InhibitDelayMaxSec -- DEFAULT 5 SECONDS -- then proceeds regardless, truncating the drain with no error. The role ships a logind.conf.d drop-in (120) and ASSERTS cap >= grace period, because the failure is invisible. 2. SCALING A StatefulSet TO 0 DELETED ITS PVC AND VOLUME. monitoring/loki carries persistentVolumeClaimRetentionPolicy whenScaled: Delete, so the quiesce destroyed storage-loki-0 outright. Loki stores type: filesystem, so that PVC was the only copy; 72h retention bounded the loss, the mechanism does not. Symptom was the volume count going 25 -> 24, first misread as a miscount. Repaired: scale back to the declared 1 recreated the PVC and loki-0 is 2/2 Running. New infra/maintenance/quiesce.sh (quiesce|resume|status) encodes the sequence that worked and REFUSES to scale any whenScaled: Delete StatefulSet -- it stays running, its volume hard-detaches, and it rebuilds on restart. A rebuild beats a deletion. Discovery is computed from the live cluster, so a newly onboarded service needs no edit here. runbooks/cluster-maintenance-shutdown.md is the operator page: order in both directions, TrueNAS-after-nodes (hard NFS mount wedges kubelet), the un-hibernate-Postgres step Flux will NOT do for you, and a "looks broken but is fine" section separating the 4/19 Flux cascade, the degraded-and-rebuilding volume and the self-recovering keycloak-operator from the one thing that needs a human (a stuck oauth2-proxy replica). THE KUBELET CHANGE HAS NEVER BEEN APPLIED OR TESTED -- it needs a rolling k3s restart, which is an operator action. Review config.yaml's kubelet-arg before applying: a bad config.yaml is how k3s fails to come up. See the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>