All three nodes had to be powered off to move them onto a UPS, and there was
NO procedure: infra/cluster.sh stop is virsh-based and only ever applied to the
decommissioned dev cluster. The sequence was improvised live. Two defects came
out of doing it.
1. GRACEFUL NODE SHUTDOWN WAS UNCONFIGURED ON EVERY PROD NODE. No
shutdownGracePeriod anywhere in infra/ansible/, so `shutdown -h` stopped k3s,
containerd went with it, and every pod was SIGKILLed where it stood -- 9 CNPG
instances mid-checkpoint, the UniFi VM, 25 attached Longhorn volumes.
roles/k3s_server now sets kubelet-arg shutdown-grace-period=90s and
shutdown-grace-period-critical-pods=20s.
The half that silently defeats the other half: this is enforced by LOGIND,
not the kubelet. The kubelet holds an inhibitor DELAY lock and systemd waits
at most InhibitDelayMaxSec -- DEFAULT 5 SECONDS -- then proceeds regardless,
truncating the drain with no error. The role ships a logind.conf.d drop-in
(120) and ASSERTS cap >= grace period, because the failure is invisible.
2. SCALING A StatefulSet TO 0 DELETED ITS PVC AND VOLUME. monitoring/loki
carries persistentVolumeClaimRetentionPolicy whenScaled: Delete, so the
quiesce destroyed storage-loki-0 outright. Loki stores type: filesystem, so
that PVC was the only copy; 72h retention bounded the loss, the mechanism
does not. Symptom was the volume count going 25 -> 24, first misread as a
miscount. Repaired: scale back to the declared 1 recreated the PVC and
loki-0 is 2/2 Running.
New infra/maintenance/quiesce.sh (quiesce|resume|status) encodes the sequence
that worked and REFUSES to scale any whenScaled: Delete StatefulSet -- it stays
running, its volume hard-detaches, and it rebuilds on restart. A rebuild beats
a deletion. Discovery is computed from the live cluster, so a newly onboarded
service needs no edit here.
runbooks/cluster-maintenance-shutdown.md is the operator page: order in both
directions, TrueNAS-after-nodes (hard NFS mount wedges kubelet), the
un-hibernate-Postgres step Flux will NOT do for you, and a "looks broken but is
fine" section separating the 4/19 Flux cascade, the degraded-and-rebuilding
volume and the self-recovering keycloak-operator from the one thing that needs
a human (a stuck oauth2-proxy replica).
THE KUBELET CHANGE HAS NEVER BEEN APPLIED OR TESTED -- it needs a rolling k3s
restart, which is an operator action. Review config.yaml's kubelet-arg before
applying: a bad config.yaml is how k3s fails to come up. See the handoff entry.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
jakman85andClaude Opus 5