feat(shutdown): configure graceful node shutdown, and never scale a Delete-policy StatefulSet

All three nodes had to be powered off to move them onto a UPS, and there was
NO procedure: infra/cluster.sh stop is virsh-based and only ever applied to the
decommissioned dev cluster. The sequence was improvised live. Two defects came
out of doing it.

1. GRACEFUL NODE SHUTDOWN WAS UNCONFIGURED ON EVERY PROD NODE. No
   shutdownGracePeriod anywhere in infra/ansible/, so `shutdown -h` stopped k3s,
   containerd went with it, and every pod was SIGKILLed where it stood -- 9 CNPG
   instances mid-checkpoint, the UniFi VM, 25 attached Longhorn volumes.
   roles/k3s_server now sets kubelet-arg shutdown-grace-period=90s and
   shutdown-grace-period-critical-pods=20s.

   The half that silently defeats the other half: this is enforced by LOGIND,
   not the kubelet. The kubelet holds an inhibitor DELAY lock and systemd waits
   at most InhibitDelayMaxSec -- DEFAULT 5 SECONDS -- then proceeds regardless,
   truncating the drain with no error. The role ships a logind.conf.d drop-in
   (120) and ASSERTS cap >= grace period, because the failure is invisible.

2. SCALING A StatefulSet TO 0 DELETED ITS PVC AND VOLUME. monitoring/loki
   carries persistentVolumeClaimRetentionPolicy whenScaled: Delete, so the
   quiesce destroyed storage-loki-0 outright. Loki stores type: filesystem, so
   that PVC was the only copy; 72h retention bounded the loss, the mechanism
   does not. Symptom was the volume count going 25 -> 24, first misread as a
   miscount. Repaired: scale back to the declared 1 recreated the PVC and
   loki-0 is 2/2 Running.

New infra/maintenance/quiesce.sh (quiesce|resume|status) encodes the sequence
that worked and REFUSES to scale any whenScaled: Delete StatefulSet -- it stays
running, its volume hard-detaches, and it rebuilds on restart. A rebuild beats
a deletion. Discovery is computed from the live cluster, so a newly onboarded
service needs no edit here.

runbooks/cluster-maintenance-shutdown.md is the operator page: order in both
directions, TrueNAS-after-nodes (hard NFS mount wedges kubelet), the
un-hibernate-Postgres step Flux will NOT do for you, and a "looks broken but is
fine" section separating the 4/19 Flux cascade, the degraded-and-rebuilding
volume and the self-recovering keycloak-operator from the one thing that needs
a human (a stuck oauth2-proxy replica).

THE KUBELET CHANGE HAS NEVER BEEN APPLIED OR TESTED -- it needs a rolling k3s
restart, which is an operator action. Review config.yaml's kubelet-arg before
applying: a bad config.yaml is how k3s fails to come up. See the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

runbooks/kustomization.yaml

@@ -52,6 +52,13 @@ configMapGenerator:
       - backup-failure=backup-failure.md
       - backup-missed=backup-missed.md
       - backup-partial=backup-partial.md
+      # Operator procedure, not an alert page — nothing links here from an alert. Served because
+      # the whole-cluster shutdown had NO written procedure until 2026-10-17 (infra/cluster.sh is
+      # virsh-based and only ever applied to the decommissioned dev cluster), so the UPS move was
+      # improvised live, and the one hazard it exposed -- a StatefulSet with
+      # persistentVolumeClaimRetentionPolicy whenScaled: Delete loses its PVC when scaled to 0 --
+      # destroyed monitoring/loki's volume before anyone noticed.
+      - cluster-maintenance-shutdown=cluster-maintenance-shutdown.md
       - cdi-import-stuck=cdi-import-stuck.md
       - cert-expiry=cert-expiry.md
       - cert-issuance=cert-issuance.md