feat(shutdown): configure graceful node shutdown, and never scale a Delete-policy StatefulSet

All three nodes had to be powered off to move them onto a UPS, and there was
NO procedure: infra/cluster.sh stop is virsh-based and only ever applied to the
decommissioned dev cluster. The sequence was improvised live. Two defects came
out of doing it.

1. GRACEFUL NODE SHUTDOWN WAS UNCONFIGURED ON EVERY PROD NODE. No
   shutdownGracePeriod anywhere in infra/ansible/, so `shutdown -h` stopped k3s,
   containerd went with it, and every pod was SIGKILLed where it stood -- 9 CNPG
   instances mid-checkpoint, the UniFi VM, 25 attached Longhorn volumes.
   roles/k3s_server now sets kubelet-arg shutdown-grace-period=90s and
   shutdown-grace-period-critical-pods=20s.

   The half that silently defeats the other half: this is enforced by LOGIND,
   not the kubelet. The kubelet holds an inhibitor DELAY lock and systemd waits
   at most InhibitDelayMaxSec -- DEFAULT 5 SECONDS -- then proceeds regardless,
   truncating the drain with no error. The role ships a logind.conf.d drop-in
   (120) and ASSERTS cap >= grace period, because the failure is invisible.

2. SCALING A StatefulSet TO 0 DELETED ITS PVC AND VOLUME. monitoring/loki
   carries persistentVolumeClaimRetentionPolicy whenScaled: Delete, so the
   quiesce destroyed storage-loki-0 outright. Loki stores type: filesystem, so
   that PVC was the only copy; 72h retention bounded the loss, the mechanism
   does not. Symptom was the volume count going 25 -> 24, first misread as a
   miscount. Repaired: scale back to the declared 1 recreated the PVC and
   loki-0 is 2/2 Running.

New infra/maintenance/quiesce.sh (quiesce|resume|status) encodes the sequence
that worked and REFUSES to scale any whenScaled: Delete StatefulSet -- it stays
running, its volume hard-detaches, and it rebuilds on restart. A rebuild beats
a deletion. Discovery is computed from the live cluster, so a newly onboarded
service needs no edit here.

runbooks/cluster-maintenance-shutdown.md is the operator page: order in both
directions, TrueNAS-after-nodes (hard NFS mount wedges kubelet), the
un-hibernate-Postgres step Flux will NOT do for you, and a "looks broken but is
fine" section separating the 4/19 Flux cascade, the degraded-and-rebuilding
volume and the self-recovering keycloak-operator from the one thing that needs
a human (a stuck oauth2-proxy replica).

THE KUBELET CHANGE HAS NEVER BEEN APPLIED OR TESTED -- it needs a rolling k3s
restart, which is an operator action. Review config.yaml's kubelet-arg before
applying: a bad config.yaml is how k3s fails to come up. See the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

infra/ansible/roles/k3s_server/defaults/main.yml

@@ -107,3 +107,31 @@ k3s_base_domain: "home.keithjackson.dev"
 
 # Derived, so the realm path is stated once. Override k3s_base_domain, not this.
 k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"
+
+# --- Graceful node shutdown (kubelet) ----------------------------------------------------------
+# MEASURED 2026-10-17, during the UPS move: graceful node shutdown was NOT configured on any prod
+# node, so `shutdown -h` did NOT drain pods. systemd stops k3s, containerd goes with it, and every
+# pod is SIGKILLed where it stands -- including 9 CloudNativePG instances mid-checkpoint, the UniFi
+# VM, and 25 attached Longhorn volumes. The whole-cluster shutdown that day had to quiesce
+# everything BY HAND first (runbooks/cluster-maintenance-shutdown.md), and the one volume that did
+# not finish detaching came back `degraded` and had to rebuild.
+#
+# With these set, the kubelet takes a systemd inhibitor lock and, on shutdown, evicts ordinary pods
+# first and critical ones last, so a plain `shutdown -h`/reboot drains the node. That makes the
+# manual scale-down in the runbook a belt-and-braces step for a PLANNED whole-cluster shutdown
+# rather than the only thing standing between a reboot and a hard kill.
+#
+# 90s/20s, not larger: the budget is spent on every reboot and node upgrade, and the kubelet waits
+# the FULL ordinary period before it starts on critical pods. Postgres does a fast shutdown in
+# well under 90s here, and Longhorn needs only enough time to flush and detach.
+k3s_kubelet_shutdown_grace_period: "90s"
+k3s_kubelet_shutdown_grace_period_critical_pods: "20s"
+
+# *** THE HALF THAT IS EASY TO MISS, AND SILENTLY BREAKS THE OTHER HALF. ***
+# Graceful node shutdown works by the kubelet holding a logind INHIBITOR DELAY lock. systemd caps
+# how long any inhibitor may delay shutdown at `InhibitDelayMaxSec`, which DEFAULTS TO 5 SECONDS.
+# If that cap is below the grace period above, systemd proceeds anyway and the drain is truncated
+# to ~5s -- i.e. the feature appears configured, reports no error, and still hard-kills pods. So
+# this MUST be >= k3s_kubelet_shutdown_grace_period; the role asserts it rather than trusting it.
+# Set a little above the kubelet's total (90s + 20s) to leave headroom.
+k3s_logind_inhibit_delay_max_sec: 120

infra/ansible/roles/k3s_server/handlers/main.yml

@@ -15,3 +15,12 @@
     port: "{{ k3s_api_port }}"
     timeout: 180
   listen: Restart k3s
+
+# logind re-reads InhibitDelayMaxSec only on restart. Restarting logind does NOT log users out or
+# disturb k3s -- it manages login sessions and inhibitor locks, not workloads. The kubelet picks up
+# the new cap when it next takes its lock, so no k3s restart is needed for THIS half.
+- name: Restart logind
+  ansible.builtin.systemd_service:
+    name: systemd-logind
+    state: restarted
+  listen: Restart logind

infra/ansible/roles/k3s_server/tasks/main.yml

@@ -86,6 +86,16 @@
       # k3s_base_domain (defaults/main.yml), whose [parity] contract with the cluster's
       # BASE_DOMAIN is documented there. Dev's default renders the value it already runs, so
       # config.yaml is unchanged and the Restart handler does not fire.
+      # ── Graceful node shutdown ──────────────────────────────────────────────────────────────
+      # Without these the kubelet takes no systemd inhibitor lock, so `shutdown -h` stops
+      # containerd and SIGKILLs every pod where it stands -- MEASURED on all three prod nodes
+      # 2026-10-17 (defaults/main.yml has the detail). The companion logind drop-in below raises
+      # InhibitDelayMaxSec, WITHOUT WHICH systemd caps the drain at ~5s and these values are
+      # quietly ignored.
+      kubelet-arg:
+        - "shutdown-grace-period={{ k3s_kubelet_shutdown_grace_period }}"
+        - "shutdown-grace-period-critical-pods={{ k3s_kubelet_shutdown_grace_period_critical_pods }}"
+
       kube-apiserver-arg:
         - "oidc-issuer-url={{ k3s_oidc_issuer_url }}"
         - "oidc-client-id=kubernetes"
@@ -183,3 +193,46 @@
     host: "127.0.0.1"
     port: "{{ k3s_api_port }}"
     timeout: 180
+
+# ── Graceful node shutdown: the systemd half — tag `shutdown` ──────────────────────────────────
+# The kubelet's shutdown-grace-period is ENFORCED BY logind, not by the kubelet: the kubelet holds
+# an inhibitor DELAY lock, and systemd will wait at most `InhibitDelayMaxSec` for it (default 5s)
+# before proceeding regardless. A grace period above that cap is therefore silently truncated --
+# the feature looks configured, logs no error, and still hard-kills pods. Assert the relationship
+# instead of documenting it, because the failure is invisible.
+- name: Assert the logind inhibitor cap can actually cover the kubelet grace period
+  ansible.builtin.assert:
+    that:
+      - (k3s_logind_inhibit_delay_max_sec | int) >= (k3s_kubelet_shutdown_grace_period | regex_replace('[^0-9]', '') | int)
+    fail_msg: >-
+      k3s_logind_inhibit_delay_max_sec ({{ k3s_logind_inhibit_delay_max_sec }}s) is below
+      k3s_kubelet_shutdown_grace_period ({{ k3s_kubelet_shutdown_grace_period }}). systemd would
+      stop waiting for the kubelet's inhibitor lock before the drain finished, so pods would be
+      SIGKILLed anyway and graceful node shutdown would be configured but ineffective.
+    quiet: true
+  tags: [shutdown]
+
+- name: Ensure /etc/systemd/logind.conf.d exists
+  ansible.builtin.file:
+    path: /etc/systemd/logind.conf.d
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+  tags: [shutdown]
+
+# A drop-in, not an edit of logind.conf: the distro file stays pristine and `state: absent` here
+# would fully revert the change.
+- name: Raise the logind inhibitor delay cap so the kubelet can finish draining
+  ansible.builtin.copy:
+    dest: /etc/systemd/logind.conf.d/10-k3s-graceful-shutdown.conf
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      # Managed by Ansible (roles/k3s_server). See defaults/main.yml.
+      # MUST stay >= kubelet shutdown-grace-period, or the drain is truncated to this value.
+      [Login]
+      InhibitDelayMaxSec={{ k3s_logind_inhibit_delay_max_sec }}
+  notify: Restart logind
+  tags: [shutdown]

infra/maintenance/quiesce.sh

@@ -0,0 +1,265 @@
+#!/usr/bin/env bash
+# =================================================================================================
+# quiesce.sh — bring the prod cluster to a safe standstill before powering nodes off, and bring it
+#              back afterwards. Companion to runbooks/cluster-maintenance-shutdown.md.
+# =================================================================================================
+# WHY THIS EXISTS. On 2026-10-17 all three nodes had to be powered off to move them onto a UPS, and
+# there was NO procedure for it: `infra/cluster.sh stop` is `virsh`-based and only ever applied to
+# the decommissioned dev cluster. The whole sequence was improvised live. This script is that
+# sequence, so the next person does not improvise it.
+#
+# WHY IT IS NOT JUST `shutdown -h`. Graceful node shutdown was unconfigured on every prod node that
+# day, so systemd stopped containerd and SIGKILLed every pod where it stood. roles/k3s_server now
+# sets shutdown-grace-period (+ the logind InhibitDelayMaxSec cap that actually enforces it), which
+# makes a bare reboot safe. This script is still the right tool for a PLANNED whole-cluster
+# shutdown, for three reasons the kubelet's drain does not cover:
+#   1. The UniFi VM needs an ACPI guest shutdown, not an eviction.
+#   2. CloudNativePG wants hibernation, not pod deletion -- and hibernation is IMPERATIVE state
+#      that Flux will not set or clear for you.
+#   3. Draining three nodes one at a time makes pods RESCHEDULE onto the survivors, churning
+#      Longhorn replicas for no reason when the whole cluster is going down anyway. Scaling in
+#      place avoids all of it.
+#
+# SAFE TO RE-RUN. Every step is idempotent and `resume` is the exact inverse of `quiesce`.
+#
+# WHAT IT DELIBERATELY DOES NOT DO: power anything off. Node poweroff is an operator action
+# (CLAUDE.md §5) and needs the become password; the runbook gives the ansible one-liner. It also
+# does not touch TrueNAS (not in the inventory, SSH refused -- web UI only) or the router.
+set -euo pipefail
+
+CONTEXT="${CONTEXT:-k3s-prod}"
+KUBECTL=(kubectl --context "$CONTEXT" --request-timeout=30s)
+TIMEOUT="${TIMEOUT:-600}"
+
+# The three CloudNativePG clusters, as namespace/name. Hibernation is per-Cluster.
+CNPG=(keycloak/keycloak-db nextcloud-db/nextcloud-db semaphore-db/semaphore-db)
+
+log()  { printf '\033[34m[quiesce]\033[0m %s\n' "$*"; }
+ok()   { printf '\033[32m[ ok ]\033[0m %s\n' "$*"; }
+warn() { printf '\033[33m[warn]\033[0m %s\n' "$*"; }
+die()  { printf '\033[31m[fail]\033[0m %s\n' "$*" >&2; exit 1; }
+
+need() { command -v "$1" >/dev/null || die "$1 not on PATH — run this inside ./infra/devcontainer/dev.sh"; }
+
+# ---------------------------------------------------------------- discovery
+# Every Deployment/StatefulSet whose pods mount a PVC — Longhorn OR the TrueNAS NFS bulk tier.
+# Computed from the LIVE cluster rather than hardcoded, so a newly onboarded service is covered
+# without anyone remembering to add it here. PROTECTED namespaces are excluded because scaling
+# them down would remove the very machinery that performs the detach (longhorn-system), keep the
+# API reachable (kube-system), or finish the hibernation and the VM shutdown (cnpg-system,
+# kubevirt).
+pvc_workloads() {
+  "${KUBECTL[@]}" get pods -A -o json | python3 -c '
+import json,sys
+PROTECT={"longhorn-system","kube-system","cilium-secrets","cnpg-system","kubevirt","flux-system","cdi"}
+out={}
+for p in json.load(sys.stdin)["items"]:
+    ns=p["metadata"]["namespace"]
+    if ns in PROTECT: continue
+    if not any(v.get("persistentVolumeClaim") for v in (p["spec"].get("volumes") or [])): continue
+    ow=(p["metadata"].get("ownerReferences") or [])
+    if not ow: continue
+    k,n=ow[0]["kind"],ow[0]["name"]
+    if k=="ReplicaSet": k,n="deployment",n.rsplit("-",1)[0]
+    elif k=="StatefulSet": k="statefulset"
+    else: continue          # Jobs, and CNPG Clusters, are handled separately
+    out[(ns,k,n)]=1
+for ns,k,n in sorted(out): print(ns,k,n)
+'
+}
+
+# ---------------------------------------------------------------- the PVC-deletion guard
+# *** DO NOT REMOVE THIS CHECK. IT EXISTS BECAUSE SCALING A StatefulSet DESTROYED DATA. ***
+# A StatefulSet may carry `persistentVolumeClaimRetentionPolicy.whenScaled: Delete`, and then
+# scaling it to 0 DELETES ITS PVCs -- and with them the Longhorn volumes. MEASURED 2026-10-17: the
+# quiesce for the UPS move scaled `monitoring/loki` (whenScaled: Delete) to 0 and its
+# `storage-loki-0` PVC and volume were deleted outright. Loki stores with `type: filesystem`, so
+# that PVC was the only copy; 72h retention bounded the loss to a rolling window, but nothing about
+# the mechanism is bounded -- the same action on a Delete-policy StatefulSet holding real state
+# would be unrecoverable. The volume count going 25 -> 24 was the only visible symptom, and it was
+# first misread as a miscount.
+#
+# So: such a StatefulSet is NEVER scaled. It is left RUNNING and its volume hard-detaches at
+# poweroff, which costs a replica rebuild on restart (degraded -> healthy, automatic). A rebuild is
+# an enormously better outcome than a deletion, and that is the whole trade being made here.
+sts_deletes_pvc_on_scale() { # ns name -> 0 if scaling it would delete its PVCs
+  local policy
+  policy=$("${KUBECTL[@]}" -n "$1" get statefulset "$2" \
+            -o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy.whenScaled}' 2>/dev/null || true)
+  [ "$policy" = "Delete" ]
+}
+
+wait_for() { # wait_for <description> <shell-condition>
+  local desc="$1" cond="$2" waited=0
+  log "waiting: $desc (up to ${TIMEOUT}s)"
+  until eval "$cond"; do
+    (( waited += 10 )); (( waited >= TIMEOUT )) && { warn "TIMED OUT waiting: $desc"; return 1; }
+    sleep 10
+  done
+  ok "$desc"
+}
+
+# ---------------------------------------------------------------- quiesce
+do_quiesce() {
+  need kubectl; need python3
+
+  # Refuse to start if Longhorn is already unhealthy. Powering off on top of a degraded volume is
+  # how one replica becomes zero; better to stop here and let it rebuild first.
+  local unhealthy
+  unhealthy=$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io \
+    -o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true)
+  [ "$unhealthy" = "0" ] || die "$unhealthy Longhorn volume(s) are not healthy — fix or wait for rebuild BEFORE powering off"
+  ok "Longhorn: all volumes healthy"
+
+  # 1. Flux first, or it reverts every scale-down below within its reconcile interval.
+  log "suspending Flux Kustomizations"
+  local k
+  for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do
+    "${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":true}}' >/dev/null
+  done
+  ok "all Kustomizations suspended"
+
+  # 2. The VM needs an ACPI shutdown of the GUEST. KubeVirt does that on running:false; a plain
+  #    pod eviction would be a power cut to the VM.
+  if "${KUBECTL[@]}" -n unifi get vm unifi >/dev/null 2>&1; then
+    log "ACPI-stopping the UniFi VM"
+    "${KUBECTL[@]}" -n unifi patch vm unifi --type=merge -p '{"spec":{"running":false}}' >/dev/null
+  fi
+
+  # 3. Postgres: hibernation is a clean smart shutdown that KEEPS the PVCs. CNPG does one instance
+  #    at a time, primary last-ish, so this is not instant.
+  log "hibernating CloudNativePG clusters"
+  local c ns name
+  for c in "${CNPG[@]}"; do
+    ns="${c%%/*}"; name="${c##*/}"
+    "${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \
+      cnpg.io/hibernation=on --overwrite >/dev/null 2>&1 || warn "could not hibernate $c"
+  done
+
+  # 4. Everything holding a PVC, scaled in place. NFS consumers first so nothing holds a hard
+  #    TrueNAS mount; PiHole last because it is the LAN resolver.
+  log "scaling down PVC-holding workloads"
+  local line ns2 kind nm deferred=() skipped_delete_policy=()
+  while read -r ns2 kind nm; do
+    [ -z "${ns2:-}" ] && continue
+    if [ "$kind" = "statefulset" ] && sts_deletes_pvc_on_scale "$ns2" "$nm"; then
+      warn "SKIPPING $ns2/$nm — whenScaled: Delete would DELETE its PVC (see the guard above)."
+      warn "  Left running; its volume hard-detaches at poweroff and rebuilds on restart."
+      skipped_delete_policy+=("$ns2/$nm")
+      continue
+    fi
+    if [ "$ns2" = "pihole" ]; then deferred+=("$ns2 $kind $nm"); continue; fi
+    "${KUBECTL[@]}" -n "$ns2" scale "$kind" "$nm" --replicas=0 >/dev/null 2>&1 \
+      && printf '    scaled 0  %s/%s %s\n' "$ns2" "$kind" "$nm" \
+      || warn "could not scale $ns2/$kind/$nm"
+  done < <(pvc_workloads)
+  for line in "${deferred[@]:-}"; do
+    [ -z "${line:-}" ] && continue
+    # shellcheck disable=SC2086
+    set -- $line
+    "${KUBECTL[@]}" -n "$1" scale "$2" "$3" --replicas=0 >/dev/null 2>&1 \
+      && printf '    scaled 0  %s/%s %s  (LAN DNS — expect name resolution to stop)\n' "$1" "$2" "$3"
+  done
+
+  # 5. The only assertion that matters. An attached volume at poweroff comes back degraded and
+  #    rebuilds; this is what we are buying with all of the above.
+  wait_for "UniFi VMI gone" \
+    '[ "$('"${KUBECTL[*]}"' -n unifi get vmi --no-headers 2>/dev/null | wc -l)" = "0" ]' || true
+  wait_for "all Postgres instances stopped" \
+    '[ "$('"${KUBECTL[*]}"' get pods -A --no-headers 2>/dev/null | grep -cE "(keycloak|nextcloud|semaphore)-db-[0-9]")" = "0" ]' || true
+  wait_for "all Longhorn volumes detached" \
+    '[ "$('"${KUBECTL[*]}"' -n longhorn-system get volumes.longhorn.io -o jsonpath="{range .items[*]}{.status.state}{\"\n\"}{end}" 2>/dev/null | grep -c attached)" = "0" ]' || \
+      warn "some volumes are STILL ATTACHED — they will come back degraded and rebuild; acceptable, but note which"
+
+  if [ "${#skipped_delete_policy[@]}" -gt 0 ]; then
+    echo
+    warn "NOT scaled down (whenScaled: Delete — scaling them would delete their PVCs):"
+    printf '      %s\n' "${skipped_delete_policy[@]}"
+    warn "Expect these volumes to come back 'degraded' and rebuild. That is correct and automatic."
+  fi
+  echo
+  ok "QUIESCED. Safe to power off."
+  cat <<'NEXT'
+
+  Next, from inside the devcontainer (it holds the become password):
+    cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \
+      -m shell -a 'shutdown -h +1 "maintenance"'
+
+  THEN, in this order and not before:
+    1. TrueNAS (192.168.2.30) — web UI only; not in the ansible inventory and SSH is refused.
+    2. Router / switch last.
+NEXT
+}
+
+# ---------------------------------------------------------------- resume
+do_resume() {
+  need kubectl
+
+  # THE STEP THAT IS ALWAYS FORGOTTEN. Hibernation is imperative state: Flux neither sets nor
+  # clears it, so resuming Flux alone leaves all three databases down and every app that needs
+  # one looking broken for reasons that point nowhere near Postgres.
+  log "un-hibernating CloudNativePG clusters"
+  local c ns name
+  for c in "${CNPG[@]}"; do
+    ns="${c%%/*}"; name="${c##*/}"
+    "${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \
+      cnpg.io/hibernation=off --overwrite >/dev/null 2>&1 || warn "could not un-hibernate $c"
+  done
+
+  # Flux restores every replica count AND the VM's running:true from git, so nothing here needs to
+  # undo the scale-downs by hand.
+  log "resuming Flux Kustomizations"
+  local k
+  for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do
+    "${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":false}}' >/dev/null
+  done
+  ok "all Kustomizations resumed"
+
+  wait_for "Flux fully reconciled" \
+    '[ "$('"${KUBECTL[*]}"' -n flux-system get kustomizations --no-headers 2>/dev/null | grep -vc True)" = "0" ]' || \
+      warn "some Kustomizations are not Ready — the graph serialises behind platform-config, so give it a few minutes before worrying"
+
+  echo
+  ok "RESUMED."
+  cat <<'NEXT'
+
+  Known cold-start race, not a fault in this script: oauth2-proxy performs OIDC discovery at
+  startup and exits 1 on the 503 it gets while Keycloak is still coming up, then sits in
+  exponential CrashLoopBackOff. If a replica is stuck, delete the pod once:
+    kubectl --context k3s-prod -n oauth2-proxy delete pod <name>
+NEXT
+}
+
+do_status() {
+  need kubectl
+  printf 'nodes:      %s Ready\n' "$("${KUBECTL[@]}" get nodes --no-headers | grep -c ' Ready')"
+  printf 'flux:       %s/%s Ready\n' \
+    "$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | grep -c True)" \
+    "$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | wc -l)"
+  printf 'longhorn:   %s attached, %s detached, %s not healthy\n' \
+    "$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c attached)" \
+    "$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c detached)" \
+    "$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true)"
+  printf 'postgres:   %s instance pod(s)\n' \
+    "$("${KUBECTL[@]}" get pods -A --no-headers | grep -cE '(keycloak|nextcloud|semaphore)-db-[0-9]' || true)"
+  printf 'unifi VM:   %s VMI\n' "$("${KUBECTL[@]}" -n unifi get vmi --no-headers 2>/dev/null | wc -l)"
+}
+
+case "${1:-}" in
+  quiesce) do_quiesce ;;
+  resume)  do_resume ;;
+  status)  do_status ;;
+  *) cat >&2 <<USAGE
+usage: $0 {quiesce|resume|status}
+
+  quiesce  suspend Flux, ACPI-stop the UniFi VM, hibernate Postgres, scale down every
+           PVC-holding workload, and wait for all Longhorn volumes to detach.
+           Does NOT power anything off — see runbooks/cluster-maintenance-shutdown.md.
+  resume   the exact inverse: un-hibernate Postgres, resume Flux, wait for reconcile.
+  status   one-screen readout of nodes / Flux / Longhorn / Postgres / VM.
+
+Run it inside the pinned toolbox, which supplies kubectl and the context:
+  ./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
+USAGE
+     exit 2 ;;
+esac