home-network-k3s-impl/decisions/2026-10-17-graceful-cluster-shutdown.md

date, branch, title

date	branch	title
2026-10-17	feat/graceful-cluster-shutdown	Graceful node shutdown was never configured, and scaling a StatefulSet deleted its PVC

What changed

All three nodes had to be powered off to move them onto a UPS. There was no procedure for it: infra/cluster.sh stop is virsh-based and only ever applied to the decommissioned dev cluster, so the whole sequence was improvised live. Two defects came out of doing it.

1. Kubelet graceful node shutdown was unconfigured on every prod node. No shutdownGracePeriod anywhere in infra/ansible/, so shutdown -h stopped k3s, containerd went with it, and every pod was SIGKILLed where it stood — 9 CloudNativePG instances mid-checkpoint, the UniFi VM, and 25 attached Longhorn volumes. roles/k3s_server now writes kubelet-arg into /etc/rancher/k3s/config.yaml with shutdown-grace-period=90s and shutdown-grace-period-critical-pods=20s.

The half that is easy to miss and silently defeats the other half: graceful node shutdown is enforced by logind, not by the kubelet. The kubelet holds an inhibitor delay lock and systemd waits at most InhibitDelayMaxSec — default 5 seconds — before proceeding regardless. A grace period above that cap is truncated with no error, so the feature looks configured and still hard-kills pods. The role now ships a logind.conf.d drop-in setting it to 120, and asserts k3s_logind_inhibit_delay_max_sec >= k3s_kubelet_shutdown_grace_period rather than documenting the relationship, because the failure is invisible.

2. Scaling a StatefulSet to 0 DELETED its PVC and Longhorn volume. monitoring/loki carries persistentVolumeClaimRetentionPolicy: {whenDeleted: Delete, whenScaled: Delete}, so the quiesce's scale --replicas=0 destroyed storage-loki-0 outright. Loki stores with type: filesystem, so that PVC was the only copy of its log data. This is the defect in this change set that actually destroyed something, and it was mine.

New infra/maintenance/quiesce.sh (quiesce | resume | status) encodes the sequence that worked, and refuses to scale any StatefulSet whose whenScaled is Delete — it leaves it running, accepts that its volume hard-detaches at poweroff, and prints which volumes to expect degraded. A rebuild is a vastly better outcome than a deletion. Workload discovery is computed from the live cluster (every Deployment/StatefulSet whose pods mount a PVC, Longhorn or TrueNAS NFS) rather than hardcoded, so a newly onboarded service is covered without anyone remembering to add it.

runbooks/cluster-maintenance-shutdown.md (new, registered in runbooks/kustomization.yaml) is the operator page: shutdown order, the TrueNAS-after-nodes constraint, power-on order, the un-hibernate-Postgres step Flux will not do for you, and a "what looks broken but is fine" section separating the 4/19-Kustomizations-Ready cascade, the degraded-and-rebuilding volume and the self-recovering keycloak-operator from the one thing that genuinely needs a human (a stuck oauth2-proxy replica).

What I verified

• Graceful shutdown was genuinely absent, not merely undocumented: grep -rn 'shutdownGracePeriod|graceful-node-shutdown' over infra/ansible/ returns nothing, and no logind or InhibitDelay setting exists anywhere in the tree.
• The whole shutdown/restart cycle was executed end to end on prod 2026-10-17 and the cluster came back: 3/3 nodes Ready with etcd healthz ok, 19/19 Flux Kustomizations Ready, platform-config's lastAppliedRevision equal to the GitRepository artifact revision, all 3 CNPG clusters healthy with primaries elected, the UniFi VM Running, PiHole resolving (home.keithjackson.dev → 192.168.2.20 via 192.168.2.23), and the Gateway serving 200/200/401 for apex/gitea/plex.
• The PVC-deletion mechanism is confirmed, not inferred: kubectl get sts loki -o jsonpath={.spec.persistentVolumeClaimRetentionPolicy} returns {"whenDeleted":"Delete","whenScaled":"Delete"}, the PVC was NotFound after the cycle, and the Longhorn volume count had gone 25 → 24.
• The damage was repaired and verified: scale sts loki --replicas=1 (restoring the replica count the chart declares) recreated storage-loki-0 as a fresh 50Gi Bound PVC, loki-0 reached 2/2 Running, and the volume count returned to 25.
• The guard works against the live cluster, tested both ways: sts_deletes_pvc_on_scale monitoring loki returns true (would be skipped) and pihole pihole returns false (safe to scale).
• The script's read-only paths run clean on prod: status reports correctly, and the discovery function returns the 13 workloads currently holding a PVC.
• bash -n clean; the two edited Ansible YAML files parse.

What I did NOT verify

• The kubelet change has NEVER BEEN APPLIED OR TESTED. This is the significant gap. I did not run the role, did not restart k3s, and have not observed a node actually drain on shutdown -h. Everything about it is reasoning from kubelet/systemd semantics plus the measured absence of the setting. Applying it needs a rolling k3s restart (serial: 1 — kube-vip keeps the API up, etcd holds at 2/3), which is an operator action (CLAUDE.md §5). Verify by draining one node for real: apply, then shutdown -h now on a single node and watch whether pods terminate gracefully instead of vanishing.
• Whether 90s/20s are the right numbers. Chosen to comfortably cover a Postgres fast shutdown and a Longhorn flush; not measured against either. The budget is spent on every reboot, so if a node takes noticeably longer to go down, this is why.
• Whether k3s accepts kubelet-arg in config.yaml as written. The file already carries kube-apiserver-arg as a list and kubelet-arg is the documented sibling, but I did not render or apply it, so a typo would surface only at k3s restart — and a bad config.yaml is exactly the kind of thing that makes k3s fail to come up. Review this one before applying.
• The assert task's regex_replace on the grace period assumes a plain 90s-style value; a value expressed as 1m30s would parse to the nonsense integer 130. Fine for the committed defaults, wrong if someone writes a compound duration.
• resume's Flux-reconcile wait has never run, and neither has quiesce as a script — the 2026-10-17 cycle was done with the equivalent commands by hand, so the script encodes a sequence that worked but has not itself been executed end to end. Only status and the discovery and guard functions have run.
• Loki's lost log window is gone for good. Up to 72h of logs (its retention_period), destroyed by my scale-down. Nothing recovers it; Velero's schedules cover the namespace but the PVC was deleted, not corrupted, and I did not attempt a restore because the data self-expires anyway.
• No other whenScaled: Delete StatefulSet was audited. The guard now catches them generically, but I did not enumerate which other workloads carry the policy, so I cannot say loki was the only one at risk. The runbook gives the one-liner to list them.
• TrueNAS and the router are entirely outside this change. TrueNAS is not in the Ansible inventory and refuses SSH, so its shutdown stays a web-UI step with no automation and no check.
• No runtime conformance. vars.CLUSTER_AVAILABLE is still unset, so CI proves the L0/L1 offline subset only.
• I did not re-measure branch protection or runner state (CLAUDE.md §2b).