--- date: 2026-10-17 branch: feat/graceful-cluster-shutdown title: "Graceful node shutdown was never configured, and scaling a StatefulSet deleted its PVC" --- ## What changed All three nodes had to be powered off to move them onto a UPS. There was **no procedure for it**: `infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev cluster, so the whole sequence was improvised live. Two defects came out of doing it. **1. Kubelet graceful node shutdown was unconfigured on every prod node.** No `shutdownGracePeriod` anywhere in `infra/ansible/`, so `shutdown -h` stopped k3s, containerd went with it, and every pod was SIGKILLed where it stood — 9 CloudNativePG instances mid-checkpoint, the UniFi VM, and 25 attached Longhorn volumes. `roles/k3s_server` now writes `kubelet-arg` into `/etc/rancher/k3s/config.yaml` with `shutdown-grace-period=90s` and `shutdown-grace-period-critical-pods=20s`. The half that is easy to miss and silently defeats the other half: **graceful node shutdown is enforced by logind, not by the kubelet.** The kubelet holds an inhibitor *delay* lock and systemd waits at most `InhibitDelayMaxSec` — **default 5 seconds** — before proceeding regardless. A grace period above that cap is truncated with no error, so the feature looks configured and still hard-kills pods. The role now ships a `logind.conf.d` drop-in setting it to 120, and **asserts** `k3s_logind_inhibit_delay_max_sec >= k3s_kubelet_shutdown_grace_period` rather than documenting the relationship, because the failure is invisible. **2. Scaling a StatefulSet to 0 DELETED its PVC and Longhorn volume.** `monitoring/loki` carries `persistentVolumeClaimRetentionPolicy: {whenDeleted: Delete, whenScaled: Delete}`, so the quiesce's `scale --replicas=0` destroyed `storage-loki-0` outright. Loki stores with `type: filesystem`, so that PVC was the only copy of its log data. This is the defect in this change set that actually destroyed something, and it was mine. New `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`) encodes the sequence that worked, and **refuses to scale any StatefulSet whose `whenScaled` is `Delete`** — it leaves it running, accepts that its volume hard-detaches at poweroff, and prints which volumes to expect `degraded`. A rebuild is a vastly better outcome than a deletion. Workload discovery is computed from the live cluster (every Deployment/StatefulSet whose pods mount a PVC, Longhorn or TrueNAS NFS) rather than hardcoded, so a newly onboarded service is covered without anyone remembering to add it. `runbooks/cluster-maintenance-shutdown.md` (new, registered in `runbooks/kustomization.yaml`) is the operator page: shutdown order, the TrueNAS-after-nodes constraint, power-on order, the un-hibernate-Postgres step Flux will not do for you, and a "what looks broken but is fine" section separating the `4/19`-Kustomizations-Ready cascade, the degraded-and-rebuilding volume and the self-recovering `keycloak-operator` from the one thing that genuinely needs a human (a stuck `oauth2-proxy` replica). ## What I verified * **Graceful shutdown was genuinely absent**, not merely undocumented: `grep -rn 'shutdownGracePeriod|graceful-node-shutdown'` over `infra/ansible/` returns nothing, and no `logind` or `InhibitDelay` setting exists anywhere in the tree. * **The whole shutdown/restart cycle was executed end to end on prod 2026-10-17** and the cluster came back: 3/3 nodes Ready with etcd `healthz` ok, 19/19 Flux Kustomizations Ready, `platform-config`'s `lastAppliedRevision` equal to the GitRepository artifact revision, all 3 CNPG clusters `healthy` with primaries elected, the UniFi VM `Running`, PiHole resolving (`home.keithjackson.dev` → `192.168.2.20` via `192.168.2.23`), and the Gateway serving `200/200/401` for apex/gitea/plex. * **The PVC-deletion mechanism is confirmed, not inferred:** `kubectl get sts loki -o jsonpath={.spec.persistentVolumeClaimRetentionPolicy}` returns `{"whenDeleted":"Delete","whenScaled":"Delete"}`, the PVC was `NotFound` after the cycle, and the Longhorn volume count had gone 25 → 24. * **The damage was repaired and verified:** `scale sts loki --replicas=1` (restoring the replica count the chart declares) recreated `storage-loki-0` as a fresh 50Gi Bound PVC, `loki-0` reached `2/2 Running`, and the volume count returned to 25. * **The guard works against the live cluster**, tested both ways: `sts_deletes_pvc_on_scale monitoring loki` returns true (would be skipped) and `pihole pihole` returns false (safe to scale). * **The script's read-only paths run clean on prod:** `status` reports correctly, and the discovery function returns the 13 workloads currently holding a PVC. * `bash -n` clean; the two edited Ansible YAML files parse. ## What I did NOT verify * **The kubelet change has NEVER BEEN APPLIED OR TESTED.** This is the significant gap. I did not run the role, did not restart k3s, and have **not** observed a node actually drain on `shutdown -h`. Everything about it is reasoning from kubelet/systemd semantics plus the measured absence of the setting. Applying it needs a **rolling** k3s restart (`serial: 1` — kube-vip keeps the API up, etcd holds at 2/3), which is an operator action (CLAUDE.md §5). **Verify by draining one node for real:** apply, then `shutdown -h now` on a single node and watch whether pods terminate gracefully instead of vanishing. * **Whether 90s/20s are the right numbers.** Chosen to comfortably cover a Postgres fast shutdown and a Longhorn flush; not measured against either. The budget is spent on every reboot, so if a node takes noticeably longer to go down, this is why. * **Whether k3s accepts `kubelet-arg` in `config.yaml` as written.** The file already carries `kube-apiserver-arg` as a list and `kubelet-arg` is the documented sibling, but I did not render or apply it, so a typo would surface only at k3s restart — and a bad `config.yaml` is exactly the kind of thing that makes k3s fail to come up. **Review this one before applying.** * **The `assert` task's `regex_replace` on the grace period** assumes a plain `90s`-style value; a value expressed as `1m30s` would parse to the nonsense integer `130`. Fine for the committed defaults, wrong if someone writes a compound duration. * **`resume`'s Flux-reconcile wait has never run**, and neither has `quiesce` as a script — the 2026-10-17 cycle was done with the equivalent commands by hand, so the script encodes a sequence that worked but has not itself been executed end to end. Only `status` and the discovery and guard functions have run. * **Loki's lost log window is gone for good.** Up to 72h of logs (its `retention_period`), destroyed by my scale-down. Nothing recovers it; Velero's schedules cover the namespace but the PVC was deleted, not corrupted, and I did not attempt a restore because the data self-expires anyway. * **No other `whenScaled: Delete` StatefulSet was audited.** The guard now catches them generically, but I did not enumerate which other workloads carry the policy, so I cannot say loki was the only one at risk. The runbook gives the one-liner to list them. * **TrueNAS and the router are entirely outside this change.** TrueNAS is not in the Ansible inventory and refuses SSH, so its shutdown stays a web-UI step with no automation and no check. * **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1 offline subset only. * I did not re-measure branch protection or runner state (CLAUDE.md §2b).