feat(shutdown): configure graceful node shutdown, and never scale a Delete-policy StatefulSet #71

2026-10-18T04:16:10Z

gitea-admin commented

2026-10-18 04:16:10 +00:00

Context

All three nodes had to be powered off to move them onto a UPS, and there was no procedure for it: infra/cluster.sh stop is virsh-based and only ever applied to the decommissioned dev cluster. The whole sequence was improvised live. Two defects came out of doing it.

1. Graceful node shutdown was unconfigured on every prod node

No shutdownGracePeriod anywhere in infra/ansible/, so shutdown -h stopped k3s, containerd went with it, and every pod was SIGKILLed where it stood — 9 CloudNativePG instances mid-checkpoint, the UniFi VM, and 25 attached Longhorn volumes.

roles/k3s_server now writes kubelet-arg into /etc/rancher/k3s/config.yaml: shutdown-grace-period=90s, shutdown-grace-period-critical-pods=20s.

The half that is easy to miss and silently defeats the other half: this is enforced by logind, not the kubelet. The kubelet holds an inhibitor delay lock, and systemd waits at most InhibitDelayMaxSec — default 5 seconds — before proceeding regardless. A grace period above that cap is truncated with no error, so the feature looks configured and still hard-kills pods. The role ships a logind.conf.d drop-in (120) and asserts cap >= grace period rather than documenting the relationship, because the failure is invisible.

2. Scaling a StatefulSet to 0 DELETED its PVC and Longhorn volume

monitoring/loki carries persistentVolumeClaimRetentionPolicy: {whenDeleted: Delete, whenScaled: Delete}, so the quiesce's scale --replicas=0 destroyed storage-loki-0 outright. Loki stores with type: filesystem, so that PVC was the only copy of its log data. Its 72h retention bounded the loss; nothing about the mechanism is bounded.

The only visible symptom was the Longhorn volume count going 25 → 24, which was first misread as a miscount. Helm never noticed — the HelmRelease still reported InstallSucceeded. Repaired by scaling back to the chart's declared 1, which recreated the PVC; loki-0 returned to 2/2 Running and the count to 25.

What this adds

infra/maintenance/quiesce.sh (quiesce | resume | status) encodes the sequence that worked, and refuses to scale any StatefulSet whose whenScaled is Delete — it leaves it running, accepts that its volume hard-detaches at poweroff, and prints which volumes to expect degraded. A rebuild is a vastly better outcome than a deletion. Workload discovery is computed from the live cluster (every Deployment/StatefulSet whose pods mount a PVC, Longhorn or TrueNAS NFS) rather than hardcoded, so a newly onboarded service is covered without anyone remembering to add it here.

It deliberately does not power anything off — that needs the become password and is the operator's action.

runbooks/cluster-maintenance-shutdown.md is the operator page: order in both directions, the TrueNAS-after-nodes constraint (a hard NFS mount on a live node wedges kubelet), the un-hibernate-Postgres step Flux will not do for you (hibernation is imperative state that survives a Flux resume, so resuming Flux alone leaves all three databases down looking broken for reasons that point nowhere near Postgres), and a "looks broken but is fine" section separating the 4/19-Kustomizations-Ready cascade, the degraded-and-rebuilding volume and the self-recovering keycloak-operator from the one thing that genuinely needs a human.

Verification

The whole shutdown/restart cycle was executed end to end on prod 2026-10-17 and the cluster came back: 3/3 nodes Ready with etcd healthz ok, 19/19 Flux Kustomizations Ready, platform-config's lastAppliedRevision equal to the GitRepository artifact, all 3 CNPG clusters healthy with primaries elected, UniFi VM Running, PiHole resolving, Gateway serving 200/200/401.

The PVC-deletion mechanism is confirmed, not inferred (jsonpath on the retention policy, PVC NotFound, volume count 25→24), and the repair verified. The guard was tested both ways against the live cluster: fires for loki, correctly does not for pihole. The script's status and discovery paths run clean on prod.

Not verified — please read before applying

The kubelet change has never been applied or tested. I did not run the role, did not restart k3s, and have not seen a node actually drain. It is reasoned from kubelet/systemd semantics plus the measured absence of the setting. Applying needs a rolling k3s restart (serial: 1). Review config.yaml's kubelet-arg before applying — a bad config.yaml is how k3s fails to come up. Verify by draining one node for real.

Also unverified: whether 90s/20s are the right numbers (not measured against a Postgres shutdown or a Longhorn flush); the assert's regex_replace assumes a plain 90s-style value and would mis-parse 1m30s; quiesce/resume have never run as a script (the cycle was done with the equivalent commands by hand — only status, discovery and the guard have executed); no other whenScaled: Delete StatefulSet was audited; and Loki's lost log window is gone for good.

Local gate: rc 3, zero failures — only gitea-settings (CI-only token) and L2/L3 conformance (CLUSTER_AVAILABLE unset) skipped, both structurally impossible locally per CLAUDE.md §3.

Full detail: decisions/2026-10-17-graceful-cluster-shutdown.md

🤖 Generated with Claude Code

## Context All three nodes had to be powered off to move them onto a UPS, and there was **no procedure for it**: `infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev cluster. The whole sequence was improvised live. Two defects came out of doing it. ## 1. Graceful node shutdown was unconfigured on every prod node No `shutdownGracePeriod` anywhere in `infra/ansible/`, so `shutdown -h` stopped k3s, containerd went with it, and **every pod was SIGKILLed where it stood** — 9 CloudNativePG instances mid-checkpoint, the UniFi VM, and 25 attached Longhorn volumes. `roles/k3s_server` now writes `kubelet-arg` into `/etc/rancher/k3s/config.yaml`: `shutdown-grace-period=90s`, `shutdown-grace-period-critical-pods=20s`. **The half that is easy to miss and silently defeats the other half:** this is enforced by **logind, not the kubelet**. The kubelet holds an inhibitor *delay* lock, and systemd waits at most `InhibitDelayMaxSec` — **default 5 seconds** — before proceeding regardless. A grace period above that cap is truncated with no error, so the feature looks configured and still hard-kills pods. The role ships a `logind.conf.d` drop-in (120) and **asserts** `cap >= grace period` rather than documenting the relationship, because the failure is invisible. ## 2. Scaling a StatefulSet to 0 DELETED its PVC and Longhorn volume `monitoring/loki` carries `persistentVolumeClaimRetentionPolicy: {whenDeleted: Delete, whenScaled: Delete}`, so the quiesce's `scale --replicas=0` destroyed `storage-loki-0` outright. Loki stores with `type: filesystem`, so that PVC was the only copy of its log data. Its 72h retention bounded the loss; **nothing about the mechanism is bounded.** The only visible symptom was the Longhorn volume count going 25 → 24, which was first misread as a miscount. Helm never noticed — the HelmRelease still reported `InstallSucceeded`. Repaired by scaling back to the chart's declared `1`, which recreated the PVC; `loki-0` returned to `2/2 Running` and the count to 25. ## What this adds **`infra/maintenance/quiesce.sh`** (`quiesce` | `resume` | `status`) encodes the sequence that worked, and **refuses to scale any StatefulSet whose `whenScaled` is `Delete`** — it leaves it running, accepts that its volume hard-detaches at poweroff, and prints which volumes to expect `degraded`. A rebuild is a vastly better outcome than a deletion. Workload discovery is computed from the live cluster (every Deployment/StatefulSet whose pods mount a PVC, Longhorn or TrueNAS NFS) rather than hardcoded, so a newly onboarded service is covered without anyone remembering to add it here. It deliberately **does not power anything off** — that needs the become password and is the operator's action. **`runbooks/cluster-maintenance-shutdown.md`** is the operator page: order in both directions, the TrueNAS-after-nodes constraint (a hard NFS mount on a live node wedges kubelet), the un-hibernate-Postgres step **Flux will not do for you** (hibernation is imperative state that survives a Flux resume, so resuming Flux alone leaves all three databases down looking broken for reasons that point nowhere near Postgres), and a "looks broken but is fine" section separating the `4/19`-Kustomizations-Ready cascade, the degraded-and-rebuilding volume and the self-recovering `keycloak-operator` from the one thing that genuinely needs a human. ## Verification The **whole shutdown/restart cycle was executed end to end on prod 2026-10-17** and the cluster came back: 3/3 nodes Ready with etcd `healthz` ok, 19/19 Flux Kustomizations Ready, `platform-config`'s `lastAppliedRevision` equal to the GitRepository artifact, all 3 CNPG clusters healthy with primaries elected, UniFi VM Running, PiHole resolving, Gateway serving `200/200/401`. The PVC-deletion mechanism is confirmed, not inferred (`jsonpath` on the retention policy, PVC `NotFound`, volume count 25→24), and the repair verified. The guard was tested **both ways** against the live cluster: fires for `loki`, correctly does not for `pihole`. The script's `status` and discovery paths run clean on prod. ## Not verified — please read before applying **The kubelet change has never been applied or tested.** I did not run the role, did not restart k3s, and have not seen a node actually drain. It is reasoned from kubelet/systemd semantics plus the measured absence of the setting. Applying needs a **rolling** k3s restart (`serial: 1`). **Review `config.yaml`'s `kubelet-arg` before applying — a bad `config.yaml` is how k3s fails to come up.** Verify by draining one node for real. Also unverified: whether 90s/20s are the right numbers (not measured against a Postgres shutdown or a Longhorn flush); the `assert`'s `regex_replace` assumes a plain `90s`-style value and would mis-parse `1m30s`; `quiesce`/`resume` have never run as a script (the cycle was done with the equivalent commands by hand — only `status`, discovery and the guard have executed); no other `whenScaled: Delete` StatefulSet was audited; and **Loki's lost log window is gone for good.** Local gate: **rc 3, zero failures** — only `gitea-settings` (CI-only token) and L2/L3 `conformance` (`CLUSTER_AVAILABLE` unset) skipped, both structurally impossible locally per CLAUDE.md §3. Full detail: `decisions/2026-10-17-graceful-cluster-shutdown.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-18 04:16:10 +00:00

feat(shutdown): configure graceful node shutdown, and never scale a Delete-policy StatefulSet

gate / gate (pull_request) Successful in 27m44s

Details

gate / conformance (pull_request) Has been skipped

Details

2f2e98aa19

All three nodes had to be powered off to move them onto a UPS, and there was
NO procedure: infra/cluster.sh stop is virsh-based and only ever applied to the
decommissioned dev cluster. The sequence was improvised live. Two defects came
out of doing it.

1. GRACEFUL NODE SHUTDOWN WAS UNCONFIGURED ON EVERY PROD NODE. No
   shutdownGracePeriod anywhere in infra/ansible/, so `shutdown -h` stopped k3s,
   containerd went with it, and every pod was SIGKILLed where it stood -- 9 CNPG
   instances mid-checkpoint, the UniFi VM, 25 attached Longhorn volumes.
   roles/k3s_server now sets kubelet-arg shutdown-grace-period=90s and
   shutdown-grace-period-critical-pods=20s.

   The half that silently defeats the other half: this is enforced by LOGIND,
   not the kubelet. The kubelet holds an inhibitor DELAY lock and systemd waits
   at most InhibitDelayMaxSec -- DEFAULT 5 SECONDS -- then proceeds regardless,
   truncating the drain with no error. The role ships a logind.conf.d drop-in
   (120) and ASSERTS cap >= grace period, because the failure is invisible.

2. SCALING A StatefulSet TO 0 DELETED ITS PVC AND VOLUME. monitoring/loki
   carries persistentVolumeClaimRetentionPolicy whenScaled: Delete, so the
   quiesce destroyed storage-loki-0 outright. Loki stores type: filesystem, so
   that PVC was the only copy; 72h retention bounded the loss, the mechanism
   does not. Symptom was the volume count going 25 -> 24, first misread as a
   miscount. Repaired: scale back to the declared 1 recreated the PVC and
   loki-0 is 2/2 Running.

New infra/maintenance/quiesce.sh (quiesce|resume|status) encodes the sequence
that worked and REFUSES to scale any whenScaled: Delete StatefulSet -- it stays
running, its volume hard-detaches, and it rebuilds on restart. A rebuild beats
a deletion. Discovery is computed from the live cluster, so a newly onboarded
service needs no edit here.

runbooks/cluster-maintenance-shutdown.md is the operator page: order in both
directions, TrueNAS-after-nodes (hard NFS mount wedges kubelet), the
un-hibernate-Postgres step Flux will NOT do for you, and a "looks broken but is
fine" section separating the 4/19 Flux cascade, the degraded-and-rebuilding
volume and the self-recovering keycloak-operator from the one thing that needs
a human (a stuck oauth2-proxy replica).

THE KUBELET CHANGE HAS NEVER BEEN APPLIED OR TESTED -- it needs a rolling k3s
restart, which is an operator action. Review config.yaml's kubelet-arg before
applying: a bad config.yaml is how k3s fails to come up. See the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 merged commit 83d3568150 into main

2026-10-18 05:51:20 +00:00

jakman85 referenced this issue from a commit

2026-10-18 05:51:21 +00:00

Merge pull request 'feat(shutdown): configure graceful node shutdown, and never scale a Delete-policy StatefulSet' (#71) from feat/graceful-cluster-shutdown into main

gitea-admin referenced this pull request

2026-10-18 07:40:25 +00:00

fix(conformance): one regex made the merge gate take 28 minutes instead of 2 #76

gitea-admin referenced this issue from a commit

2026-10-18 07:40:25 +00:00

fix(conformance): one regex made the merge gate take 28 minutes instead of 2

Sign in to join this conversation.