From 2f2e98aa19db712a22696a61591fa38885e1d43b Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 17 Oct 2026 22:26:47 -0500 Subject: [PATCH] feat(shutdown): configure graceful node shutdown, and never scale a Delete-policy StatefulSet All three nodes had to be powered off to move them onto a UPS, and there was NO procedure: infra/cluster.sh stop is virsh-based and only ever applied to the decommissioned dev cluster. The sequence was improvised live. Two defects came out of doing it. 1. GRACEFUL NODE SHUTDOWN WAS UNCONFIGURED ON EVERY PROD NODE. No shutdownGracePeriod anywhere in infra/ansible/, so `shutdown -h` stopped k3s, containerd went with it, and every pod was SIGKILLed where it stood -- 9 CNPG instances mid-checkpoint, the UniFi VM, 25 attached Longhorn volumes. roles/k3s_server now sets kubelet-arg shutdown-grace-period=90s and shutdown-grace-period-critical-pods=20s. The half that silently defeats the other half: this is enforced by LOGIND, not the kubelet. The kubelet holds an inhibitor DELAY lock and systemd waits at most InhibitDelayMaxSec -- DEFAULT 5 SECONDS -- then proceeds regardless, truncating the drain with no error. The role ships a logind.conf.d drop-in (120) and ASSERTS cap >= grace period, because the failure is invisible. 2. SCALING A StatefulSet TO 0 DELETED ITS PVC AND VOLUME. monitoring/loki carries persistentVolumeClaimRetentionPolicy whenScaled: Delete, so the quiesce destroyed storage-loki-0 outright. Loki stores type: filesystem, so that PVC was the only copy; 72h retention bounded the loss, the mechanism does not. Symptom was the volume count going 25 -> 24, first misread as a miscount. Repaired: scale back to the declared 1 recreated the PVC and loki-0 is 2/2 Running. New infra/maintenance/quiesce.sh (quiesce|resume|status) encodes the sequence that worked and REFUSES to scale any whenScaled: Delete StatefulSet -- it stays running, its volume hard-detaches, and it rebuilds on restart. A rebuild beats a deletion. Discovery is computed from the live cluster, so a newly onboarded service needs no edit here. runbooks/cluster-maintenance-shutdown.md is the operator page: order in both directions, TrueNAS-after-nodes (hard NFS mount wedges kubelet), the un-hibernate-Postgres step Flux will NOT do for you, and a "looks broken but is fine" section separating the 4/19 Flux cascade, the degraded-and-rebuilding volume and the self-recovering keycloak-operator from the one thing that needs a human (a stuck oauth2-proxy replica). THE KUBELET CHANGE HAS NEVER BEEN APPLIED OR TESTED -- it needs a rolling k3s restart, which is an operator action. Review config.yaml's kubelet-arg before applying: a bad config.yaml is how k3s fails to come up. See the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-10-17-graceful-cluster-shutdown.md | 106 +++++++ .../roles/k3s_server/defaults/main.yml | 28 ++ .../roles/k3s_server/handlers/main.yml | 9 + infra/ansible/roles/k3s_server/tasks/main.yml | 53 ++++ infra/maintenance/quiesce.sh | 265 ++++++++++++++++++ runbooks/cluster-maintenance-shutdown.md | 162 +++++++++++ runbooks/kustomization.yaml | 7 + 7 files changed, 630 insertions(+) create mode 100644 decisions/2026-10-17-graceful-cluster-shutdown.md create mode 100755 infra/maintenance/quiesce.sh create mode 100644 runbooks/cluster-maintenance-shutdown.md diff --git a/decisions/2026-10-17-graceful-cluster-shutdown.md b/decisions/2026-10-17-graceful-cluster-shutdown.md new file mode 100644 index 0000000..ff43169 --- /dev/null +++ b/decisions/2026-10-17-graceful-cluster-shutdown.md @@ -0,0 +1,106 @@ +--- +date: 2026-10-17 +branch: feat/graceful-cluster-shutdown +title: "Graceful node shutdown was never configured, and scaling a StatefulSet deleted its PVC" +--- + +## What changed + +All three nodes had to be powered off to move them onto a UPS. There was **no procedure for it**: +`infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev cluster, so +the whole sequence was improvised live. Two defects came out of doing it. + +**1. Kubelet graceful node shutdown was unconfigured on every prod node.** No `shutdownGracePeriod` +anywhere in `infra/ansible/`, so `shutdown -h` stopped k3s, containerd went with it, and every pod +was SIGKILLed where it stood — 9 CloudNativePG instances mid-checkpoint, the UniFi VM, and 25 +attached Longhorn volumes. `roles/k3s_server` now writes `kubelet-arg` into +`/etc/rancher/k3s/config.yaml` with `shutdown-grace-period=90s` and +`shutdown-grace-period-critical-pods=20s`. + +The half that is easy to miss and silently defeats the other half: **graceful node shutdown is +enforced by logind, not by the kubelet.** The kubelet holds an inhibitor *delay* lock and systemd +waits at most `InhibitDelayMaxSec` — **default 5 seconds** — before proceeding regardless. A grace +period above that cap is truncated with no error, so the feature looks configured and still +hard-kills pods. The role now ships a `logind.conf.d` drop-in setting it to 120, and **asserts** +`k3s_logind_inhibit_delay_max_sec >= k3s_kubelet_shutdown_grace_period` rather than documenting the +relationship, because the failure is invisible. + +**2. Scaling a StatefulSet to 0 DELETED its PVC and Longhorn volume.** `monitoring/loki` carries +`persistentVolumeClaimRetentionPolicy: {whenDeleted: Delete, whenScaled: Delete}`, so the quiesce's +`scale --replicas=0` destroyed `storage-loki-0` outright. Loki stores with `type: filesystem`, so +that PVC was the only copy of its log data. This is the defect in this change set that actually +destroyed something, and it was mine. + +New `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`) encodes the sequence that +worked, and **refuses to scale any StatefulSet whose `whenScaled` is `Delete`** — it leaves it +running, accepts that its volume hard-detaches at poweroff, and prints which volumes to expect +`degraded`. A rebuild is a vastly better outcome than a deletion. Workload discovery is computed +from the live cluster (every Deployment/StatefulSet whose pods mount a PVC, Longhorn or TrueNAS NFS) +rather than hardcoded, so a newly onboarded service is covered without anyone remembering to add it. + +`runbooks/cluster-maintenance-shutdown.md` (new, registered in `runbooks/kustomization.yaml`) is the +operator page: shutdown order, the TrueNAS-after-nodes constraint, power-on order, the +un-hibernate-Postgres step Flux will not do for you, and a "what looks broken but is fine" section +separating the `4/19`-Kustomizations-Ready cascade, the degraded-and-rebuilding volume and the +self-recovering `keycloak-operator` from the one thing that genuinely needs a human (a stuck +`oauth2-proxy` replica). + +## What I verified + +* **Graceful shutdown was genuinely absent**, not merely undocumented: `grep -rn + 'shutdownGracePeriod|graceful-node-shutdown'` over `infra/ansible/` returns nothing, and no + `logind` or `InhibitDelay` setting exists anywhere in the tree. +* **The whole shutdown/restart cycle was executed end to end on prod 2026-10-17** and the cluster + came back: 3/3 nodes Ready with etcd `healthz` ok, 19/19 Flux Kustomizations Ready, + `platform-config`'s `lastAppliedRevision` equal to the GitRepository artifact revision, all 3 + CNPG clusters `healthy` with primaries elected, the UniFi VM `Running`, PiHole resolving + (`home.keithjackson.dev` → `192.168.2.20` via `192.168.2.23`), and the Gateway serving + `200/200/401` for apex/gitea/plex. +* **The PVC-deletion mechanism is confirmed, not inferred:** `kubectl get sts loki -o + jsonpath={.spec.persistentVolumeClaimRetentionPolicy}` returns + `{"whenDeleted":"Delete","whenScaled":"Delete"}`, the PVC was `NotFound` after the cycle, and the + Longhorn volume count had gone 25 → 24. +* **The damage was repaired and verified:** `scale sts loki --replicas=1` (restoring the replica + count the chart declares) recreated `storage-loki-0` as a fresh 50Gi Bound PVC, `loki-0` reached + `2/2 Running`, and the volume count returned to 25. +* **The guard works against the live cluster**, tested both ways: `sts_deletes_pvc_on_scale + monitoring loki` returns true (would be skipped) and `pihole pihole` returns false (safe to + scale). +* **The script's read-only paths run clean on prod:** `status` reports correctly, and the discovery + function returns the 13 workloads currently holding a PVC. +* `bash -n` clean; the two edited Ansible YAML files parse. + +## What I did NOT verify + +* **The kubelet change has NEVER BEEN APPLIED OR TESTED.** This is the significant gap. I did not + run the role, did not restart k3s, and have **not** observed a node actually drain on + `shutdown -h`. Everything about it is reasoning from kubelet/systemd semantics plus the measured + absence of the setting. Applying it needs a **rolling** k3s restart (`serial: 1` — kube-vip keeps + the API up, etcd holds at 2/3), which is an operator action (CLAUDE.md §5). **Verify by draining + one node for real:** apply, then `shutdown -h now` on a single node and watch whether pods + terminate gracefully instead of vanishing. +* **Whether 90s/20s are the right numbers.** Chosen to comfortably cover a Postgres fast shutdown + and a Longhorn flush; not measured against either. The budget is spent on every reboot, so if a + node takes noticeably longer to go down, this is why. +* **Whether k3s accepts `kubelet-arg` in `config.yaml` as written.** The file already carries + `kube-apiserver-arg` as a list and `kubelet-arg` is the documented sibling, but I did not render + or apply it, so a typo would surface only at k3s restart — and a bad `config.yaml` is exactly the + kind of thing that makes k3s fail to come up. **Review this one before applying.** +* **The `assert` task's `regex_replace` on the grace period** assumes a plain `90s`-style value; a + value expressed as `1m30s` would parse to the nonsense integer `130`. Fine for the committed + defaults, wrong if someone writes a compound duration. +* **`resume`'s Flux-reconcile wait has never run**, and neither has `quiesce` as a script — the + 2026-10-17 cycle was done with the equivalent commands by hand, so the script encodes a sequence + that worked but has not itself been executed end to end. Only `status` and the discovery and guard + functions have run. +* **Loki's lost log window is gone for good.** Up to 72h of logs (its `retention_period`), destroyed + by my scale-down. Nothing recovers it; Velero's schedules cover the namespace but the PVC was + deleted, not corrupted, and I did not attempt a restore because the data self-expires anyway. +* **No other `whenScaled: Delete` StatefulSet was audited.** The guard now catches them generically, + but I did not enumerate which other workloads carry the policy, so I cannot say loki was the only + one at risk. The runbook gives the one-liner to list them. +* **TrueNAS and the router are entirely outside this change.** TrueNAS is not in the Ansible + inventory and refuses SSH, so its shutdown stays a web-UI step with no automation and no check. +* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1 + offline subset only. +* I did not re-measure branch protection or runner state (CLAUDE.md §2b). diff --git a/infra/ansible/roles/k3s_server/defaults/main.yml b/infra/ansible/roles/k3s_server/defaults/main.yml index 31e950f..60f9bc7 100644 --- a/infra/ansible/roles/k3s_server/defaults/main.yml +++ b/infra/ansible/roles/k3s_server/defaults/main.yml @@ -107,3 +107,31 @@ k3s_base_domain: "home.keithjackson.dev" # Derived, so the realm path is stated once. Override k3s_base_domain, not this. k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab" + +# --- Graceful node shutdown (kubelet) ---------------------------------------------------------- +# MEASURED 2026-10-17, during the UPS move: graceful node shutdown was NOT configured on any prod +# node, so `shutdown -h` did NOT drain pods. systemd stops k3s, containerd goes with it, and every +# pod is SIGKILLed where it stands -- including 9 CloudNativePG instances mid-checkpoint, the UniFi +# VM, and 25 attached Longhorn volumes. The whole-cluster shutdown that day had to quiesce +# everything BY HAND first (runbooks/cluster-maintenance-shutdown.md), and the one volume that did +# not finish detaching came back `degraded` and had to rebuild. +# +# With these set, the kubelet takes a systemd inhibitor lock and, on shutdown, evicts ordinary pods +# first and critical ones last, so a plain `shutdown -h`/reboot drains the node. That makes the +# manual scale-down in the runbook a belt-and-braces step for a PLANNED whole-cluster shutdown +# rather than the only thing standing between a reboot and a hard kill. +# +# 90s/20s, not larger: the budget is spent on every reboot and node upgrade, and the kubelet waits +# the FULL ordinary period before it starts on critical pods. Postgres does a fast shutdown in +# well under 90s here, and Longhorn needs only enough time to flush and detach. +k3s_kubelet_shutdown_grace_period: "90s" +k3s_kubelet_shutdown_grace_period_critical_pods: "20s" + +# *** THE HALF THAT IS EASY TO MISS, AND SILENTLY BREAKS THE OTHER HALF. *** +# Graceful node shutdown works by the kubelet holding a logind INHIBITOR DELAY lock. systemd caps +# how long any inhibitor may delay shutdown at `InhibitDelayMaxSec`, which DEFAULTS TO 5 SECONDS. +# If that cap is below the grace period above, systemd proceeds anyway and the drain is truncated +# to ~5s -- i.e. the feature appears configured, reports no error, and still hard-kills pods. So +# this MUST be >= k3s_kubelet_shutdown_grace_period; the role asserts it rather than trusting it. +# Set a little above the kubelet's total (90s + 20s) to leave headroom. +k3s_logind_inhibit_delay_max_sec: 120 diff --git a/infra/ansible/roles/k3s_server/handlers/main.yml b/infra/ansible/roles/k3s_server/handlers/main.yml index 2070f87..66acb0c 100644 --- a/infra/ansible/roles/k3s_server/handlers/main.yml +++ b/infra/ansible/roles/k3s_server/handlers/main.yml @@ -15,3 +15,12 @@ port: "{{ k3s_api_port }}" timeout: 180 listen: Restart k3s + +# logind re-reads InhibitDelayMaxSec only on restart. Restarting logind does NOT log users out or +# disturb k3s -- it manages login sessions and inhibitor locks, not workloads. The kubelet picks up +# the new cap when it next takes its lock, so no k3s restart is needed for THIS half. +- name: Restart logind + ansible.builtin.systemd_service: + name: systemd-logind + state: restarted + listen: Restart logind diff --git a/infra/ansible/roles/k3s_server/tasks/main.yml b/infra/ansible/roles/k3s_server/tasks/main.yml index 8cfbee1..cf60d16 100644 --- a/infra/ansible/roles/k3s_server/tasks/main.yml +++ b/infra/ansible/roles/k3s_server/tasks/main.yml @@ -86,6 +86,16 @@ # k3s_base_domain (defaults/main.yml), whose [parity] contract with the cluster's # BASE_DOMAIN is documented there. Dev's default renders the value it already runs, so # config.yaml is unchanged and the Restart handler does not fire. + # ── Graceful node shutdown ────────────────────────────────────────────────────────────── + # Without these the kubelet takes no systemd inhibitor lock, so `shutdown -h` stops + # containerd and SIGKILLs every pod where it stands -- MEASURED on all three prod nodes + # 2026-10-17 (defaults/main.yml has the detail). The companion logind drop-in below raises + # InhibitDelayMaxSec, WITHOUT WHICH systemd caps the drain at ~5s and these values are + # quietly ignored. + kubelet-arg: + - "shutdown-grace-period={{ k3s_kubelet_shutdown_grace_period }}" + - "shutdown-grace-period-critical-pods={{ k3s_kubelet_shutdown_grace_period_critical_pods }}" + kube-apiserver-arg: - "oidc-issuer-url={{ k3s_oidc_issuer_url }}" - "oidc-client-id=kubernetes" @@ -183,3 +193,46 @@ host: "127.0.0.1" port: "{{ k3s_api_port }}" timeout: 180 + +# ── Graceful node shutdown: the systemd half — tag `shutdown` ────────────────────────────────── +# The kubelet's shutdown-grace-period is ENFORCED BY logind, not by the kubelet: the kubelet holds +# an inhibitor DELAY lock, and systemd will wait at most `InhibitDelayMaxSec` for it (default 5s) +# before proceeding regardless. A grace period above that cap is therefore silently truncated -- +# the feature looks configured, logs no error, and still hard-kills pods. Assert the relationship +# instead of documenting it, because the failure is invisible. +- name: Assert the logind inhibitor cap can actually cover the kubelet grace period + ansible.builtin.assert: + that: + - (k3s_logind_inhibit_delay_max_sec | int) >= (k3s_kubelet_shutdown_grace_period | regex_replace('[^0-9]', '') | int) + fail_msg: >- + k3s_logind_inhibit_delay_max_sec ({{ k3s_logind_inhibit_delay_max_sec }}s) is below + k3s_kubelet_shutdown_grace_period ({{ k3s_kubelet_shutdown_grace_period }}). systemd would + stop waiting for the kubelet's inhibitor lock before the drain finished, so pods would be + SIGKILLed anyway and graceful node shutdown would be configured but ineffective. + quiet: true + tags: [shutdown] + +- name: Ensure /etc/systemd/logind.conf.d exists + ansible.builtin.file: + path: /etc/systemd/logind.conf.d + state: directory + owner: root + group: root + mode: "0755" + tags: [shutdown] + +# A drop-in, not an edit of logind.conf: the distro file stays pristine and `state: absent` here +# would fully revert the change. +- name: Raise the logind inhibitor delay cap so the kubelet can finish draining + ansible.builtin.copy: + dest: /etc/systemd/logind.conf.d/10-k3s-graceful-shutdown.conf + owner: root + group: root + mode: "0644" + content: | + # Managed by Ansible (roles/k3s_server). See defaults/main.yml. + # MUST stay >= kubelet shutdown-grace-period, or the drain is truncated to this value. + [Login] + InhibitDelayMaxSec={{ k3s_logind_inhibit_delay_max_sec }} + notify: Restart logind + tags: [shutdown] diff --git a/infra/maintenance/quiesce.sh b/infra/maintenance/quiesce.sh new file mode 100755 index 0000000..8577fb7 --- /dev/null +++ b/infra/maintenance/quiesce.sh @@ -0,0 +1,265 @@ +#!/usr/bin/env bash +# ================================================================================================= +# quiesce.sh — bring the prod cluster to a safe standstill before powering nodes off, and bring it +# back afterwards. Companion to runbooks/cluster-maintenance-shutdown.md. +# ================================================================================================= +# WHY THIS EXISTS. On 2026-10-17 all three nodes had to be powered off to move them onto a UPS, and +# there was NO procedure for it: `infra/cluster.sh stop` is `virsh`-based and only ever applied to +# the decommissioned dev cluster. The whole sequence was improvised live. This script is that +# sequence, so the next person does not improvise it. +# +# WHY IT IS NOT JUST `shutdown -h`. Graceful node shutdown was unconfigured on every prod node that +# day, so systemd stopped containerd and SIGKILLed every pod where it stood. roles/k3s_server now +# sets shutdown-grace-period (+ the logind InhibitDelayMaxSec cap that actually enforces it), which +# makes a bare reboot safe. This script is still the right tool for a PLANNED whole-cluster +# shutdown, for three reasons the kubelet's drain does not cover: +# 1. The UniFi VM needs an ACPI guest shutdown, not an eviction. +# 2. CloudNativePG wants hibernation, not pod deletion -- and hibernation is IMPERATIVE state +# that Flux will not set or clear for you. +# 3. Draining three nodes one at a time makes pods RESCHEDULE onto the survivors, churning +# Longhorn replicas for no reason when the whole cluster is going down anyway. Scaling in +# place avoids all of it. +# +# SAFE TO RE-RUN. Every step is idempotent and `resume` is the exact inverse of `quiesce`. +# +# WHAT IT DELIBERATELY DOES NOT DO: power anything off. Node poweroff is an operator action +# (CLAUDE.md §5) and needs the become password; the runbook gives the ansible one-liner. It also +# does not touch TrueNAS (not in the inventory, SSH refused -- web UI only) or the router. +set -euo pipefail + +CONTEXT="${CONTEXT:-k3s-prod}" +KUBECTL=(kubectl --context "$CONTEXT" --request-timeout=30s) +TIMEOUT="${TIMEOUT:-600}" + +# The three CloudNativePG clusters, as namespace/name. Hibernation is per-Cluster. +CNPG=(keycloak/keycloak-db nextcloud-db/nextcloud-db semaphore-db/semaphore-db) + +log() { printf '\033[34m[quiesce]\033[0m %s\n' "$*"; } +ok() { printf '\033[32m[ ok ]\033[0m %s\n' "$*"; } +warn() { printf '\033[33m[warn]\033[0m %s\n' "$*"; } +die() { printf '\033[31m[fail]\033[0m %s\n' "$*" >&2; exit 1; } + +need() { command -v "$1" >/dev/null || die "$1 not on PATH — run this inside ./infra/devcontainer/dev.sh"; } + +# ---------------------------------------------------------------- discovery +# Every Deployment/StatefulSet whose pods mount a PVC — Longhorn OR the TrueNAS NFS bulk tier. +# Computed from the LIVE cluster rather than hardcoded, so a newly onboarded service is covered +# without anyone remembering to add it here. PROTECTED namespaces are excluded because scaling +# them down would remove the very machinery that performs the detach (longhorn-system), keep the +# API reachable (kube-system), or finish the hibernation and the VM shutdown (cnpg-system, +# kubevirt). +pvc_workloads() { + "${KUBECTL[@]}" get pods -A -o json | python3 -c ' +import json,sys +PROTECT={"longhorn-system","kube-system","cilium-secrets","cnpg-system","kubevirt","flux-system","cdi"} +out={} +for p in json.load(sys.stdin)["items"]: + ns=p["metadata"]["namespace"] + if ns in PROTECT: continue + if not any(v.get("persistentVolumeClaim") for v in (p["spec"].get("volumes") or [])): continue + ow=(p["metadata"].get("ownerReferences") or []) + if not ow: continue + k,n=ow[0]["kind"],ow[0]["name"] + if k=="ReplicaSet": k,n="deployment",n.rsplit("-",1)[0] + elif k=="StatefulSet": k="statefulset" + else: continue # Jobs, and CNPG Clusters, are handled separately + out[(ns,k,n)]=1 +for ns,k,n in sorted(out): print(ns,k,n) +' +} + +# ---------------------------------------------------------------- the PVC-deletion guard +# *** DO NOT REMOVE THIS CHECK. IT EXISTS BECAUSE SCALING A StatefulSet DESTROYED DATA. *** +# A StatefulSet may carry `persistentVolumeClaimRetentionPolicy.whenScaled: Delete`, and then +# scaling it to 0 DELETES ITS PVCs -- and with them the Longhorn volumes. MEASURED 2026-10-17: the +# quiesce for the UPS move scaled `monitoring/loki` (whenScaled: Delete) to 0 and its +# `storage-loki-0` PVC and volume were deleted outright. Loki stores with `type: filesystem`, so +# that PVC was the only copy; 72h retention bounded the loss to a rolling window, but nothing about +# the mechanism is bounded -- the same action on a Delete-policy StatefulSet holding real state +# would be unrecoverable. The volume count going 25 -> 24 was the only visible symptom, and it was +# first misread as a miscount. +# +# So: such a StatefulSet is NEVER scaled. It is left RUNNING and its volume hard-detaches at +# poweroff, which costs a replica rebuild on restart (degraded -> healthy, automatic). A rebuild is +# an enormously better outcome than a deletion, and that is the whole trade being made here. +sts_deletes_pvc_on_scale() { # ns name -> 0 if scaling it would delete its PVCs + local policy + policy=$("${KUBECTL[@]}" -n "$1" get statefulset "$2" \ + -o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy.whenScaled}' 2>/dev/null || true) + [ "$policy" = "Delete" ] +} + +wait_for() { # wait_for + local desc="$1" cond="$2" waited=0 + log "waiting: $desc (up to ${TIMEOUT}s)" + until eval "$cond"; do + (( waited += 10 )); (( waited >= TIMEOUT )) && { warn "TIMED OUT waiting: $desc"; return 1; } + sleep 10 + done + ok "$desc" +} + +# ---------------------------------------------------------------- quiesce +do_quiesce() { + need kubectl; need python3 + + # Refuse to start if Longhorn is already unhealthy. Powering off on top of a degraded volume is + # how one replica becomes zero; better to stop here and let it rebuild first. + local unhealthy + unhealthy=$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io \ + -o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true) + [ "$unhealthy" = "0" ] || die "$unhealthy Longhorn volume(s) are not healthy — fix or wait for rebuild BEFORE powering off" + ok "Longhorn: all volumes healthy" + + # 1. Flux first, or it reverts every scale-down below within its reconcile interval. + log "suspending Flux Kustomizations" + local k + for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do + "${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":true}}' >/dev/null + done + ok "all Kustomizations suspended" + + # 2. The VM needs an ACPI shutdown of the GUEST. KubeVirt does that on running:false; a plain + # pod eviction would be a power cut to the VM. + if "${KUBECTL[@]}" -n unifi get vm unifi >/dev/null 2>&1; then + log "ACPI-stopping the UniFi VM" + "${KUBECTL[@]}" -n unifi patch vm unifi --type=merge -p '{"spec":{"running":false}}' >/dev/null + fi + + # 3. Postgres: hibernation is a clean smart shutdown that KEEPS the PVCs. CNPG does one instance + # at a time, primary last-ish, so this is not instant. + log "hibernating CloudNativePG clusters" + local c ns name + for c in "${CNPG[@]}"; do + ns="${c%%/*}"; name="${c##*/}" + "${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \ + cnpg.io/hibernation=on --overwrite >/dev/null 2>&1 || warn "could not hibernate $c" + done + + # 4. Everything holding a PVC, scaled in place. NFS consumers first so nothing holds a hard + # TrueNAS mount; PiHole last because it is the LAN resolver. + log "scaling down PVC-holding workloads" + local line ns2 kind nm deferred=() skipped_delete_policy=() + while read -r ns2 kind nm; do + [ -z "${ns2:-}" ] && continue + if [ "$kind" = "statefulset" ] && sts_deletes_pvc_on_scale "$ns2" "$nm"; then + warn "SKIPPING $ns2/$nm — whenScaled: Delete would DELETE its PVC (see the guard above)." + warn " Left running; its volume hard-detaches at poweroff and rebuilds on restart." + skipped_delete_policy+=("$ns2/$nm") + continue + fi + if [ "$ns2" = "pihole" ]; then deferred+=("$ns2 $kind $nm"); continue; fi + "${KUBECTL[@]}" -n "$ns2" scale "$kind" "$nm" --replicas=0 >/dev/null 2>&1 \ + && printf ' scaled 0 %s/%s %s\n' "$ns2" "$kind" "$nm" \ + || warn "could not scale $ns2/$kind/$nm" + done < <(pvc_workloads) + for line in "${deferred[@]:-}"; do + [ -z "${line:-}" ] && continue + # shellcheck disable=SC2086 + set -- $line + "${KUBECTL[@]}" -n "$1" scale "$2" "$3" --replicas=0 >/dev/null 2>&1 \ + && printf ' scaled 0 %s/%s %s (LAN DNS — expect name resolution to stop)\n' "$1" "$2" "$3" + done + + # 5. The only assertion that matters. An attached volume at poweroff comes back degraded and + # rebuilds; this is what we are buying with all of the above. + wait_for "UniFi VMI gone" \ + '[ "$('"${KUBECTL[*]}"' -n unifi get vmi --no-headers 2>/dev/null | wc -l)" = "0" ]' || true + wait_for "all Postgres instances stopped" \ + '[ "$('"${KUBECTL[*]}"' get pods -A --no-headers 2>/dev/null | grep -cE "(keycloak|nextcloud|semaphore)-db-[0-9]")" = "0" ]' || true + wait_for "all Longhorn volumes detached" \ + '[ "$('"${KUBECTL[*]}"' -n longhorn-system get volumes.longhorn.io -o jsonpath="{range .items[*]}{.status.state}{\"\n\"}{end}" 2>/dev/null | grep -c attached)" = "0" ]' || \ + warn "some volumes are STILL ATTACHED — they will come back degraded and rebuild; acceptable, but note which" + + if [ "${#skipped_delete_policy[@]}" -gt 0 ]; then + echo + warn "NOT scaled down (whenScaled: Delete — scaling them would delete their PVCs):" + printf ' %s\n' "${skipped_delete_policy[@]}" + warn "Expect these volumes to come back 'degraded' and rebuild. That is correct and automatic." + fi + echo + ok "QUIESCED. Safe to power off." + cat <<'NEXT' + + Next, from inside the devcontainer (it holds the become password): + cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \ + -m shell -a 'shutdown -h +1 "maintenance"' + + THEN, in this order and not before: + 1. TrueNAS (192.168.2.30) — web UI only; not in the ansible inventory and SSH is refused. + 2. Router / switch last. +NEXT +} + +# ---------------------------------------------------------------- resume +do_resume() { + need kubectl + + # THE STEP THAT IS ALWAYS FORGOTTEN. Hibernation is imperative state: Flux neither sets nor + # clears it, so resuming Flux alone leaves all three databases down and every app that needs + # one looking broken for reasons that point nowhere near Postgres. + log "un-hibernating CloudNativePG clusters" + local c ns name + for c in "${CNPG[@]}"; do + ns="${c%%/*}"; name="${c##*/}" + "${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \ + cnpg.io/hibernation=off --overwrite >/dev/null 2>&1 || warn "could not un-hibernate $c" + done + + # Flux restores every replica count AND the VM's running:true from git, so nothing here needs to + # undo the scale-downs by hand. + log "resuming Flux Kustomizations" + local k + for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do + "${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":false}}' >/dev/null + done + ok "all Kustomizations resumed" + + wait_for "Flux fully reconciled" \ + '[ "$('"${KUBECTL[*]}"' -n flux-system get kustomizations --no-headers 2>/dev/null | grep -vc True)" = "0" ]' || \ + warn "some Kustomizations are not Ready — the graph serialises behind platform-config, so give it a few minutes before worrying" + + echo + ok "RESUMED." + cat <<'NEXT' + + Known cold-start race, not a fault in this script: oauth2-proxy performs OIDC discovery at + startup and exits 1 on the 503 it gets while Keycloak is still coming up, then sits in + exponential CrashLoopBackOff. If a replica is stuck, delete the pod once: + kubectl --context k3s-prod -n oauth2-proxy delete pod +NEXT +} + +do_status() { + need kubectl + printf 'nodes: %s Ready\n' "$("${KUBECTL[@]}" get nodes --no-headers | grep -c ' Ready')" + printf 'flux: %s/%s Ready\n' \ + "$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | grep -c True)" \ + "$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | wc -l)" + printf 'longhorn: %s attached, %s detached, %s not healthy\n' \ + "$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c attached)" \ + "$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c detached)" \ + "$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true)" + printf 'postgres: %s instance pod(s)\n' \ + "$("${KUBECTL[@]}" get pods -A --no-headers | grep -cE '(keycloak|nextcloud|semaphore)-db-[0-9]' || true)" + printf 'unifi VM: %s VMI\n' "$("${KUBECTL[@]}" -n unifi get vmi --no-headers 2>/dev/null | wc -l)" +} + +case "${1:-}" in + quiesce) do_quiesce ;; + resume) do_resume ;; + status) do_status ;; + *) cat >&2 </dev/null && echo "$h UP" || echo "$h down"; done +``` + +**Then, in this order and not before:** + +| # | Machine | How | +|---|---|---| +| 4 | **TrueNAS** `192.168.2.30` | **Web UI only.** Not in the ansible inventory, and SSH is **refused** (service disabled). `https://192.168.2.30` → power icon → Shut Down. Safe once the nodes are down, because every NFS consumer (`plex-media`, `*-media`, `qbittorrent-downloads`, `nextcloud-legacy-archive`) is already scaled to zero and nothing holds a hard mount. | +| 5 | **Router / switch** | Last. | + +TrueNAS must go **after** the k3s nodes: a hard NFS mount on a live node wedges kubelet in +D-state (`/runbooks/nfs-export-latency`, and the load-average symptom in the 2026-10-?? note). + +## Bringing it back + +**Power on in the reverse order: router/switch → TrueNAS → all three nodes together.** etcd needs +2 of 3 for quorum, so start the nodes as a group rather than one at a time. + +```bash +# 1. Nodes Ready, etcd quorum, storage reachable. +./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' +./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get --raw=/healthz/etcd' + +# 2. Resume: un-hibernate Postgres, then resume Flux. +./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh resume' +``` + +**Un-hibernating Postgres is the step everyone forgets.** The annotation is imperative and survives +a Flux resume, so resuming Flux alone leaves all three databases down and Keycloak, Nextcloud and +Semaphore looking broken for reasons that point nowhere near Postgres. `resume` does it first, for +exactly that reason. + +Flux restores every scaled-down replica count **and** the VM's `running: true` from git, so nothing +needs undoing by hand. + +### What "still broken" looks like when it is actually fine + +* **`4/19` Kustomizations Ready for several minutes.** The dependency graph serialises behind + `platform-config`; 15 of them report `dependency ... is not ready` until it finishes. Check + `platform-config` itself is `Ready=True` and its `lastAppliedRevision` matches the GitRepository + artifact — if it is, wait. +* **One or more Longhorn volumes `degraded`.** Any volume that was still attached at poweroff + rebuilds its lagging replica automatically. On 2026-10-17 it was Prometheus's 50Gi TSDB; it went + healthy on its own. Worry only if it is still degraded after ~30 minutes. +* **`keycloak-operator` crashlooping, then recovering.** It is the CRD reconciler, not in the + request path; Keycloak's StatefulSet stays 2/2 and auth never goes down. See + `/runbooks/keycloak-*` and the known OOM-at-450Mi behaviour. + +### What is actually broken and needs you + +* **An `oauth2-proxy` replica stuck in CrashLoopBackOff.** It performs OIDC discovery at startup and + **exits 1** on the `503 no healthy upstream` it gets while Keycloak is still coming up, then sits + in exponential backoff long after Keycloak is fine. It has no startup dependency on Keycloak. + Delete the pod once and it comes up clean: + + ```bash + ./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n oauth2-proxy delete pod ' + ``` + +* **Orphaned `Unknown`/`Error` Job pods** from CronJobs that were mid-run at poweroff. They never + recover; delete them. + +## Verifying you are actually back + +```bash +./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status' +# nodes 3 Ready | flux 19/19 | longhorn 0 not healthy | postgres 9 pods | unifi 1 VMI + +# DNS, through PiHole's VIP +./infra/devcontainer/dev.sh run bash -c 'nslookup home.keithjackson.dev 192.168.2.23' + +# The Gateway actually serving (401 on an auth-gated host is CORRECT) +for h in home.keithjackson.dev gitea.home.keithjackson.dev plex.home.keithjackson.dev; do + curl -s -o /dev/null -w "$h %{http_code}\n" -k --resolve $h:443:192.168.2.20 https://$h/ +done +``` + +Expect `200`, `200`, `401`. diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index f01643a..bbed9f1 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -52,6 +52,13 @@ configMapGenerator: - backup-failure=backup-failure.md - backup-missed=backup-missed.md - backup-partial=backup-partial.md + # Operator procedure, not an alert page — nothing links here from an alert. Served because + # the whole-cluster shutdown had NO written procedure until 2026-10-17 (infra/cluster.sh is + # virsh-based and only ever applied to the decommissioned dev cluster), so the UPS move was + # improvised live, and the one hazard it exposed -- a StatefulSet with + # persistentVolumeClaimRetentionPolicy whenScaled: Delete loses its PVC when scaled to 0 -- + # destroyed monitoring/loki's volume before anyone noticed. + - cluster-maintenance-shutdown=cluster-maintenance-shutdown.md - cdi-import-stuck=cdi-import-stuck.md - cert-expiry=cert-expiry.md - cert-issuance=cert-issuance.md