diff --git a/decisions/2026-10-17-graceful-cluster-shutdown.md b/decisions/2026-10-17-graceful-cluster-shutdown.md new file mode 100644 index 0000000..ff43169 --- /dev/null +++ b/decisions/2026-10-17-graceful-cluster-shutdown.md @@ -0,0 +1,106 @@ +--- +date: 2026-10-17 +branch: feat/graceful-cluster-shutdown +title: "Graceful node shutdown was never configured, and scaling a StatefulSet deleted its PVC" +--- + +## What changed + +All three nodes had to be powered off to move them onto a UPS. There was **no procedure for it**: +`infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev cluster, so +the whole sequence was improvised live. Two defects came out of doing it. + +**1. Kubelet graceful node shutdown was unconfigured on every prod node.** No `shutdownGracePeriod` +anywhere in `infra/ansible/`, so `shutdown -h` stopped k3s, containerd went with it, and every pod +was SIGKILLed where it stood — 9 CloudNativePG instances mid-checkpoint, the UniFi VM, and 25 +attached Longhorn volumes. `roles/k3s_server` now writes `kubelet-arg` into +`/etc/rancher/k3s/config.yaml` with `shutdown-grace-period=90s` and +`shutdown-grace-period-critical-pods=20s`. + +The half that is easy to miss and silently defeats the other half: **graceful node shutdown is +enforced by logind, not by the kubelet.** The kubelet holds an inhibitor *delay* lock and systemd +waits at most `InhibitDelayMaxSec` — **default 5 seconds** — before proceeding regardless. A grace +period above that cap is truncated with no error, so the feature looks configured and still +hard-kills pods. The role now ships a `logind.conf.d` drop-in setting it to 120, and **asserts** +`k3s_logind_inhibit_delay_max_sec >= k3s_kubelet_shutdown_grace_period` rather than documenting the +relationship, because the failure is invisible. + +**2. Scaling a StatefulSet to 0 DELETED its PVC and Longhorn volume.** `monitoring/loki` carries +`persistentVolumeClaimRetentionPolicy: {whenDeleted: Delete, whenScaled: Delete}`, so the quiesce's +`scale --replicas=0` destroyed `storage-loki-0` outright. Loki stores with `type: filesystem`, so +that PVC was the only copy of its log data. This is the defect in this change set that actually +destroyed something, and it was mine. + +New `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`) encodes the sequence that +worked, and **refuses to scale any StatefulSet whose `whenScaled` is `Delete`** — it leaves it +running, accepts that its volume hard-detaches at poweroff, and prints which volumes to expect +`degraded`. A rebuild is a vastly better outcome than a deletion. Workload discovery is computed +from the live cluster (every Deployment/StatefulSet whose pods mount a PVC, Longhorn or TrueNAS NFS) +rather than hardcoded, so a newly onboarded service is covered without anyone remembering to add it. + +`runbooks/cluster-maintenance-shutdown.md` (new, registered in `runbooks/kustomization.yaml`) is the +operator page: shutdown order, the TrueNAS-after-nodes constraint, power-on order, the +un-hibernate-Postgres step Flux will not do for you, and a "what looks broken but is fine" section +separating the `4/19`-Kustomizations-Ready cascade, the degraded-and-rebuilding volume and the +self-recovering `keycloak-operator` from the one thing that genuinely needs a human (a stuck +`oauth2-proxy` replica). + +## What I verified + +* **Graceful shutdown was genuinely absent**, not merely undocumented: `grep -rn + 'shutdownGracePeriod|graceful-node-shutdown'` over `infra/ansible/` returns nothing, and no + `logind` or `InhibitDelay` setting exists anywhere in the tree. +* **The whole shutdown/restart cycle was executed end to end on prod 2026-10-17** and the cluster + came back: 3/3 nodes Ready with etcd `healthz` ok, 19/19 Flux Kustomizations Ready, + `platform-config`'s `lastAppliedRevision` equal to the GitRepository artifact revision, all 3 + CNPG clusters `healthy` with primaries elected, the UniFi VM `Running`, PiHole resolving + (`home.keithjackson.dev` → `192.168.2.20` via `192.168.2.23`), and the Gateway serving + `200/200/401` for apex/gitea/plex. +* **The PVC-deletion mechanism is confirmed, not inferred:** `kubectl get sts loki -o + jsonpath={.spec.persistentVolumeClaimRetentionPolicy}` returns + `{"whenDeleted":"Delete","whenScaled":"Delete"}`, the PVC was `NotFound` after the cycle, and the + Longhorn volume count had gone 25 → 24. +* **The damage was repaired and verified:** `scale sts loki --replicas=1` (restoring the replica + count the chart declares) recreated `storage-loki-0` as a fresh 50Gi Bound PVC, `loki-0` reached + `2/2 Running`, and the volume count returned to 25. +* **The guard works against the live cluster**, tested both ways: `sts_deletes_pvc_on_scale + monitoring loki` returns true (would be skipped) and `pihole pihole` returns false (safe to + scale). +* **The script's read-only paths run clean on prod:** `status` reports correctly, and the discovery + function returns the 13 workloads currently holding a PVC. +* `bash -n` clean; the two edited Ansible YAML files parse. + +## What I did NOT verify + +* **The kubelet change has NEVER BEEN APPLIED OR TESTED.** This is the significant gap. I did not + run the role, did not restart k3s, and have **not** observed a node actually drain on + `shutdown -h`. Everything about it is reasoning from kubelet/systemd semantics plus the measured + absence of the setting. Applying it needs a **rolling** k3s restart (`serial: 1` — kube-vip keeps + the API up, etcd holds at 2/3), which is an operator action (CLAUDE.md §5). **Verify by draining + one node for real:** apply, then `shutdown -h now` on a single node and watch whether pods + terminate gracefully instead of vanishing. +* **Whether 90s/20s are the right numbers.** Chosen to comfortably cover a Postgres fast shutdown + and a Longhorn flush; not measured against either. The budget is spent on every reboot, so if a + node takes noticeably longer to go down, this is why. +* **Whether k3s accepts `kubelet-arg` in `config.yaml` as written.** The file already carries + `kube-apiserver-arg` as a list and `kubelet-arg` is the documented sibling, but I did not render + or apply it, so a typo would surface only at k3s restart — and a bad `config.yaml` is exactly the + kind of thing that makes k3s fail to come up. **Review this one before applying.** +* **The `assert` task's `regex_replace` on the grace period** assumes a plain `90s`-style value; a + value expressed as `1m30s` would parse to the nonsense integer `130`. Fine for the committed + defaults, wrong if someone writes a compound duration. +* **`resume`'s Flux-reconcile wait has never run**, and neither has `quiesce` as a script — the + 2026-10-17 cycle was done with the equivalent commands by hand, so the script encodes a sequence + that worked but has not itself been executed end to end. Only `status` and the discovery and guard + functions have run. +* **Loki's lost log window is gone for good.** Up to 72h of logs (its `retention_period`), destroyed + by my scale-down. Nothing recovers it; Velero's schedules cover the namespace but the PVC was + deleted, not corrupted, and I did not attempt a restore because the data self-expires anyway. +* **No other `whenScaled: Delete` StatefulSet was audited.** The guard now catches them generically, + but I did not enumerate which other workloads carry the policy, so I cannot say loki was the only + one at risk. The runbook gives the one-liner to list them. +* **TrueNAS and the router are entirely outside this change.** TrueNAS is not in the Ansible + inventory and refuses SSH, so its shutdown stays a web-UI step with no automation and no check. +* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1 + offline subset only. +* I did not re-measure branch protection or runner state (CLAUDE.md §2b). diff --git a/infra/ansible/roles/k3s_server/defaults/main.yml b/infra/ansible/roles/k3s_server/defaults/main.yml index 31e950f..60f9bc7 100644 --- a/infra/ansible/roles/k3s_server/defaults/main.yml +++ b/infra/ansible/roles/k3s_server/defaults/main.yml @@ -107,3 +107,31 @@ k3s_base_domain: "home.keithjackson.dev" # Derived, so the realm path is stated once. Override k3s_base_domain, not this. k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab" + +# --- Graceful node shutdown (kubelet) ---------------------------------------------------------- +# MEASURED 2026-10-17, during the UPS move: graceful node shutdown was NOT configured on any prod +# node, so `shutdown -h` did NOT drain pods. systemd stops k3s, containerd goes with it, and every +# pod is SIGKILLed where it stands -- including 9 CloudNativePG instances mid-checkpoint, the UniFi +# VM, and 25 attached Longhorn volumes. The whole-cluster shutdown that day had to quiesce +# everything BY HAND first (runbooks/cluster-maintenance-shutdown.md), and the one volume that did +# not finish detaching came back `degraded` and had to rebuild. +# +# With these set, the kubelet takes a systemd inhibitor lock and, on shutdown, evicts ordinary pods +# first and critical ones last, so a plain `shutdown -h`/reboot drains the node. That makes the +# manual scale-down in the runbook a belt-and-braces step for a PLANNED whole-cluster shutdown +# rather than the only thing standing between a reboot and a hard kill. +# +# 90s/20s, not larger: the budget is spent on every reboot and node upgrade, and the kubelet waits +# the FULL ordinary period before it starts on critical pods. Postgres does a fast shutdown in +# well under 90s here, and Longhorn needs only enough time to flush and detach. +k3s_kubelet_shutdown_grace_period: "90s" +k3s_kubelet_shutdown_grace_period_critical_pods: "20s" + +# *** THE HALF THAT IS EASY TO MISS, AND SILENTLY BREAKS THE OTHER HALF. *** +# Graceful node shutdown works by the kubelet holding a logind INHIBITOR DELAY lock. systemd caps +# how long any inhibitor may delay shutdown at `InhibitDelayMaxSec`, which DEFAULTS TO 5 SECONDS. +# If that cap is below the grace period above, systemd proceeds anyway and the drain is truncated +# to ~5s -- i.e. the feature appears configured, reports no error, and still hard-kills pods. So +# this MUST be >= k3s_kubelet_shutdown_grace_period; the role asserts it rather than trusting it. +# Set a little above the kubelet's total (90s + 20s) to leave headroom. +k3s_logind_inhibit_delay_max_sec: 120 diff --git a/infra/ansible/roles/k3s_server/handlers/main.yml b/infra/ansible/roles/k3s_server/handlers/main.yml index 2070f87..66acb0c 100644 --- a/infra/ansible/roles/k3s_server/handlers/main.yml +++ b/infra/ansible/roles/k3s_server/handlers/main.yml @@ -15,3 +15,12 @@ port: "{{ k3s_api_port }}" timeout: 180 listen: Restart k3s + +# logind re-reads InhibitDelayMaxSec only on restart. Restarting logind does NOT log users out or +# disturb k3s -- it manages login sessions and inhibitor locks, not workloads. The kubelet picks up +# the new cap when it next takes its lock, so no k3s restart is needed for THIS half. +- name: Restart logind + ansible.builtin.systemd_service: + name: systemd-logind + state: restarted + listen: Restart logind diff --git a/infra/ansible/roles/k3s_server/tasks/main.yml b/infra/ansible/roles/k3s_server/tasks/main.yml index 8cfbee1..cf60d16 100644 --- a/infra/ansible/roles/k3s_server/tasks/main.yml +++ b/infra/ansible/roles/k3s_server/tasks/main.yml @@ -86,6 +86,16 @@ # k3s_base_domain (defaults/main.yml), whose [parity] contract with the cluster's # BASE_DOMAIN is documented there. Dev's default renders the value it already runs, so # config.yaml is unchanged and the Restart handler does not fire. + # ── Graceful node shutdown ────────────────────────────────────────────────────────────── + # Without these the kubelet takes no systemd inhibitor lock, so `shutdown -h` stops + # containerd and SIGKILLs every pod where it stands -- MEASURED on all three prod nodes + # 2026-10-17 (defaults/main.yml has the detail). The companion logind drop-in below raises + # InhibitDelayMaxSec, WITHOUT WHICH systemd caps the drain at ~5s and these values are + # quietly ignored. + kubelet-arg: + - "shutdown-grace-period={{ k3s_kubelet_shutdown_grace_period }}" + - "shutdown-grace-period-critical-pods={{ k3s_kubelet_shutdown_grace_period_critical_pods }}" + kube-apiserver-arg: - "oidc-issuer-url={{ k3s_oidc_issuer_url }}" - "oidc-client-id=kubernetes" @@ -183,3 +193,46 @@ host: "127.0.0.1" port: "{{ k3s_api_port }}" timeout: 180 + +# ── Graceful node shutdown: the systemd half — tag `shutdown` ────────────────────────────────── +# The kubelet's shutdown-grace-period is ENFORCED BY logind, not by the kubelet: the kubelet holds +# an inhibitor DELAY lock, and systemd will wait at most `InhibitDelayMaxSec` for it (default 5s) +# before proceeding regardless. A grace period above that cap is therefore silently truncated -- +# the feature looks configured, logs no error, and still hard-kills pods. Assert the relationship +# instead of documenting it, because the failure is invisible. +- name: Assert the logind inhibitor cap can actually cover the kubelet grace period + ansible.builtin.assert: + that: + - (k3s_logind_inhibit_delay_max_sec | int) >= (k3s_kubelet_shutdown_grace_period | regex_replace('[^0-9]', '') | int) + fail_msg: >- + k3s_logind_inhibit_delay_max_sec ({{ k3s_logind_inhibit_delay_max_sec }}s) is below + k3s_kubelet_shutdown_grace_period ({{ k3s_kubelet_shutdown_grace_period }}). systemd would + stop waiting for the kubelet's inhibitor lock before the drain finished, so pods would be + SIGKILLed anyway and graceful node shutdown would be configured but ineffective. + quiet: true + tags: [shutdown] + +- name: Ensure /etc/systemd/logind.conf.d exists + ansible.builtin.file: + path: /etc/systemd/logind.conf.d + state: directory + owner: root + group: root + mode: "0755" + tags: [shutdown] + +# A drop-in, not an edit of logind.conf: the distro file stays pristine and `state: absent` here +# would fully revert the change. +- name: Raise the logind inhibitor delay cap so the kubelet can finish draining + ansible.builtin.copy: + dest: /etc/systemd/logind.conf.d/10-k3s-graceful-shutdown.conf + owner: root + group: root + mode: "0644" + content: | + # Managed by Ansible (roles/k3s_server). See defaults/main.yml. + # MUST stay >= kubelet shutdown-grace-period, or the drain is truncated to this value. + [Login] + InhibitDelayMaxSec={{ k3s_logind_inhibit_delay_max_sec }} + notify: Restart logind + tags: [shutdown] diff --git a/infra/maintenance/quiesce.sh b/infra/maintenance/quiesce.sh new file mode 100755 index 0000000..8577fb7 --- /dev/null +++ b/infra/maintenance/quiesce.sh @@ -0,0 +1,265 @@ +#!/usr/bin/env bash +# ================================================================================================= +# quiesce.sh — bring the prod cluster to a safe standstill before powering nodes off, and bring it +# back afterwards. Companion to runbooks/cluster-maintenance-shutdown.md. +# ================================================================================================= +# WHY THIS EXISTS. On 2026-10-17 all three nodes had to be powered off to move them onto a UPS, and +# there was NO procedure for it: `infra/cluster.sh stop` is `virsh`-based and only ever applied to +# the decommissioned dev cluster. The whole sequence was improvised live. This script is that +# sequence, so the next person does not improvise it. +# +# WHY IT IS NOT JUST `shutdown -h`. Graceful node shutdown was unconfigured on every prod node that +# day, so systemd stopped containerd and SIGKILLed every pod where it stood. roles/k3s_server now +# sets shutdown-grace-period (+ the logind InhibitDelayMaxSec cap that actually enforces it), which +# makes a bare reboot safe. This script is still the right tool for a PLANNED whole-cluster +# shutdown, for three reasons the kubelet's drain does not cover: +# 1. The UniFi VM needs an ACPI guest shutdown, not an eviction. +# 2. CloudNativePG wants hibernation, not pod deletion -- and hibernation is IMPERATIVE state +# that Flux will not set or clear for you. +# 3. Draining three nodes one at a time makes pods RESCHEDULE onto the survivors, churning +# Longhorn replicas for no reason when the whole cluster is going down anyway. Scaling in +# place avoids all of it. +# +# SAFE TO RE-RUN. Every step is idempotent and `resume` is the exact inverse of `quiesce`. +# +# WHAT IT DELIBERATELY DOES NOT DO: power anything off. Node poweroff is an operator action +# (CLAUDE.md §5) and needs the become password; the runbook gives the ansible one-liner. It also +# does not touch TrueNAS (not in the inventory, SSH refused -- web UI only) or the router. +set -euo pipefail + +CONTEXT="${CONTEXT:-k3s-prod}" +KUBECTL=(kubectl --context "$CONTEXT" --request-timeout=30s) +TIMEOUT="${TIMEOUT:-600}" + +# The three CloudNativePG clusters, as namespace/name. Hibernation is per-Cluster. +CNPG=(keycloak/keycloak-db nextcloud-db/nextcloud-db semaphore-db/semaphore-db) + +log() { printf '\033[34m[quiesce]\033[0m %s\n' "$*"; } +ok() { printf '\033[32m[ ok ]\033[0m %s\n' "$*"; } +warn() { printf '\033[33m[warn]\033[0m %s\n' "$*"; } +die() { printf '\033[31m[fail]\033[0m %s\n' "$*" >&2; exit 1; } + +need() { command -v "$1" >/dev/null || die "$1 not on PATH — run this inside ./infra/devcontainer/dev.sh"; } + +# ---------------------------------------------------------------- discovery +# Every Deployment/StatefulSet whose pods mount a PVC — Longhorn OR the TrueNAS NFS bulk tier. +# Computed from the LIVE cluster rather than hardcoded, so a newly onboarded service is covered +# without anyone remembering to add it here. PROTECTED namespaces are excluded because scaling +# them down would remove the very machinery that performs the detach (longhorn-system), keep the +# API reachable (kube-system), or finish the hibernation and the VM shutdown (cnpg-system, +# kubevirt). +pvc_workloads() { + "${KUBECTL[@]}" get pods -A -o json | python3 -c ' +import json,sys +PROTECT={"longhorn-system","kube-system","cilium-secrets","cnpg-system","kubevirt","flux-system","cdi"} +out={} +for p in json.load(sys.stdin)["items"]: + ns=p["metadata"]["namespace"] + if ns in PROTECT: continue + if not any(v.get("persistentVolumeClaim") for v in (p["spec"].get("volumes") or [])): continue + ow=(p["metadata"].get("ownerReferences") or []) + if not ow: continue + k,n=ow[0]["kind"],ow[0]["name"] + if k=="ReplicaSet": k,n="deployment",n.rsplit("-",1)[0] + elif k=="StatefulSet": k="statefulset" + else: continue # Jobs, and CNPG Clusters, are handled separately + out[(ns,k,n)]=1 +for ns,k,n in sorted(out): print(ns,k,n) +' +} + +# ---------------------------------------------------------------- the PVC-deletion guard +# *** DO NOT REMOVE THIS CHECK. IT EXISTS BECAUSE SCALING A StatefulSet DESTROYED DATA. *** +# A StatefulSet may carry `persistentVolumeClaimRetentionPolicy.whenScaled: Delete`, and then +# scaling it to 0 DELETES ITS PVCs -- and with them the Longhorn volumes. MEASURED 2026-10-17: the +# quiesce for the UPS move scaled `monitoring/loki` (whenScaled: Delete) to 0 and its +# `storage-loki-0` PVC and volume were deleted outright. Loki stores with `type: filesystem`, so +# that PVC was the only copy; 72h retention bounded the loss to a rolling window, but nothing about +# the mechanism is bounded -- the same action on a Delete-policy StatefulSet holding real state +# would be unrecoverable. The volume count going 25 -> 24 was the only visible symptom, and it was +# first misread as a miscount. +# +# So: such a StatefulSet is NEVER scaled. It is left RUNNING and its volume hard-detaches at +# poweroff, which costs a replica rebuild on restart (degraded -> healthy, automatic). A rebuild is +# an enormously better outcome than a deletion, and that is the whole trade being made here. +sts_deletes_pvc_on_scale() { # ns name -> 0 if scaling it would delete its PVCs + local policy + policy=$("${KUBECTL[@]}" -n "$1" get statefulset "$2" \ + -o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy.whenScaled}' 2>/dev/null || true) + [ "$policy" = "Delete" ] +} + +wait_for() { # wait_for + local desc="$1" cond="$2" waited=0 + log "waiting: $desc (up to ${TIMEOUT}s)" + until eval "$cond"; do + (( waited += 10 )); (( waited >= TIMEOUT )) && { warn "TIMED OUT waiting: $desc"; return 1; } + sleep 10 + done + ok "$desc" +} + +# ---------------------------------------------------------------- quiesce +do_quiesce() { + need kubectl; need python3 + + # Refuse to start if Longhorn is already unhealthy. Powering off on top of a degraded volume is + # how one replica becomes zero; better to stop here and let it rebuild first. + local unhealthy + unhealthy=$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io \ + -o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true) + [ "$unhealthy" = "0" ] || die "$unhealthy Longhorn volume(s) are not healthy — fix or wait for rebuild BEFORE powering off" + ok "Longhorn: all volumes healthy" + + # 1. Flux first, or it reverts every scale-down below within its reconcile interval. + log "suspending Flux Kustomizations" + local k + for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do + "${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":true}}' >/dev/null + done + ok "all Kustomizations suspended" + + # 2. The VM needs an ACPI shutdown of the GUEST. KubeVirt does that on running:false; a plain + # pod eviction would be a power cut to the VM. + if "${KUBECTL[@]}" -n unifi get vm unifi >/dev/null 2>&1; then + log "ACPI-stopping the UniFi VM" + "${KUBECTL[@]}" -n unifi patch vm unifi --type=merge -p '{"spec":{"running":false}}' >/dev/null + fi + + # 3. Postgres: hibernation is a clean smart shutdown that KEEPS the PVCs. CNPG does one instance + # at a time, primary last-ish, so this is not instant. + log "hibernating CloudNativePG clusters" + local c ns name + for c in "${CNPG[@]}"; do + ns="${c%%/*}"; name="${c##*/}" + "${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \ + cnpg.io/hibernation=on --overwrite >/dev/null 2>&1 || warn "could not hibernate $c" + done + + # 4. Everything holding a PVC, scaled in place. NFS consumers first so nothing holds a hard + # TrueNAS mount; PiHole last because it is the LAN resolver. + log "scaling down PVC-holding workloads" + local line ns2 kind nm deferred=() skipped_delete_policy=() + while read -r ns2 kind nm; do + [ -z "${ns2:-}" ] && continue + if [ "$kind" = "statefulset" ] && sts_deletes_pvc_on_scale "$ns2" "$nm"; then + warn "SKIPPING $ns2/$nm — whenScaled: Delete would DELETE its PVC (see the guard above)." + warn " Left running; its volume hard-detaches at poweroff and rebuilds on restart." + skipped_delete_policy+=("$ns2/$nm") + continue + fi + if [ "$ns2" = "pihole" ]; then deferred+=("$ns2 $kind $nm"); continue; fi + "${KUBECTL[@]}" -n "$ns2" scale "$kind" "$nm" --replicas=0 >/dev/null 2>&1 \ + && printf ' scaled 0 %s/%s %s\n' "$ns2" "$kind" "$nm" \ + || warn "could not scale $ns2/$kind/$nm" + done < <(pvc_workloads) + for line in "${deferred[@]:-}"; do + [ -z "${line:-}" ] && continue + # shellcheck disable=SC2086 + set -- $line + "${KUBECTL[@]}" -n "$1" scale "$2" "$3" --replicas=0 >/dev/null 2>&1 \ + && printf ' scaled 0 %s/%s %s (LAN DNS — expect name resolution to stop)\n' "$1" "$2" "$3" + done + + # 5. The only assertion that matters. An attached volume at poweroff comes back degraded and + # rebuilds; this is what we are buying with all of the above. + wait_for "UniFi VMI gone" \ + '[ "$('"${KUBECTL[*]}"' -n unifi get vmi --no-headers 2>/dev/null | wc -l)" = "0" ]' || true + wait_for "all Postgres instances stopped" \ + '[ "$('"${KUBECTL[*]}"' get pods -A --no-headers 2>/dev/null | grep -cE "(keycloak|nextcloud|semaphore)-db-[0-9]")" = "0" ]' || true + wait_for "all Longhorn volumes detached" \ + '[ "$('"${KUBECTL[*]}"' -n longhorn-system get volumes.longhorn.io -o jsonpath="{range .items[*]}{.status.state}{\"\n\"}{end}" 2>/dev/null | grep -c attached)" = "0" ]' || \ + warn "some volumes are STILL ATTACHED — they will come back degraded and rebuild; acceptable, but note which" + + if [ "${#skipped_delete_policy[@]}" -gt 0 ]; then + echo + warn "NOT scaled down (whenScaled: Delete — scaling them would delete their PVCs):" + printf ' %s\n' "${skipped_delete_policy[@]}" + warn "Expect these volumes to come back 'degraded' and rebuild. That is correct and automatic." + fi + echo + ok "QUIESCED. Safe to power off." + cat <<'NEXT' + + Next, from inside the devcontainer (it holds the become password): + cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \ + -m shell -a 'shutdown -h +1 "maintenance"' + + THEN, in this order and not before: + 1. TrueNAS (192.168.2.30) — web UI only; not in the ansible inventory and SSH is refused. + 2. Router / switch last. +NEXT +} + +# ---------------------------------------------------------------- resume +do_resume() { + need kubectl + + # THE STEP THAT IS ALWAYS FORGOTTEN. Hibernation is imperative state: Flux neither sets nor + # clears it, so resuming Flux alone leaves all three databases down and every app that needs + # one looking broken for reasons that point nowhere near Postgres. + log "un-hibernating CloudNativePG clusters" + local c ns name + for c in "${CNPG[@]}"; do + ns="${c%%/*}"; name="${c##*/}" + "${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \ + cnpg.io/hibernation=off --overwrite >/dev/null 2>&1 || warn "could not un-hibernate $c" + done + + # Flux restores every replica count AND the VM's running:true from git, so nothing here needs to + # undo the scale-downs by hand. + log "resuming Flux Kustomizations" + local k + for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do + "${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":false}}' >/dev/null + done + ok "all Kustomizations resumed" + + wait_for "Flux fully reconciled" \ + '[ "$('"${KUBECTL[*]}"' -n flux-system get kustomizations --no-headers 2>/dev/null | grep -vc True)" = "0" ]' || \ + warn "some Kustomizations are not Ready — the graph serialises behind platform-config, so give it a few minutes before worrying" + + echo + ok "RESUMED." + cat <<'NEXT' + + Known cold-start race, not a fault in this script: oauth2-proxy performs OIDC discovery at + startup and exits 1 on the 503 it gets while Keycloak is still coming up, then sits in + exponential CrashLoopBackOff. If a replica is stuck, delete the pod once: + kubectl --context k3s-prod -n oauth2-proxy delete pod +NEXT +} + +do_status() { + need kubectl + printf 'nodes: %s Ready\n' "$("${KUBECTL[@]}" get nodes --no-headers | grep -c ' Ready')" + printf 'flux: %s/%s Ready\n' \ + "$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | grep -c True)" \ + "$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | wc -l)" + printf 'longhorn: %s attached, %s detached, %s not healthy\n' \ + "$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c attached)" \ + "$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c detached)" \ + "$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true)" + printf 'postgres: %s instance pod(s)\n' \ + "$("${KUBECTL[@]}" get pods -A --no-headers | grep -cE '(keycloak|nextcloud|semaphore)-db-[0-9]' || true)" + printf 'unifi VM: %s VMI\n' "$("${KUBECTL[@]}" -n unifi get vmi --no-headers 2>/dev/null | wc -l)" +} + +case "${1:-}" in + quiesce) do_quiesce ;; + resume) do_resume ;; + status) do_status ;; + *) cat >&2 </dev/null && echo "$h UP" || echo "$h down"; done +``` + +**Then, in this order and not before:** + +| # | Machine | How | +|---|---|---| +| 4 | **TrueNAS** `192.168.2.30` | **Web UI only.** Not in the ansible inventory, and SSH is **refused** (service disabled). `https://192.168.2.30` → power icon → Shut Down. Safe once the nodes are down, because every NFS consumer (`plex-media`, `*-media`, `qbittorrent-downloads`, `nextcloud-legacy-archive`) is already scaled to zero and nothing holds a hard mount. | +| 5 | **Router / switch** | Last. | + +TrueNAS must go **after** the k3s nodes: a hard NFS mount on a live node wedges kubelet in +D-state (`/runbooks/nfs-export-latency`, and the load-average symptom in the 2026-10-?? note). + +## Bringing it back + +**Power on in the reverse order: router/switch → TrueNAS → all three nodes together.** etcd needs +2 of 3 for quorum, so start the nodes as a group rather than one at a time. + +```bash +# 1. Nodes Ready, etcd quorum, storage reachable. +./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' +./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get --raw=/healthz/etcd' + +# 2. Resume: un-hibernate Postgres, then resume Flux. +./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh resume' +``` + +**Un-hibernating Postgres is the step everyone forgets.** The annotation is imperative and survives +a Flux resume, so resuming Flux alone leaves all three databases down and Keycloak, Nextcloud and +Semaphore looking broken for reasons that point nowhere near Postgres. `resume` does it first, for +exactly that reason. + +Flux restores every scaled-down replica count **and** the VM's `running: true` from git, so nothing +needs undoing by hand. + +### What "still broken" looks like when it is actually fine + +* **`4/19` Kustomizations Ready for several minutes.** The dependency graph serialises behind + `platform-config`; 15 of them report `dependency ... is not ready` until it finishes. Check + `platform-config` itself is `Ready=True` and its `lastAppliedRevision` matches the GitRepository + artifact — if it is, wait. +* **One or more Longhorn volumes `degraded`.** Any volume that was still attached at poweroff + rebuilds its lagging replica automatically. On 2026-10-17 it was Prometheus's 50Gi TSDB; it went + healthy on its own. Worry only if it is still degraded after ~30 minutes. +* **`keycloak-operator` crashlooping, then recovering.** It is the CRD reconciler, not in the + request path; Keycloak's StatefulSet stays 2/2 and auth never goes down. See + `/runbooks/keycloak-*` and the known OOM-at-450Mi behaviour. + +### What is actually broken and needs you + +* **An `oauth2-proxy` replica stuck in CrashLoopBackOff.** It performs OIDC discovery at startup and + **exits 1** on the `503 no healthy upstream` it gets while Keycloak is still coming up, then sits + in exponential backoff long after Keycloak is fine. It has no startup dependency on Keycloak. + Delete the pod once and it comes up clean: + + ```bash + ./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n oauth2-proxy delete pod ' + ``` + +* **Orphaned `Unknown`/`Error` Job pods** from CronJobs that were mid-run at poweroff. They never + recover; delete them. + +## Verifying you are actually back + +```bash +./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status' +# nodes 3 Ready | flux 19/19 | longhorn 0 not healthy | postgres 9 pods | unifi 1 VMI + +# DNS, through PiHole's VIP +./infra/devcontainer/dev.sh run bash -c 'nslookup home.keithjackson.dev 192.168.2.23' + +# The Gateway actually serving (401 on an auth-gated host is CORRECT) +for h in home.keithjackson.dev gitea.home.keithjackson.dev plex.home.keithjackson.dev; do + curl -s -o /dev/null -w "$h %{http_code}\n" -k --resolve $h:443:192.168.2.20 https://$h/ +done +``` + +Expect `200`, `200`, `401`. diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index f01643a..bbed9f1 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -52,6 +52,13 @@ configMapGenerator: - backup-failure=backup-failure.md - backup-missed=backup-missed.md - backup-partial=backup-partial.md + # Operator procedure, not an alert page — nothing links here from an alert. Served because + # the whole-cluster shutdown had NO written procedure until 2026-10-17 (infra/cluster.sh is + # virsh-based and only ever applied to the decommissioned dev cluster), so the UPS move was + # improvised live, and the one hazard it exposed -- a StatefulSet with + # persistentVolumeClaimRetentionPolicy whenScaled: Delete loses its PVC when scaled to 0 -- + # destroyed monitoring/loki's volume before anyone noticed. + - cluster-maintenance-shutdown=cluster-maintenance-shutdown.md - cdi-import-stuck=cdi-import-stuck.md - cert-expiry=cert-expiry.md - cert-issuance=cert-issuance.md