home-network-k3s-impl/runbooks/cluster-maintenance-shutdown.md

cluster-maintenance-shutdown — powering the whole cluster off, and back on

Operator procedure, not an alert page. Nothing links here from an alert. Script: infra/maintenance/quiesce.sh (quiesce | resume | status) Related: /runbooks/update-rollback is for rolling node work where the cluster stays up. This page is for taking all three nodes down at once.

First written after the 2026-10-17 UPS move, which was improvised live because no procedure existed — infra/cluster.sh stop is virsh-based and only ever applied to the decommissioned dev cluster, so it does nothing for these three baremetal nodes.

Why you cannot just run shutdown -h

You mostly can now, and could not before.

Until 2026-10-17 kubelet graceful node shutdown was unconfigured on every prod node, so systemd stopped k3s, containerd went with it, and every pod was SIGKILLed where it stood — 9 Postgres instances mid-checkpoint, the UniFi VM, and 25 attached Longhorn volumes. roles/k3s_server now sets shutdown-grace-period (and, critically, the InhibitDelayMaxSec logind cap that actually enforces it — without that systemd stops waiting after 5s and the drain is silently truncated). A plain reboot of one node is therefore safe.

A planned whole-cluster shutdown still wants quiesce, for three things the kubelet's drain does not do:

1. The UniFi VM needs an ACPI guest shutdown, not an eviction. KubeVirt does that on running: false; an eviction is a power cut to the guest.
2. CloudNativePG wants hibernation, not pod deletion — and hibernation is imperative state that Flux will neither set nor clear.
3. Draining three nodes one at a time makes pods reschedule onto the survivors, churning Longhorn replicas for no reason when everything is going down anyway. Scaling in place avoids it.

The hazard that bit us — read this before scaling anything by hand

A StatefulSet can carry persistentVolumeClaimRetentionPolicy.whenScaled: Delete, and scaling it to 0 then DELETES its PVCs and their Longhorn volumes.

On 2026-10-17 the hand-rolled quiesce scaled monitoring/loki (which has that policy) to 0, and storage-loki-0 was deleted outright. Loki stores with type: filesystem, so that PVC was the only copy. Its 72h retention bounded the loss to a rolling window — nothing about the mechanism is bounded. The only visible symptom was the Longhorn volume count going 25 → 24, which was first misread as a miscount.

quiesce.sh now refuses to scale any such StatefulSet, leaves it running, and tells you which volumes to expect degraded on restart. If you quiesce by hand, check the policy first:

kubectl --context k3s-prod get statefulset -A \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,WHEN_SCALED:.spec.persistentVolumeClaimRetentionPolicy.whenScaled'

A hard-detached volume comes back degraded and rebuilds itself automatically. That is a vastly better outcome than a deleted one — take the rebuild.

Shutting down

Run everything through the pinned toolbox; it supplies kubectl and the context.

# 0. Look before you leap. Refuses to proceed if any Longhorn volume is unhealthy —
#    powering off on top of a degraded volume is how one replica becomes zero.
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'

# 1. Quiesce: suspend Flux, ACPI-stop the VM, hibernate Postgres, scale down every
#    PVC-holding workload (NFS consumers first, PiHole last), wait for detach.
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh quiesce'

Expect DNS to stop when PiHole scales down — it is the LAN resolver. kubectl is unaffected because the kubeconfig points at the kube-vip IP 192.168.2.9, not a name.

# 2. Power the nodes off. Inside the devcontainer, which holds the become password.
./infra/devcontainer/dev.sh run bash -c \
  'cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \
     -m shell -a "shutdown -h +1 \"maintenance\""'

+1 gives Ansible time to disconnect cleanly. The group resolves to all three nodes; confirm with --list-hosts if you want to see them first.

# 3. Confirm they are actually down before touching anything else.
for h in 192.168.2.10 192.168.2.11 192.168.2.12; do ping -c1 -W2 $h >/dev/null && echo "$h UP" || echo "$h down"; done

Then, in this order and not before:

#	Machine	How
4	TrueNAS 192.168.2.30	Web UI only. Not in the ansible inventory, and SSH is refused (service disabled). https://192.168.2.30 → power icon → Shut Down. Safe once the nodes are down, because every NFS consumer (plex-media, *-media, qbittorrent-downloads, nextcloud-legacy-archive) is already scaled to zero and nothing holds a hard mount.
5	Router / switch	Last.

TrueNAS must go after the k3s nodes: a hard NFS mount on a live node wedges kubelet in D-state (/runbooks/nfs-export-latency, and the load-average symptom in the 2026-10-?? note).

Bringing it back

Power on in the reverse order: router/switch → TrueNAS → all three nodes together. etcd needs 2 of 3 for quorum, so start the nodes as a group rather than one at a time.

# 1. Nodes Ready, etcd quorum, storage reachable.
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get --raw=/healthz/etcd'

# 2. Resume: un-hibernate Postgres, then resume Flux.
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh resume'

Un-hibernating Postgres is the step everyone forgets. The annotation is imperative and survives a Flux resume, so resuming Flux alone leaves all three databases down and Keycloak, Nextcloud and Semaphore looking broken for reasons that point nowhere near Postgres. resume does it first, for exactly that reason.

Flux restores every scaled-down replica count and the VM's running: true from git, so nothing needs undoing by hand.

What "still broken" looks like when it is actually fine

• 4/19 Kustomizations Ready for several minutes. The dependency graph serialises behind platform-config; 15 of them report dependency ... is not ready until it finishes. Check platform-config itself is Ready=True and its lastAppliedRevision matches the GitRepository artifact — if it is, wait.
• One or more Longhorn volumes degraded. Any volume that was still attached at poweroff rebuilds its lagging replica automatically. On 2026-10-17 it was Prometheus's 50Gi TSDB; it went healthy on its own. Worry only if it is still degraded after ~30 minutes.
• keycloak-operator crashlooping, then recovering. It is the CRD reconciler, not in the request path; Keycloak's StatefulSet stays 2/2 and auth never goes down. See /runbooks/keycloak-* and the known OOM-at-450Mi behaviour.

What is actually broken and needs you

• An oauth2-proxy replica stuck in CrashLoopBackOff. It performs OIDC discovery at startup and exits 1 on the 503 no healthy upstream it gets while Keycloak is still coming up, then sits in exponential backoff long after Keycloak is fine. It has no startup dependency on Keycloak. Delete the pod once and it comes up clean:

  ./infra/devcontainer/dev.sh run bash -c \
    'kubectl --context k3s-prod -n oauth2-proxy delete pod <name>'
  

• Orphaned Unknown/Error Job pods from CronJobs that were mid-run at poweroff. They never recover; delete them.

Verifying you are actually back

./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
# nodes 3 Ready | flux 19/19 | longhorn 0 not healthy | postgres 9 pods | unifi 1 VMI

# DNS, through PiHole's VIP
./infra/devcontainer/dev.sh run bash -c 'nslookup home.keithjackson.dev 192.168.2.23'

# The Gateway actually serving (401 on an auth-gated host is CORRECT)
for h in home.keithjackson.dev gitea.home.keithjackson.dev plex.home.keithjackson.dev; do
  curl -s -o /dev/null -w "$h %{http_code}\n" -k --resolve $h:443:192.168.2.20 https://$h/
done

Expect 200, 200, 401.