home-network-k3s-impl/runbooks/cluster-maintenance-shutdown.md

# cluster-maintenance-shutdown — powering the whole cluster off, and back on

**Operator procedure, not an alert page.** Nothing links here from an alert.
**Script:** `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`)
**Related:** `/runbooks/update-rollback` is for rolling node work where the cluster stays up. This
page is for taking **all three nodes down at once**.

First written after the 2026-10-17 UPS move, which was improvised live because no procedure
existed — `infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev
cluster, so it does nothing for these three baremetal nodes.

## Why you cannot just run `shutdown -h`

You mostly can now, and could not before.

Until 2026-10-17 **kubelet graceful node shutdown was unconfigured on every prod node**, so systemd
stopped k3s, containerd went with it, and every pod was SIGKILLed where it stood — 9 Postgres
instances mid-checkpoint, the UniFi VM, and 25 attached Longhorn volumes.
`roles/k3s_server` now sets `shutdown-grace-period` (and, critically, the
`InhibitDelayMaxSec` logind cap that actually enforces it — without that systemd stops waiting
after 5s and the drain is silently truncated). A plain reboot of one node is therefore safe.

**A planned whole-cluster shutdown still wants `quiesce`**, for three things the kubelet's drain
does not do:

1. **The UniFi VM needs an ACPI guest shutdown**, not an eviction. KubeVirt does that on
   `running: false`; an eviction is a power cut to the guest.
2. **CloudNativePG wants hibernation**, not pod deletion — and hibernation is **imperative** state
   that Flux will neither set nor clear.
3. **Draining three nodes one at a time makes pods reschedule onto the survivors**, churning
   Longhorn replicas for no reason when everything is going down anyway. Scaling in place avoids it.

## The hazard that bit us — read this before scaling anything by hand

**A StatefulSet can carry `persistentVolumeClaimRetentionPolicy.whenScaled: Delete`, and scaling it
to 0 then DELETES its PVCs and their Longhorn volumes.**

On 2026-10-17 the hand-rolled quiesce scaled `monitoring/loki` (which has that policy) to 0, and
`storage-loki-0` was deleted outright. Loki stores with `type: filesystem`, so that PVC was the only
copy. Its 72h retention bounded the loss to a rolling window — nothing about the *mechanism* is
bounded. The only visible symptom was the Longhorn volume count going 25 → 24, which was first
misread as a miscount.

`quiesce.sh` now refuses to scale any such StatefulSet, leaves it running, and tells you which
volumes to expect `degraded` on restart. **If you quiesce by hand, check the policy first:**

```bash
kubectl --context k3s-prod get statefulset -A \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,WHEN_SCALED:.spec.persistentVolumeClaimRetentionPolicy.whenScaled'
```

A hard-detached volume comes back `degraded` and rebuilds itself automatically. That is a vastly
better outcome than a deleted one — take the rebuild.

## Shutting down

Run everything through the pinned toolbox; it supplies `kubectl` and the context.

```bash
# 0. Look before you leap. Refuses to proceed if any Longhorn volume is unhealthy —
#    powering off on top of a degraded volume is how one replica becomes zero.
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'

# 1. Quiesce: suspend Flux, ACPI-stop the VM, hibernate Postgres, scale down every
#    PVC-holding workload (NFS consumers first, PiHole last), wait for detach.
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh quiesce'
```

**Expect DNS to stop** when PiHole scales down — it is the LAN resolver. `kubectl` is unaffected
because the kubeconfig points at the kube-vip IP `192.168.2.9`, not a name.

```bash
# 2. Power the nodes off. Inside the devcontainer, which holds the become password.
./infra/devcontainer/dev.sh run bash -c \
  'cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \
     -m shell -a "shutdown -h +1 \"maintenance\""'
```

`+1` gives Ansible time to disconnect cleanly. The group resolves to all three nodes; confirm with
`--list-hosts` if you want to see them first.

```bash
# 3. Confirm they are actually down before touching anything else.
for h in 192.168.2.10 192.168.2.11 192.168.2.12; do ping -c1 -W2 $h >/dev/null && echo "$h UP" || echo "$h down"; done
```

**Then, in this order and not before:**

| # | Machine | How |
|---|---|---|
| 4 | **TrueNAS** `192.168.2.30` | **Web UI only.** Not in the ansible inventory, and SSH is **refused** (service disabled). `https://192.168.2.30` → power icon → Shut Down. Safe once the nodes are down, because every NFS consumer (`plex-media`, `*-media`, `qbittorrent-downloads`, `nextcloud-legacy-archive`) is already scaled to zero and nothing holds a hard mount. |
| 5 | **Router / switch** | Last. |

TrueNAS must go **after** the k3s nodes: a hard NFS mount on a live node wedges kubelet in
D-state (`/runbooks/nfs-export-latency`, and the load-average symptom in the 2026-10-?? note).

## Bringing it back

**Power on in the reverse order: router/switch → TrueNAS → all three nodes together.** etcd needs
2 of 3 for quorum, so start the nodes as a group rather than one at a time.

```bash
# 1. Nodes Ready, etcd quorum, storage reachable.
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get --raw=/healthz/etcd'

# 2. Resume: un-hibernate Postgres, then resume Flux.
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh resume'
```

**Un-hibernating Postgres is the step everyone forgets.** The annotation is imperative and survives
a Flux resume, so resuming Flux alone leaves all three databases down and Keycloak, Nextcloud and
Semaphore looking broken for reasons that point nowhere near Postgres. `resume` does it first, for
exactly that reason.

Flux restores every scaled-down replica count **and** the VM's `running: true` from git, so nothing
needs undoing by hand.

### What "still broken" looks like when it is actually fine

* **`4/19` Kustomizations Ready for several minutes.** The dependency graph serialises behind
  `platform-config`; 15 of them report `dependency ... is not ready` until it finishes. Check
  `platform-config` itself is `Ready=True` and its `lastAppliedRevision` matches the GitRepository
  artifact — if it is, wait.
* **One or more Longhorn volumes `degraded`.** Any volume that was still attached at poweroff
  rebuilds its lagging replica automatically. On 2026-10-17 it was Prometheus's 50Gi TSDB; it went
  healthy on its own. Worry only if it is still degraded after ~30 minutes.
* **`keycloak-operator` crashlooping, then recovering.** It is the CRD reconciler, not in the
  request path; Keycloak's StatefulSet stays 2/2 and auth never goes down. See
  `/runbooks/keycloak-*` and the known OOM-at-450Mi behaviour.

### What is actually broken and needs you

* **An `oauth2-proxy` replica stuck in CrashLoopBackOff.** It performs OIDC discovery at startup and
  **exits 1** on the `503 no healthy upstream` it gets while Keycloak is still coming up, then sits
  in exponential backoff long after Keycloak is fine. It has no startup dependency on Keycloak.
  Delete the pod once and it comes up clean:

  ```bash
  ./infra/devcontainer/dev.sh run bash -c \
    'kubectl --context k3s-prod -n oauth2-proxy delete pod <name>'
  ```

* **Orphaned `Unknown`/`Error` Job pods** from CronJobs that were mid-run at poweroff. They never
  recover; delete them.

## Verifying you are actually back

```bash
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
# nodes 3 Ready | flux 19/19 | longhorn 0 not healthy | postgres 9 pods | unifi 1 VMI

# DNS, through PiHole's VIP
./infra/devcontainer/dev.sh run bash -c 'nslookup home.keithjackson.dev 192.168.2.23'

# The Gateway actually serving (401 on an auth-gated host is CORRECT)
for h in home.keithjackson.dev gitea.home.keithjackson.dev plex.home.keithjackson.dev; do
  curl -s -o /dev/null -w "$h %{http_code}\n" -k --resolve $h:443:192.168.2.20 https://$h/
done
```

Expect `200`, `200`, `401`.