jakman85andClaude Opus 5
2f2e98aa19
All three nodes had to be powered off to move them onto a UPS, and there was NO procedure: infra/cluster.sh stop is virsh-based and only ever applied to the decommissioned dev cluster. The sequence was improvised live. Two defects came out of doing it. 1. GRACEFUL NODE SHUTDOWN WAS UNCONFIGURED ON EVERY PROD NODE. No shutdownGracePeriod anywhere in infra/ansible/, so `shutdown -h` stopped k3s, containerd went with it, and every pod was SIGKILLed where it stood -- 9 CNPG instances mid-checkpoint, the UniFi VM, 25 attached Longhorn volumes. roles/k3s_server now sets kubelet-arg shutdown-grace-period=90s and shutdown-grace-period-critical-pods=20s. The half that silently defeats the other half: this is enforced by LOGIND, not the kubelet. The kubelet holds an inhibitor DELAY lock and systemd waits at most InhibitDelayMaxSec -- DEFAULT 5 SECONDS -- then proceeds regardless, truncating the drain with no error. The role ships a logind.conf.d drop-in (120) and ASSERTS cap >= grace period, because the failure is invisible. 2. SCALING A StatefulSet TO 0 DELETED ITS PVC AND VOLUME. monitoring/loki carries persistentVolumeClaimRetentionPolicy whenScaled: Delete, so the quiesce destroyed storage-loki-0 outright. Loki stores type: filesystem, so that PVC was the only copy; 72h retention bounded the loss, the mechanism does not. Symptom was the volume count going 25 -> 24, first misread as a miscount. Repaired: scale back to the declared 1 recreated the PVC and loki-0 is 2/2 Running. New infra/maintenance/quiesce.sh (quiesce|resume|status) encodes the sequence that worked and REFUSES to scale any whenScaled: Delete StatefulSet -- it stays running, its volume hard-detaches, and it rebuilds on restart. A rebuild beats a deletion. Discovery is computed from the live cluster, so a newly onboarded service needs no edit here. runbooks/cluster-maintenance-shutdown.md is the operator page: order in both directions, TrueNAS-after-nodes (hard NFS mount wedges kubelet), the un-hibernate-Postgres step Flux will NOT do for you, and a "looks broken but is fine" section separating the 4/19 Flux cascade, the degraded-and-rebuilding volume and the self-recovering keycloak-operator from the one thing that needs a human (a stuck oauth2-proxy replica). THE KUBELET CHANGE HAS NEVER BEEN APPLIED OR TESTED -- it needs a rolling k3s restart, which is an operator action. Review config.yaml's kubelet-arg before applying: a bad config.yaml is how k3s fails to come up. See the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
138 lines
8.8 KiB
YAML
138 lines
8.8 KiB
YAML
---
|
|
# k3s_server role defaults (ADR-0001).
|
|
|
|
# Best-effort version pin (kept near the ci/versions kubectl 1.36.x line). Bump
|
|
# deliberately — a version change is a code change. Empty string => latest stable.
|
|
k3s_version: "v1.36.4+k3s1"
|
|
|
|
k3s_install_url: "https://get.k3s.io"
|
|
k3s_install_script: "/usr/local/bin/k3s-install.sh"
|
|
|
|
# Components disabled at install so Cilium + Gateway API own networking (ADR-0001/0002).
|
|
# flannel + kube-proxy + network-policy are disabled via dedicated flags below;
|
|
# traefik + servicelb via --disable.
|
|
# local-storage: k3s ships local-path as a DEFAULT StorageClass, which collides with Longhorn's
|
|
# default (two default SCs = ambiguous PVC binding). Longhorn (ADR-0003) is the cluster's storage,
|
|
# so disable k3s local-path entirely → `longhorn` is the sole default. (Effective on fresh installs;
|
|
# the k3s install task is idempotent, so this does not restart an existing cluster — see DECISIONS-LOG.)
|
|
k3s_disable: "traefik,servicelb,local-storage"
|
|
|
|
# --- k3s --data-dir (R-ENV-02) ------------------------------------------------------
|
|
# EVERYTHING k3s persists lives under this one directory: the embedded etcd database and
|
|
# WAL (server/db), the node token, the TLS CA and every issued cert, AND the containerd
|
|
# image store (agent/containerd) — which is by far the largest tenant.
|
|
#
|
|
# THE DEFAULT IS k3s'S OWN DEFAULT, restated explicitly rather than left implicit, so the
|
|
# LIVE dev cluster's install command is byte-for-byte what it already was: dev keeps
|
|
# /var/lib/rancher/k3s, where it also has a DEDICATED etcd device mounted at
|
|
# server/db (docs/plans/ssd-etcd-isolation.md). Changing this value on dev would move a
|
|
# running cluster's entire state directory out from under it and orphan that etcd mount.
|
|
#
|
|
# PROD OVERRIDES IT, in inventory/group_vars/k3s_prod/vars.yml — the same seam
|
|
# kube_vip_address / k3s_api_host / node_baseline_backports_kernel use, so a FOURTH prod
|
|
# node inherits it from group membership rather than from anyone remembering. It is set
|
|
# there as the CLUSTER-WIDE `k3s_data_dir` because TWO roles must agree on it: this role
|
|
# turns it into --data-dir, and roles/node_baseline creates the directory on the claimed
|
|
# data partition and derives etcd's location from it (node_baseline_k3s_data_dir defers to
|
|
# this same variable). One line moves both.
|
|
#
|
|
# WHY PROD MOVES IT (infra/PROD-PROVISIONING.md §2, revised 2026-09-04). Prod's /var is a
|
|
# separate partition of 11.8 GiB. Measured on the fully-deployed dev cluster, containerd
|
|
# alone is 21-25 GB per node and /var totals 58-70 GB — so prod would fill /var during its
|
|
# FIRST image pulls, and a full /var means etcd cannot write its WAL. That is cluster death,
|
|
# not degradation. Prod points --data-dir at /data/k3s on the claimed ~424 GB partition.
|
|
#
|
|
# NOT ALSO WRITTEN TO /etc/rancher/k3s/config.yaml, deliberately. --data-dir is consumed at
|
|
# INSTALL time (it is baked into the systemd unit's ExecStart); adding a `data-dir:` line to
|
|
# config.yaml would make that file `changed` on the LIVE dev cluster and fire the rolling
|
|
# Restart handler for a value that is not changing. The install flags are the correct and
|
|
# sufficient place, and prod has no k3s installed yet.
|
|
k3s_data_dir: "/var/lib/rancher/k3s"
|
|
|
|
# Set per-play: 'primary' (runs --cluster-init) or 'secondary' (joins the primary).
|
|
k3s_server_mode: "secondary"
|
|
|
|
# Subject Alternative Names baked into the kube-apiserver serving cert. Must include
|
|
# the kube-vip control-plane VIP (kube_vip_address) so TLS via the VIP validates,
|
|
# every server's real node IP (so the cert is valid whichever node answers behind
|
|
# the VIP), and 127.0.0.1 (local kubeconfig / health probes). k3s regenerates the
|
|
# serving cert on restart when this set changes. Deduplicated (the primary IP is
|
|
# also k3s_api_host).
|
|
#
|
|
# THIS LIST COVERS BOTH API ADDRESSES, WHICH IS WHY THE 2026-09-06 SPLIT NEEDED NO CERT
|
|
# CHANGE AND NO k3s RESTART. group_vars/all.yml now distinguishes k3s_api_host (the
|
|
# founder, for the cold join) from k3s_apiserver_endpoint (= kube_vip_address, what Cilium
|
|
# and every steady-state client dial). Both are already in this set, so repointing a
|
|
# consumer from one to the other cannot produce a TLS failure. VERIFIED LIVE 2026-09-06,
|
|
# not assumed: prod's apiserver answered through https://192.168.2.9:6443 with full CA
|
|
# verification and tls-server-name=192.168.2.9. Do not remove either entry — dropping
|
|
# kube_vip_address breaks every client of the VIP; dropping k3s_api_host breaks nothing
|
|
# today only because it is also in the k3s_servers_group node-IP list below, and that
|
|
# coincidence is not a contract.
|
|
#
|
|
# k3s_servers_group — NOT a literal group name — is what makes this role environment-
|
|
# agnostic (R-ENV-02): group_vars/all.yml sets it to `k3s_servers` (dev) and
|
|
# group_vars/k3s_prod/vars.yml to `k3s_prod`, so each environment SANs its own servers.
|
|
k3s_tls_sans: >-
|
|
{{ ([k3s_api_host, kube_vip_address, '127.0.0.1']
|
|
+ (groups[k3s_servers_group] | map('extract', hostvars, 'k3s_node_ip') | list))
|
|
| unique }}
|
|
|
|
# --- Keycloak OIDC issuer for kubectl SSO (R-IDP-03, R-ENV-02) -----------------------
|
|
# THE ANSIBLE-SIDE HALF OF ${BASE_DOMAIN}, and it cannot be the Flux-side half.
|
|
#
|
|
# The kube-apiserver's `oidc-issuer-url` is a k3s SERVER FLAG written into
|
|
# /etc/rancher/k3s/config.yaml by this role — it is NOT a Kubernetes manifest, so Flux never
|
|
# sees it and `postBuild.substituteFrom: cluster-vars` cannot reach it. The platform hostname
|
|
# suffix therefore exists as ONE FACT IN TWO PLACES, and they must agree:
|
|
#
|
|
# Flux side clusters/base/cluster-vars.yaml BASE_DOMAIN (per-overlay)
|
|
# Ansible side THIS variable k3s_base_domain (per-inventory-group)
|
|
#
|
|
# [parity] k3s_base_domain MUST equal the BASE_DOMAIN the SAME cluster's Flux overlay sets.
|
|
# If they disagree, the apiserver validates ID tokens against a DIFFERENT Keycloak than the
|
|
# one the cluster actually runs: every `kubectl` SSO login fails with an issuer mismatch,
|
|
# while the cert-based admin kubeconfig keeps working — so it looks like "SSO is broken",
|
|
# not like a config skew. The break-glass cert kubeconfig is why this is recoverable.
|
|
#
|
|
# THE DEFAULT IS THE DEV VALUE, restated explicitly rather than left implicit — the same
|
|
# convention as k3s_data_dir above and cluster-vars' GATEWAY_VIP / LONGHORN_DATA_PATH: the
|
|
# base default is the known-good LIVE value, so dev's rendered config.yaml is byte-for-byte
|
|
# what it already is and the Restart handler does NOT fire on a re-run.
|
|
#
|
|
# PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same inventory seam
|
|
# k3s_data_dir / k3s_api_host / kube_vip_address use, so a FOURTH prod server inherits it
|
|
# from group membership rather than from anyone remembering.
|
|
k3s_base_domain: "home.keithjackson.dev"
|
|
|
|
# Derived, so the realm path is stated once. Override k3s_base_domain, not this.
|
|
k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"
|
|
|
|
# --- Graceful node shutdown (kubelet) ----------------------------------------------------------
|
|
# MEASURED 2026-10-17, during the UPS move: graceful node shutdown was NOT configured on any prod
|
|
# node, so `shutdown -h` did NOT drain pods. systemd stops k3s, containerd goes with it, and every
|
|
# pod is SIGKILLed where it stands -- including 9 CloudNativePG instances mid-checkpoint, the UniFi
|
|
# VM, and 25 attached Longhorn volumes. The whole-cluster shutdown that day had to quiesce
|
|
# everything BY HAND first (runbooks/cluster-maintenance-shutdown.md), and the one volume that did
|
|
# not finish detaching came back `degraded` and had to rebuild.
|
|
#
|
|
# With these set, the kubelet takes a systemd inhibitor lock and, on shutdown, evicts ordinary pods
|
|
# first and critical ones last, so a plain `shutdown -h`/reboot drains the node. That makes the
|
|
# manual scale-down in the runbook a belt-and-braces step for a PLANNED whole-cluster shutdown
|
|
# rather than the only thing standing between a reboot and a hard kill.
|
|
#
|
|
# 90s/20s, not larger: the budget is spent on every reboot and node upgrade, and the kubelet waits
|
|
# the FULL ordinary period before it starts on critical pods. Postgres does a fast shutdown in
|
|
# well under 90s here, and Longhorn needs only enough time to flush and detach.
|
|
k3s_kubelet_shutdown_grace_period: "90s"
|
|
k3s_kubelet_shutdown_grace_period_critical_pods: "20s"
|
|
|
|
# *** THE HALF THAT IS EASY TO MISS, AND SILENTLY BREAKS THE OTHER HALF. ***
|
|
# Graceful node shutdown works by the kubelet holding a logind INHIBITOR DELAY lock. systemd caps
|
|
# how long any inhibitor may delay shutdown at `InhibitDelayMaxSec`, which DEFAULTS TO 5 SECONDS.
|
|
# If that cap is below the grace period above, systemd proceeds anyway and the drain is truncated
|
|
# to ~5s -- i.e. the feature appears configured, reports no error, and still hard-kills pods. So
|
|
# this MUST be >= k3s_kubelet_shutdown_grace_period; the role asserts it rather than trusting it.
|
|
# Set a little above the kubelet's total (90s + 20s) to leave headroom.
|
|
k3s_logind_inhibit_delay_max_sec: 120
|