From 44983095e1a1f7275a152f1401fa74fdde19fc8e Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 12 Sep 2026 23:49:19 -0500 Subject: [PATCH] feat(runbooks): make the 42 alert runbook_url annotations resolve MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every platform-health alert rule carries `runbook_url: "https://${BASE_DOMAIN}/runbooks/"` — 42 annotations over 34 distinct names, across platform/monitoring/alerts.yaml, platform/monitoring/pvc-capacity-alerts.yaml, platform/cnpg/alerts.yaml and platform/kubevirt/alerts.yaml. There was no runbooks/ directory and no HTTPRoute serving that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. Every alert that reached a human handed them a dead link at the exact moment the documentation is read, and the annotation made it look solved — the same green-surface-over-absent-mechanism class as the import Job that reported success while storing placeholder secrets. No alerts.yaml is touched and no runbook_url value changes. The existing URL scheme is made to resolve instead. WHAT LANDS - runbooks/<34>.md — one per name, each grounded in this cluster's real topology and, where one exists, its real incident history: the 2026-09-07 k3s-prod-1 etcd I/O starvation (measured PSI 11.36 vs 0.75/1.29, the exact journal lines, the four mechanisms proven inert), the Longhorn reboot tax and the freeze/thaw of concurrent-replica-rebuild-per-node-limit, Cilium's once-resolved egress interface cache (Ifindex 3 against a live 1780) and the absence of OSS failover, the CNPG 38-hour ContinuousArchiving=False with its measured 6.8 MiB/h disk-full clock and the 2026-09-12 aggregation-scope fix, the 2026-09-06 Keycloak lockout in which all 12 clients held literal placeholder secrets behind an entirely green cluster, the Velero R2 x-amz-tagging NotImplemented incompatibility and its v1.14.3-rc.2 pin, and swap re-enabling itself across reboots via systemd-gpt-auto-generator. Each page states what the alert means HERE, the first diagnostic commands, likely causes ordered by probability with a remediation each, and when to escalate or stop. - Every page uses FULLY-QUALIFIED resource names and says why: three CRDs register the kind `Backup` here, the bare short name resolves to Longhorn's (zero objects), so `kubectl get backup -A` prints "No resources found" on a cluster holding 6 Velero and 27 CNPG backups. Acting on that false negative is how a CLI quirk becomes data loss. - Every page states that kubectl only works via `infra/devcontainer/dev.sh run bash -c '...'` with an explicit `--context k3s-prod`, because dev.sh defaults its current-context to k3s-dev. - Pages also record which alert groups are INERT today (measured 2026-09-13: no Longhorn, cert-manager, Flux-controller or Hubble scrape job exists, so 13 rules evaluate against nothing) so nobody reads their silence as health. SERVING — runbooks/serving/ plus clusters/base/runbooks.yaml A two-replica nginx-unprivileged serving a ConfigMap generated from runbooks/*.md, at the apex `/runbooks` prefix, in its own non-gating post-platform Flux tier (dependsOn platform-config, wait:false) — the platform/homepage pattern, file for file. - The service catalog cannot express this route: ingressRoutes[].host is pattern-bound to a single DNS label plus the apex, so the bare apex is unrepresentable, and there is no path field. - Rejected: redirecting to the Gitea web UI. A runbook is read when the platform is broken, and Gitea is an app-tier workload whose own dependencies several of these runbooks document; Gateway API's RequestRedirect also has no "append .md" modifier, so it would need 34 hand-maintained ReplaceFullPath rules. - configMapGenerator maps each file to an EXTENSIONLESS key (node-down=node-down.md), so /runbooks/node-down is a plain static lookup with no rewrite rule. Served as text/plain: the namespace has DNS-only egress, so a CDN markdown renderer could not load anyway. - The tier declares NO postBuild.substituteFrom, deliberately: envsubst would run over the prose and silently empty every ${...} shell example (it fails open — the 2026-09-01 emptied Cilium k8sServiceHost). The HTTPRoute omits `hostnames:` and inherits the https-apex listener's, so nothing needs substituting. AUTH — PUBLIC, deliberately auth-mode=none. Gating these pages behind oauth2-proxy/Keycloak would make runbooks/keycloak-down unreadable when Keycloak is down and runbooks/keycloak-client-secrets unreadable during the exact 2026-09-06 failure mode. An operator who cannot log in is this route's primary reader. Content is operational procedure only, no secrets. The rationale is recorded in an annotation on the HTTPRoute so the route is left OUT of the Gateway ext-authz attachment when forward-auth enforcement lands. PROVEN — in-cluster probe through the Gateway on prod /runbooks/node-down HTTP 200 8613 bytes text/plain; charset=utf-8 /runbooks/keycloak-down HTTP 200 9378 bytes text/plain; charset=utf-8 /runbooks/cnpg-archiving HTTP 200 11729 bytes text/plain; charset=utf-8 /runbooks/longhorn-faulted HTTP 200 9135 bytes text/plain; charset=utf-8 /runbooks/vpn-down HTTP 200 11092 bytes text/plain; charset=utf-8 /runbooks/backup-failure HTTP 200 9604 bytes text/plain; charset=utf-8 HTTPRoute Accepted by io.cilium/gateway-controller on sectionName https-apex; the `/runbooks` prefix wins over homepage's apex `/` by Gateway API longest-path precedence. GUARD — conformance/tests/l0-static/alert-runbooks-resolvable (R-OBS-04) 363 offline assertions. It checks BOTH halves, because the regression is silent in both directions: an alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its configMapGenerator entry is in git, reviewed, merged, and still 404s in the cluster. Also asserts a quality floor so a stub cannot make the link look fixed, and that route/listener/nginx prefix agree with the annotations. Proven to fail, then reverted: a bogus runbook_url (2 FAIL), a runbook removed from the generator (1 FAIL), and a 31-byte stub (7 FAIL). Co-Authored-By: Claude Opus 5 --- ci/l0.sh | 38 ++ clusters/base/kustomization.yaml | 7 + clusters/base/runbooks.yaml | 57 +++ .../chainsaw-test.yaml | 76 ++++ .../alert-runbooks-resolvable/check.sh | 390 ++++++++++++++++++ runbooks/README.md | 135 ++++++ runbooks/backup-failure.md | 169 ++++++++ runbooks/backup-missed.md | 157 +++++++ runbooks/backup-partial.md | 160 +++++++ runbooks/cdi-import-stuck.md | 168 ++++++++ runbooks/cert-expiry.md | 142 +++++++ runbooks/cert-issuance.md | 161 ++++++++ runbooks/cnpg-archiving.md | 194 +++++++++ runbooks/cnpg-backup-missed.md | 159 +++++++ runbooks/cnpg-exporter.md | 164 ++++++++ runbooks/cnpg-wal-growth.md | 157 +++++++ runbooks/denied-flows.md | 150 +++++++ runbooks/flux-reconcile.md | 160 +++++++ runbooks/flux-suspended.md | 130 ++++++ runbooks/image-automation.md | 125 ++++++ runbooks/image-policy-stalled.md | 133 ++++++ runbooks/keycloak-client-secrets.md | 197 +++++++++ runbooks/keycloak-db.md | 165 ++++++++ runbooks/keycloak-down.md | 162 ++++++++ runbooks/kubevirt-control-plane.md | 152 +++++++ runbooks/kubevirt-crashloop.md | 159 +++++++ runbooks/kubevirt-launch-failed.md | 169 ++++++++ runbooks/kustomization.yaml | 85 ++++ runbooks/longhorn-capacity.md | 154 +++++++ runbooks/longhorn-degraded.md | 166 ++++++++ runbooks/longhorn-faulted.md | 160 +++++++ runbooks/monitoring-capacity.md | 129 ++++++ runbooks/node-cpu.md | 121 ++++++ runbooks/node-disk.md | 127 ++++++ runbooks/node-down.md | 152 +++++++ runbooks/node-io.md | 179 ++++++++ runbooks/node-memory.md | 119 ++++++ runbooks/pvc-capacity.md | 140 +++++++ runbooks/serving/deployment.yaml | 151 +++++++ runbooks/serving/httproute.yaml | 77 ++++ runbooks/serving/namespace.yaml | 16 + runbooks/serving/networkpolicy.yaml | 57 +++ runbooks/serving/nginx-config.yaml | 81 ++++ runbooks/serving/service.yaml | 19 + runbooks/targets-down.md | 134 ++++++ runbooks/update-rollback.md | 146 +++++++ runbooks/vpn-down.md | 183 ++++++++ 47 files changed, 6432 insertions(+) create mode 100644 clusters/base/runbooks.yaml create mode 100644 conformance/tests/l0-static/alert-runbooks-resolvable/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/alert-runbooks-resolvable/check.sh create mode 100644 runbooks/README.md create mode 100644 runbooks/backup-failure.md create mode 100644 runbooks/backup-missed.md create mode 100644 runbooks/backup-partial.md create mode 100644 runbooks/cdi-import-stuck.md create mode 100644 runbooks/cert-expiry.md create mode 100644 runbooks/cert-issuance.md create mode 100644 runbooks/cnpg-archiving.md create mode 100644 runbooks/cnpg-backup-missed.md create mode 100644 runbooks/cnpg-exporter.md create mode 100644 runbooks/cnpg-wal-growth.md create mode 100644 runbooks/denied-flows.md create mode 100644 runbooks/flux-reconcile.md create mode 100644 runbooks/flux-suspended.md create mode 100644 runbooks/image-automation.md create mode 100644 runbooks/image-policy-stalled.md create mode 100644 runbooks/keycloak-client-secrets.md create mode 100644 runbooks/keycloak-db.md create mode 100644 runbooks/keycloak-down.md create mode 100644 runbooks/kubevirt-control-plane.md create mode 100644 runbooks/kubevirt-crashloop.md create mode 100644 runbooks/kubevirt-launch-failed.md create mode 100644 runbooks/kustomization.yaml create mode 100644 runbooks/longhorn-capacity.md create mode 100644 runbooks/longhorn-degraded.md create mode 100644 runbooks/longhorn-faulted.md create mode 100644 runbooks/monitoring-capacity.md create mode 100644 runbooks/node-cpu.md create mode 100644 runbooks/node-disk.md create mode 100644 runbooks/node-down.md create mode 100644 runbooks/node-io.md create mode 100644 runbooks/node-memory.md create mode 100644 runbooks/pvc-capacity.md create mode 100644 runbooks/serving/deployment.yaml create mode 100644 runbooks/serving/httproute.yaml create mode 100644 runbooks/serving/namespace.yaml create mode 100644 runbooks/serving/networkpolicy.yaml create mode 100644 runbooks/serving/nginx-config.yaml create mode 100644 runbooks/serving/service.yaml create mode 100644 runbooks/targets-down.md create mode 100644 runbooks/update-rollback.md create mode 100644 runbooks/vpn-down.md diff --git a/ci/l0.sh b/ci/l0.sh index a298a59..e824cb0 100755 --- a/ci/l0.sh +++ b/ci/l0.sh @@ -136,6 +136,44 @@ else emit_offline_summary fi +# ---- Stage 3f: every alert runbook_url resolves to a served runbook (UNCONDITIONAL) ---- +# Every platform-health alert rule carries `runbook_url: "https://${BASE_DOMAIN}/runbooks/"` — +# 42 annotations over 34 names. For three roadmap phases there was no runbooks/ directory and no +# HTTPRoute serving that path, so a live in-cluster probe of .../runbooks/node-down returned HTTP 404: +# every alert that reached a human handed them a dead link at the one moment the documentation is read, +# while the annotation made it look solved. The check asserts the whole chain — the URL's shape, the +# runbook's existence, a quality floor that a stub cannot clear, the configMapGenerator entry that +# actually projects it into the server, and the route/listener/nginx prefix agreeing with the +# annotations — because "the file exists in git" and "the URL resolves in the cluster" are different +# claims and only the second one helps an operator. +# +# WHY ITS OWN UNCONDITIONAL STAGE, same reasoning as 3c/3d/3e: Stage 3b scopes the conformance run to +# the TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh maps platform/monitoring/* to +# "R-OBS-01 R-OBS-02" — NOT R-OBS-04 — and maps platform/kubevirt/* and runbooks/* to nothing at all. +# So the single most likely breaking change (adding an alert with a runbook_url, or adding a runbook +# without its generator entry) attributes a requirement set that would FILTER THIS CHECK OUT of exactly +# the change that introduces the bug. The invariant spans platform/, runbooks/ and clusters/, so it runs +# on EVERY gate invocation. +RUNBOOK_CHECK="${RUNBOOK_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/alert-runbooks-resolvable/check.sh}" +log "Stage: alert runbook links resolve (${RUNBOOK_CHECK})" +if [ ! -f "$RUNBOOK_CHECK" ]; then + skip "alert runbook check not present at '${RUNBOOK_CHECK}' — UNWIRED, not green" + summary "- alert-runbooks: **SKIPPED** (check missing)" + if [ "$rc" -eq 0 ]; then rc=3; fi +else + rb_rc=0 + bash "$RUNBOOK_CHECK" || rb_rc=$? + case "$rb_rc" in + 0) ok "Alert runbook links resolve" + summary "- alert-runbooks: **passed**" ;; + 3) skip "alert runbook check SKIPPED (no PrometheusRule present) — not a pass" + summary "- alert-runbooks: **SKIPPED** (nothing to audit)" + if [ "$rc" -eq 0 ]; then rc=3; fi ;; + *) err "Alert runbook links FAILED — an alert links to a runbook that does not exist or is not served" + summary "- alert-runbooks: **FAILED**"; rc=1 ;; + esac +fi + case "$rc" in 0) ok "L0 stage complete" ;; 3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;; diff --git a/clusters/base/kustomization.yaml b/clusters/base/kustomization.yaml index 6f56415..62c609a 100644 --- a/clusters/base/kustomization.yaml +++ b/clusters/base/kustomization.yaml @@ -50,6 +50,13 @@ resources: # reverse-proxy sidecar (Option B). POST-platform tier (dependsOn platform-config, wait:false); same # non-gating leaf-UI rationale as headlamp/homepage/oauth2-proxy (a UI must never gate the platform). - crafty.yaml # Flux Kustomization -> ./platform/crafty (dependsOn platform-config) + # Alert runbooks — POST-platform tier (dependsOn platform-config, wait:false). Serves the 34 + # runbooks the alert rules' `runbook_url` annotations point at, at /runbooks/; before + # this tier existed that path was a 404 and every alert handed the operator a dead link. Same + # non-gating leaf rationale as headlamp/homepage/crafty. Carries NO postBuild substitution on + # purpose — it renders documentation containing `${...}` shell examples that envsubst would empty; + # see clusters/base/runbooks.yaml. + - runbooks.yaml # Flux Kustomization -> ./runbooks (dependsOn platform-config) # Velero backup controller — POST-platform tier (dependsOn platform-config, wait:false). A backup # controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy. # Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name). diff --git a/clusters/base/runbooks.yaml b/clusters/base/runbooks.yaml new file mode 100644 index 0000000..06e33ac --- /dev/null +++ b/clusters/base/runbooks.yaml @@ -0,0 +1,57 @@ +--- +# Flux Kustomization: alert runbooks — reconciles ./runbooks. +# +# WHAT THIS SERVES. Every alert rule in this repo carries +# `runbook_url: "https://${BASE_DOMAIN}/runbooks/"` — 42 annotations over 34 distinct names. +# Until this tier existed that path returned 404 on the live cluster, so every alert that reached a +# human handed them a dead link at the moment they needed help. This Kustomization renders the +# markdown under ./runbooks into a ConfigMap and a two-replica nginx that serves it at that exact +# prefix. See runbooks/README.md and runbooks/serving/httproute.yaml. +# +# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false) — the same placement and +# the same reasoning as homepage/headlamp/oauth2-proxy. It needs the wildcard TLS cert, the shared +# Gateway (including the `https-apex` listener) and split-horizon DNS, all of which arrive with +# `platform` + `platform-config`; and a leaf web server must never be able to stall the platform +# chain, which is the whole lesson of DECISIONS-LOG's "Headlamp gating stall". wait:false guarantees +# its health can never gate anything even if something is later mistakenly pointed dependsOn it. +# +# NO postBuild.substituteFrom — THE ONE DELIBERATE DIVERGENCE FROM EVERY OTHER TIER, AND THE MOST +# IMPORTANT LINE IN THIS FILE. +# This tier renders the runbook PROSE into a ConfigMap, and runbooks are full of shell: `$POD`, +# `$(kubectl ...)`, and — unavoidably, since they document these very annotations — the literal text +# `${BASE_DOMAIN}`. Flux's postBuild substitution runs envsubst over the ENTIRE built stream, and it +# FAILS OPEN: an undefined `${X}` becomes the empty string while the Kustomization still reports +# Ready (DECISIONS-LOG 2026-09-01 — that is how `k8sServiceHost` was emptied and Cilium broke on 2 +# of 3 dev nodes). Enabled here it would silently mutilate the documentation: every braced expansion +# in every command example blanked, in a file nobody re-reads after writing. Worse, the L0 check +# `conformance/tests/l0-static/flux-postbuild-substitution` scans only files reachable through +# kustomization `resources:`, not `configMapGenerator` inputs, so the damage would be invisible to +# the gate as well. +# The fix is to have nothing to substitute. The only knob this tier would have wanted is the apex +# hostname, and runbooks/serving/httproute.yaml avoids needing it by declaring NO `hostnames:` and +# inheriting the `https-apex` listener's own hostname instead. So this tier is +# environment-independent by construction and R-ENV-02 ("authored once, promoted dev -> prod") holds +# without substitution rather than because of it. +# IF A FUTURE KNOB EVER GENUINELY NEEDS SUBSTITUTION HERE: do not add substituteFrom to this +# Kustomization. Split the serving manifests into their own substituted tier and leave the content +# ConfigMap in an unsubstituted one, or the runbooks lose their shell examples. +# +# NO SOPS decryption stanza: this tier carries no encrypted Secret (markdown plus five plain +# manifests), so — like homepage — neither clusters/dev nor clusters/prod patches it. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: runbooks + namespace: flux-system +spec: + interval: 10m + retryInterval: 2m + timeout: 5m + prune: true + wait: false + dependsOn: + - name: platform-config # wildcard TLS + Gateway (incl. the https-apex listener) first + sourceRef: + kind: GitRepository + name: flux-system + path: ./runbooks diff --git a/conformance/tests/l0-static/alert-runbooks-resolvable/chainsaw-test.yaml b/conformance/tests/l0-static/alert-runbooks-resolvable/chainsaw-test.yaml new file mode 100644 index 0000000..18fed29 --- /dev/null +++ b/conformance/tests/l0-static/alert-runbooks-resolvable/chainsaw-test.yaml @@ -0,0 +1,76 @@ +# ============================================================================= +# L0 STATIC — every alert's runbook_url resolves to a runbook that is actually served +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# +# Every platform-health alert rule in this repo carries +# `runbook_url: "https://${BASE_DOMAIN}/runbooks/"` — 42 annotations across four files, over 34 +# distinct names. For the whole of phases 4-6 there was NO runbooks/ directory and NO HTTPRoute serving +# that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. +# So an alert reaching a human handed them a dead link at the one moment documentation is read, while the +# annotation made the problem look solved — the same green-surface-over-absent-mechanism class as the +# import Job that reported success while storing placeholder secrets, and the guard whose glob silently +# stopped matching. +# +# The link is served now (runbooks/, clusters/base/runbooks.yaml). This test is the guard that keeps it +# true, and it asserts BOTH halves of the claim because the regression is silent in both directions: an +# alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its +# configMapGenerator entry is in git, reviewed, merged — and still 404s in the cluster. "The file exists" +# and "the URL resolves" are different claims; only the second one helps an operator. +# +# GIVEN every `runbook_url` annotation on every PrometheusRule in the repo, +# WHEN each is parsed out of git and resolved against runbooks/, +# THEN (1) each URL has EXACTLY the shape the server resolves — https://${BASE_DOMAIN}/runbooks/, +# the ${BASE_DOMAIN} token (never a literal hostname), no `.md`, no trailing slash, no query; +# (2) every referenced has a runbooks/.md in git; +# (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the +# dev-container kubectl invocation with --context k3s-prod — so a stub cannot satisfy (2) and +# make the link look fixed while still helping nobody; +# (4) every runbook is PROJECTED INTO THE SERVER by runbooks/kustomization.yaml's +# configMapGenerator under the extensionless key the URL asks for; +# (5) route, listener and server agree with the annotations (PathPrefix == the annotations' prefix, +# a sectionName that platform/gateway/gateway.yaml actually defines, the namespace opted in to +# the Gateway, and an nginx location for that same prefix with a text/plain default_type); +# (6) the Flux tier is wired into clusters/base AND declares NO postBuild.substituteFrom — +# substitution would run envsubst over the runbook prose and silently empty every `${...}` +# shell example in it (it fails open: DECISIONS-LOG 2026-09-01, the emptied Cilium +# k8sServiceHost), and the flux-postbuild-substitution check cannot see it because that check +# scans files reachable through `resources:`, not configMapGenerator inputs; +# AND (7) the check is LOAD-BEARING: ci/l0.sh runs it on every gate run (Stage 3f) and ci/gate.sh sets +# the merge block when the l0 stage fails. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library only); +# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test wraps the SAME +# script. +# +# NOT proven (named in check.sh's header): this does not FETCH the URL. That it returns HTTP 200 with +# real content was proven by an in-cluster probe recorded in the commit message; a live probe needs a +# cluster, which L0 by definition does not have. What is asserted here is every static precondition for +# that 200. It also does not judge whether a runbook is correct or current — (3) is a floor, not a +# review — and it does not prove Alertmanager delivers the annotation to a human (that is the L3 +# `alerting` test). R-OBS-04 is tagged for the narrow slice only: the platform's own health alerts carry +# runbook links that resolve to real, served procedure rather than to a 404. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: alert-runbooks-resolvable + labels: + conformance.home.k3s/R-OBS-04: "true" + annotations: + conformance.home.k3s/requirements: "R-OBS-04" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Every runbook_url on every alert rule resolves to a runbook that exists in git, clears a quality + floor, and is actually projected into the server that answers that path — so an alert can never + hand an operator a dead link. +spec: + steps: + - name: audit-every-alert-runbook-link + description: >- + Parse every runbook_url out of every PrometheusRule and assert the runbook exists, is real, is + served at that exact path, and that the serving tier is wired and unsubstituted. + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/alert-runbooks-resolvable/check.sh b/conformance/tests/l0-static/alert-runbooks-resolvable/check.sh new file mode 100755 index 0000000..48a0cdb --- /dev/null +++ b/conformance/tests/l0-static/alert-runbooks-resolvable/check.sh @@ -0,0 +1,390 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — alert-runbooks-resolvable +# Serves: R-OBS-04 ("The platform alerts on its own health (nodes, storage, GitOps, certs, +# VPN)." — spec docs/10-observability-and-alerting.md) +# +# WHY THIS EXISTS — A DEAD LINK IN AN ALERT IS WORSE THAN NO LINK +# -------------------------------------------------------------- +# Every platform-health alert rule in this repo carries +# runbook_url: "https://${BASE_DOMAIN}/runbooks/" +# 42 such annotations across four files, over 34 distinct names. For the whole of phases 4-6 +# there was NO runbooks/ directory and NO HTTPRoute serving that path: a live in-cluster probe +# of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. So an alert reaching a +# human at 3am handed them a link that fails, at the one moment the documentation is read. The +# annotation LOOKED like the problem was solved, which is the specific failure this repo keeps +# finding (an import Job that reports success while storing placeholder secrets; a guard whose +# glob silently stopped matching) — a green-looking surface over an absent mechanism. +# +# The link is now served (runbooks/, clusters/base/runbooks.yaml). This check is the guard that +# keeps it true, because the regression is SILENT IN BOTH DIRECTIONS and nothing else catches it: +# * add an alert with a new runbook_url and forget the runbook -> that alert links to a 404, +# and no test, lint, render or policy notices; +# * write runbooks/.md but forget to add it to the ConfigMap generator -> the file is in +# git, reviewable, apparently done, and the URL still 404s in the cluster. "The file exists" +# and "the URL resolves" are different claims and only the second one helps an operator. +# Both halves are asserted below, which is why this check reaches past the markdown into the +# kustomization, the HTTPRoute, the nginx config and the Flux tier. +# +# GIVEN the repository as authored, +# WHEN every `runbook_url` annotation on every PrometheusRule is parsed out of git, +# THEN (1) each URL has EXACTLY the shape the server resolves — +# https://${BASE_DOMAIN}/runbooks/, `${BASE_DOMAIN}` as the token (never a literal +# hostname: a literal is how prod was authored to serve dev's hostnames, see the +# ServiceDefinition schema's note), no `.md` suffix, no trailing slash, no query; +# (2) every referenced has a runbooks/.md in git; +# (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the +# dev-container kubectl invocation — so a one-line stub cannot satisfy (2) and make the +# link look fixed while still helping nobody (the failure mode this check is most likely +# to be defeated by is a placeholder file, not a missing one); +# (4) every runbook markdown file is actually PROJECTED INTO THE SERVER by +# runbooks/kustomization.yaml's configMapGenerator, under the EXTENSIONLESS key the URL +# asks for — the "exists in git but 404s in the cluster" half; +# (5) the route and the server agree with the annotations: the HTTPRoute's PathPrefix is the +# prefix the URLs use, it attaches to a Gateway listener that actually exists, its +# namespace is opted in to that Gateway, and the nginx config answers that same prefix; +# (6) the Flux tier that reconciles all of it is wired into clusters/base AND declares NO +# postBuild.substituteFrom — substitution here would run envsubst over the runbook prose +# and silently empty every `${...}` shell example in it (it fails open: DECISIONS-LOG +# 2026-09-01, the emptied Cilium k8sServiceHost); +# AND (7) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and ci/gate.sh sets the +# merge block when the l0 stage fails. +# +# WHY (7) IS AN UNCONDITIONAL ci/l0.sh STAGE and not left to the requirement-scoped run: Stage 3b +# scopes `conformance/run.sh --layer l0` to the TOUCHED requirements (R-TEST-03), and +# ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" — NOT R-OBS-04 — and +# maps platform/kubevirt/* and runbooks/* to nothing at all. So the single most likely breaking +# change (adding an alert with a runbook_url to platform/monitoring/alerts.yaml, or adding a +# runbook and forgetting the generator entry) attributes a requirement set that would FILTER THIS +# CHECK OUT. Same reasoning as stages 3c/3d/3e; the invariant is whole-repo, so it runs every time. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check): +# - It does not fetch the URL. That it returns HTTP 200 with real content was proven by an +# in-cluster probe (recorded in the commit message and DECISIONS-LOG); a live probe needs a +# cluster, which L0 by definition does not have. What is asserted here is every static +# precondition for that 200: the file, its projection, the key name, the route, the prefix. +# - It does not judge whether a runbook is CORRECT or current. (3) is a floor, not a review: it +# can tell a stub from a runbook, not a good runbook from a stale one. Nothing automatable can. +# - It does not prove Alertmanager actually delivers the annotation to a human. That is the L3 +# `alerting` test. +# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06) and is not consulted. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +# Honest SKIP (never green) if there are no alert rules at all — nothing to audit yet. +if ! grep -rl 'kind: PrometheusRule' "${ROOT}/platform" >/dev/null 2>&1; then + echo "check SKIPPED: no PrometheusRule found under platform/ — no alert rules to audit" >&2 + exit 3 +fi + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +"""Static audit: every alert runbook_url resolves to a runbook that is actually served. + +Pure standard library, no PyYAML — the same constraint and the same style as the neighbouring +L0 checks (lib/coverage.py, lib/status.py). Every parse contract is documented at its parser; +all of them are line-constrained shapes in hand-authored manifests. +""" +from __future__ import annotations + +import re +import sys +from pathlib import Path + +ROOT = Path(sys.argv[1]) +sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree + +FAILURES: list[str] = [] +ADVISORIES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def read(p: Path) -> str: + try: + return p.read_text() + except (OSError, UnicodeDecodeError): + return "" + + +# =============================================================== (1) parse the annotations +# Parse contract: `runbook_url:` is an annotation on a Prometheus alerting rule, authored one per +# line as `runbook_url: ""` (single or double quoted, or bare). Only files that declare +# `kind: PrometheusRule` are scanned, so a runbook_url quoted inside this check or inside prose +# elsewhere in the repo is not mistaken for an alert annotation. Commented-out lines are skipped: +# platform/monitoring/alerts.yaml carries a `# runbook_url points at a docs base` header comment. +RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""") + +rule_files = sorted( + p for p in ROOT.rglob("*.yaml") + if ".git" not in p.parts and "kind: PrometheusRule" in read(p) +) +check(len(rule_files) > 0, + "no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; " + "this check would silently audit nothing") + +# url -> [ ":", ... ] +refs: dict[str, list[str]] = {} +for f in rule_files: + rel = f.relative_to(ROOT) + for lineno, line in enumerate(read(f).splitlines(), 1): + if line.lstrip().startswith("#"): + continue + m = RUNBOOK_URL_RE.match(line) + if m: + refs.setdefault(m.group(1), []).append(f"{rel}:{lineno}") + +n_annotations = sum(len(v) for v in refs.values()) +check(n_annotations > 0, + "no runbook_url annotation parsed out of any PrometheusRule — the parser contract broke; " + "this check would pass vacuously while every alert link rots") + +# ---- URL SHAPE. The server resolves exactly one shape. Anything else is a link that 404s even +# though a file with a plausible name exists, so shape is a failure, not a style note. +URL_RE = re.compile(r"^https://\$\{BASE_DOMAIN\}/(runbooks)/([a-z0-9][a-z0-9-]*)$") +names: dict[str, list[str]] = {} # runbook name -> annotation sites +prefixes: set[str] = set() +for url, sites in sorted(refs.items()): + m = URL_RE.match(url) + check(m is not None, + f"runbook_url '{url}' ({', '.join(sites)}) is not the shape this platform serves. " + f"Required: https://${{BASE_DOMAIN}}/runbooks/ — the ${{BASE_DOMAIN}} TOKEN (never a " + f"literal hostname: a literal is how prod got authored to serve dev's public names), a " + f"lowercase-hyphen , and NO '.md' suffix, trailing slash or query. The runbook " + f"server maps the URL path straight onto an extensionless ConfigMap key, so any other " + f"shape is a 404 no matter which files exist") + if m: + prefixes.add(m.group(1)) + names.setdefault(m.group(2), []).extend(sites) + +check(len(prefixes) <= 1, + f"alert annotations use more than one runbook path prefix ({sorted(prefixes)}); the single " + f"HTTPRoute + nginx location can only serve one, so the others are dead links") +PREFIX = (sorted(prefixes) or ["runbooks"])[0] + +# =================================================== (2)(3) the runbooks exist and are real +RUNBOOK_DIR = ROOT / PREFIX +check(RUNBOOK_DIR.is_dir(), + f"{PREFIX}/ is not a directory in this repo, so every one of the {n_annotations} " + f"runbook_url annotations is a dead link") + +# The quality floor. (2) alone is satisfiable by `touch runbooks/node-down.md`, which makes the +# link resolve and still abandons the operator — a worse outcome than the 404, because the 404 at +# least told the truth. These four sections are the operator's path through an incident: what the +# alert means HERE, what to type first, what is probably wrong, and when to stop. `dev.sh` + +# `--context k3s-prod` are required because kubectl does not work on the workstation at all +# (the toolchain lives only in the pinned dev container) and because dev.sh defaults its +# current-context to k3s-DEV — a runbook that omits the context teaches the reader to answer +# questions about the wrong cluster during an incident on this one. +REQUIRED_SECTIONS = ( + "## What this means here", + "## First commands", + "## Likely causes, most likely first", + "## When to escalate or stop", +) +MIN_BYTES = 1500 + +for name, sites in sorted(names.items()): + md = RUNBOOK_DIR / f"{name}.md" + if not md.is_file(): + check(False, + f"runbook '{name}' does not exist: {md.relative_to(ROOT)} is missing, but " + f"{len(sites)} alert annotation(s) link to it ({', '.join(sites[:3])}" + f"{', ...' if len(sites) > 3 else ''}). An alert firing now hands the operator a 404. " + f"Write {PREFIX}/{name}.md (see {PREFIX}/README.md for the required sections)") + continue + body = read(md) + check(len(body) >= MIN_BYTES, + f"{md.relative_to(ROOT)} is {len(body)} bytes, under the {MIN_BYTES}-byte floor — a stub " + f"that makes the link resolve while still helping nobody is worse than the 404 it " + f"replaced, because it looks fixed") + for sec in REQUIRED_SECTIONS: + check(sec in body, + f"{md.relative_to(ROOT)} has no '{sec}' section — the four operator sections are what " + f"make this a runbook rather than a restatement of the alert name") + check("dev.sh" in body, + f"{md.relative_to(ROOT)} never mentions dev.sh — kubectl does not exist on the " + f"workstation; every command in a runbook here must be runnable as " + f"`infra/devcontainer/dev.sh run bash -c '...'` and the runbook must say so") + check("--context k3s-prod" in body, + f"{md.relative_to(ROOT)} never passes --context k3s-prod — dev.sh merges both clusters " + f"and defaults to k3s-DEV, so a context-less command silently answers about the wrong " + f"cluster in the middle of an incident") + +# ADVISORY, not a failure: a runbook nothing links to yet. Writing the runbook BEFORE the alert is +# the right order, and the repo's precedent for "authored but unreferenced" is advisory (see the +# dead-knob advisory in flux-postbuild-substitution/check.sh). +served_extra = {"README"} +if RUNBOOK_DIR.is_dir(): + for md in sorted(RUNBOOK_DIR.glob("*.md")): + stem = md.stem + if stem in served_extra or stem in names: + continue + ADVISORIES.append(f"{md.relative_to(ROOT)} is a runbook no alert links to (written ahead of " + f"its alert, or the alert was removed — advisory, not a failure)") + +# ================================= (4) every runbook is PROJECTED INTO THE SERVER, not just in git +# THIS IS THE HALF A FILE-EXISTENCE CHECK WOULD MISS. The pod serves a ConfigMap, not the git tree. +# Parse contract: runbooks/kustomization.yaml declares one configMapGenerator whose `files:` entries +# are `- =.md`, one per line. The key is EXTENSIONLESS on purpose (the URL carries no +# `.md`, and the ConfigMap volume mount turns each key into a file the nginx root resolves directly). +KUST = RUNBOOK_DIR / "kustomization.yaml" +check(KUST.is_file(), f"{PREFIX}/kustomization.yaml is missing — the Flux tier renders nothing and " + f"every runbook 404s no matter what is in git") +GEN_FILE_RE = re.compile(r"^\s*-\s*([A-Za-z0-9._-]+)=([A-Za-z0-9._/-]+\.md)\s*(?:#.*)?$") +generated: dict[str, str] = {} +kust_text = read(KUST) +if kust_text: + check("configMapGenerator:" in kust_text, + f"{PREFIX}/kustomization.yaml has no configMapGenerator — the runbook markdown is not " + f"rendered into the ConfigMap the server mounts") + for line in kust_text.splitlines(): + if line.lstrip().startswith("#"): + continue + m = GEN_FILE_RE.match(line) + if m: + generated[m.group(1)] = m.group(2) + check(len(generated) > 0, + f"{PREFIX}/kustomization.yaml declares no `=.md` generator entries — the " + f"parser contract broke, or the markdown is being added with implicit keys (which would " + f"carry the '.md' suffix the alert URLs do NOT have, so every link would 404)") + +for name in sorted(names): + check(generated.get(name) == f"{name}.md", + f"runbook '{name}' is not projected into the server: {PREFIX}/kustomization.yaml has no " + f"configMapGenerator entry `- {name}={name}.md`. The file can be in git, reviewed and " + f"merged and the URL still returns 404 — 'the file exists' and 'the URL resolves' are " + f"different claims") + +# And the reverse, so a generator entry cannot point at a file that is not there. +for key, fname in sorted(generated.items()): + check((RUNBOOK_DIR / fname).is_file(), + f"{PREFIX}/kustomization.yaml generates key '{key}' from {PREFIX}/{fname}, which does not " + f"exist — `kustomize build` fails, so the WHOLE tier stops reconciling and every runbook " + f"goes stale at once") + +# ============================== (5) the route, the listener and the server agree with the URLs +ROUTE = RUNBOOK_DIR / "serving" / "httproute.yaml" +NGINX = RUNBOOK_DIR / "serving" / "nginx-config.yaml" +NS = RUNBOOK_DIR / "serving" / "namespace.yaml" +GATEWAY = ROOT / "platform" / "gateway" / "gateway.yaml" + +route_text = read(ROUTE) +check(bool(route_text), f"{PREFIX}/serving/httproute.yaml is missing — nothing routes the " + f"/{PREFIX} prefix to the server") +# Parse contract: a single HTTPRoute with one PathPrefix match and one parentRefs entry naming a +# listener via sectionName; hand-authored, one key per line. +m_path = re.search(r"^\s*type:\s*PathPrefix\s*$\n\s*value:\s*(\S+)\s*$", route_text, re.M) +check(m_path is not None and m_path.group(1) == f"/{PREFIX}", + f"{PREFIX}/serving/httproute.yaml does not match PathPrefix /{PREFIX} " + f"(found {m_path.group(1) if m_path else 'no PathPrefix match'}). The alert annotations point " + f"at /{PREFIX}/; if the route's prefix drifts from the annotations the links 404 while " + f"both files look individually correct") +m_sec = re.search(r"^\s*sectionName:\s*(\S+)", route_text, re.M) +check(m_sec is not None, + f"{PREFIX}/serving/httproute.yaml declares no parentRefs sectionName. The apex host needs the " + f"dedicated apex listener: the wildcard `*.${{BASE_DOMAIN}}` listener does NOT match the bare " + f"apex, so a route without sectionName binds nothing and serves nothing") +if m_sec: + listener = m_sec.group(1) + gw = read(GATEWAY) + check(re.search(rf"^\s*-\s*name:\s*{re.escape(listener)}\s*$", gw, re.M) is not None, + f"{PREFIX}/serving/httproute.yaml attaches to Gateway listener '{listener}', which " + f"platform/gateway/gateway.yaml does not define — the route would report NotAllowed and " + f"never serve") +# An HTTPRoute with no `hostnames:` inherits the listener's hostname, which is how this tier avoids +# needing Flux substitution (see (6)). Either form is routable; a LITERAL hostname is not acceptable, +# for the same reason the schema forbids it in a ServiceDefinition. +m_host = re.search(r"^\s*hostnames:\s*$\n\s*-\s*(\S+)", route_text, re.M) +check(m_host is None or "${BASE_DOMAIN}" in m_host.group(1), + f"{PREFIX}/serving/httproute.yaml hardcodes hostname '{m_host.group(1) if m_host else ''}' — " + f"use the ${{BASE_DOMAIN}} token, or omit `hostnames:` entirely and inherit the listener's") + +ns_text = read(NS) +check('gateway-access: "shared"' in ns_text or "gateway-access: shared" in ns_text, + f"{PREFIX}/serving/namespace.yaml does not carry gateway-access=shared — the shared Gateway " + f"selects namespaces by that label, so without it the HTTPRoute is refused attachment") + +nginx_text = read(NGINX) +check(f"location /{PREFIX}/" in nginx_text, + f"{PREFIX}/serving/nginx-config.yaml has no `location /{PREFIX}/` block — the route would " + f"deliver the request and the server would answer 404 for every runbook") +check("default_type" in nginx_text and "text/plain" in nginx_text, + f"{PREFIX}/serving/nginx-config.yaml does not force a text/plain default_type. The ConfigMap " + f"keys are EXTENSIONLESS, so with the stock MIME table nginx serves them as " + f"application/octet-stream and a browser DOWNLOADS the runbook instead of showing it") + +# ====================================== (6) the Flux tier is wired, and is NOT substituted +FLUX = ROOT / "clusters" / "base" / f"{PREFIX}.yaml" +BASE_KUST = ROOT / "clusters" / "base" / "kustomization.yaml" +flux_text = read(FLUX) +check(bool(flux_text), + f"clusters/base/{PREFIX}.yaml is missing — nothing reconciles {PREFIX}/ into the cluster, so " + f"the runbooks exist in git and the URL still 404s") +check(re.search(rf"^\s*path:\s*\./{re.escape(PREFIX)}\s*$", flux_text, re.M) is not None, + f"clusters/base/{PREFIX}.yaml does not set `path: ./{PREFIX}`") +check(re.search(r"^\s*wait:\s*false\s*$", flux_text, re.M) is not None, + f"clusters/base/{PREFIX}.yaml is not wait:false — a leaf web server that can be legitimately " + f"unhealthy must never gate the reconcile chain (DECISIONS-LOG: 'Headlamp gating stall')") +# Strip comments before testing: this file's own comment block EXPLAINS at length why +# substituteFrom must never be added here, and a naive substring test would fire on the warning. +flux_code = "\n".join(l for l in flux_text.splitlines() if not l.lstrip().startswith("#")) +check("substituteFrom" not in flux_code, + f"clusters/base/{PREFIX}.yaml declares postBuild.substituteFrom. It MUST NOT: this tier " + f"renders documentation into a ConfigMap, and Flux's envsubst would run over the prose and " + f"silently EMPTY every `${{...}}` shell example in every runbook — it fails open, so the " + f"Kustomization would still report Ready (DECISIONS-LOG 2026-09-01, the emptied Cilium " + f"k8sServiceHost). The flux-postbuild-substitution check cannot see this: it scans files " + f"reachable through `resources:`, not configMapGenerator inputs") +check(re.search(rf"^\s*-\s*{re.escape(PREFIX)}\.yaml\b", read(BASE_KUST), re.M) is not None, + f"clusters/base/kustomization.yaml does not list {PREFIX}.yaml — the Flux Kustomization is " + f"authored and never applied, which is the 'looks done, is absent' class this repo has been " + f"bitten by repeatedly") + +# ============================================ (7) this check is load-bearing in the merge gate +L0 = read(ROOT / "ci" / "l0.sh") +GATE = read(ROOT / "ci" / "gate.sh") +check("alert-runbooks-resolvable/check.sh" in L0, + "ci/l0.sh does not run alert-runbooks-resolvable/check.sh. The requirement-scoped conformance " + "selection in Stage 3b would filter this check out of exactly the change that breaks it: " + "ci/touched-requirements.sh maps platform/monitoring/* to 'R-OBS-01 R-OBS-02' (not R-OBS-04) " + "and maps platform/kubevirt/* and runbooks/* to nothing, so adding an alert with a new " + "runbook_url attributes no requirement this test carries. It must be invoked directly and " + "unconditionally, like stages 3c/3d/3e") +check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE), + "ci/gate.sh does not set the merge block when the l0 stage fails") + +# ======================================================================== report +print(f"Alert runbook inventory: {n_annotations} runbook_url annotation(s) across " + f"{len(rule_files)} PrometheusRule file(s) -> {len(names)} distinct runbook(s), " + f"prefix /{PREFIX}") +missing = sorted(n for n in names if not (RUNBOOK_DIR / f"{n}.md").is_file()) +print(f" runbooks present: {len(names) - len(missing)}/{len(names)} " + f"projected into the ConfigMap: {sum(1 for n in names if generated.get(n) == f'{n}.md')}/{len(names)}") +if missing: + print(f" MISSING: {', '.join(missing)}", file=sys.stderr) + +if ADVISORIES: + print("\nADVISORIES (not failures):") + for a in ADVISORIES: + print(f" - {a}") + +print() +if not FAILURES: + print(f"check OK: alert-runbooks-resolvable — {CHECKS} assertions (R-OBS-04)") + raise SystemExit(0) +print(f"CHECK FAIL (alert-runbooks-resolvable): {len(FAILURES)} failed, " + f"{CHECKS - len(FAILURES)} passed", file=sys.stderr) +for f in FAILURES: + print(f" FAIL {f}", file=sys.stderr) +raise SystemExit(1) +PY diff --git a/runbooks/README.md b/runbooks/README.md new file mode 100644 index 0000000..655066b --- /dev/null +++ b/runbooks/README.md @@ -0,0 +1,135 @@ +# Alert runbooks — home-network-k3s prod + +This is the index page served at `https://home.keithjackson.dev/runbooks/`. + +Every alert rule in this platform carries an annotation of the form + + runbook_url: "https://${BASE_DOMAIN}/runbooks/" + +42 of them, over the 34 names below. Until this directory existed, that path returned **HTTP 404** on the +live cluster: every alert that reached a human handed them a dead link at the exact moment they needed +help. These pages are what makes the link resolve. + +## How to run the commands in these runbooks + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang +is not the cluster being down. + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod`: `dev.sh` merges both clusters into one kubeconfig and defaults its +current-context to k3s-**dev**, so a command without `--context` silently answers about the wrong cluster. +`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is the equivalent form. Do not run two `dev.sh` +invocations at once — they share one container and interfere. + +**Use fully-qualified resource names, always.** Three CRDs on this cluster register the kind `Backup` — +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1`, `velero.io/v1` — and the bare short name resolves to +Longhorn's, which holds zero objects. `kubectl get backup -A` therefore prints **"No resources found"** on +a cluster holding 6 Velero and 27 CNPG backups (verified on prod 2026-09-13). Acting on that false +negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`, +`backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, +`volumes.longhorn.io`, `nodes.longhorn.io`. + +`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git. + +## Alerts you should not trust the silence of (measured 2026-09-13) + +Prometheus scrapes 19 jobs on prod. There is **no** Longhorn, cert-manager, Flux-controller or +Hubble/Cilium job, so these rule groups currently evaluate against nothing and **cannot fire**: + +| Missing scrape job | Inert alerts | +|---|---| +| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornVolumeCapacityHigh`, `LonghornNodeStorageHigh` | +| cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` | +| Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` | +| Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` | +| etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all | + +Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`. + +## The runbooks + +### Nodes +- **`node-down`** — a node's node-exporter is unscrapable; the node may be gone, or its kubelet may have + stopped posting status while the host stays up (the 2026-09-07 shape). +- **`node-io`** — disk I/O pressure/starvation. The etcd WAL shares a partition with Longhorn replicas; + this is the alert that would have called the 2026-09-07 outage and did not exist at the time. +- **`node-memory`** — memory pressure. Includes why `SwapTotal == 0` is not evidence swap is off. +- **`node-disk`** — a node filesystem filling up. `/data` is the one that matters. +- **`node-cpu`** — CPU PSI and run-queue depth, and how to tell CPU contention from I/O contention. + +### Storage +- **`longhorn-degraded`** — a volume has lost redundancy. Normal after any reboot, and the rebuild is the + expensive part. Freeze/thaw commands here. +- **`longhorn-faulted`** — no usable replica. Data at risk; stop before anything destructive. +- **`longhorn-capacity`** — a volume or a node's Longhorn disk near capacity. +- **`pvc-capacity`** — PVC filling up / almost full / predicted to fill. These rules work. +- **`monitoring-capacity`** — Prometheus or Loki running out of disk; don't let the monitor blind itself. + +### GitOps and updates +- **`flux-reconcile`** — a Flux resource stuck at Ready=False. Which tier decides the blast radius. +- **`flux-suspended`** — reconciliation turned off and forgotten. Drift becomes silent. +- **`update-rollback`** — a HelmRelease failed its health gate and is rolling back. +- **`image-automation`** — the git write-back engine is failing (feature is disabled here). +- **`image-policy-stalled`** — an ImagePolicy/ImageRepository cannot resolve a tag. + +### Certificates +- **`cert-issuance`** — ACME issuance failing. The DNS-01 self-check is the usual cause. +- **`cert-expiry`** — a certificate is expiring. Almost always a late symptom of `cert-issuance`. + +### Networking +- **`vpn-down`** — the egress VPN tunnel. Includes the silent egress-gateway staleness that this alert + cannot see, and what the kill-switch actually does. +- **`denied-flows`** — policy-denied flow spike. The Cilium hairpin/identity regression class. + +### Backups and DR +- **`backup-failure`** — a Velero backup failed. The R2 object-tagging incompatibility is documented here. +- **`backup-partial`** — a Velero backup completed with partial failures, and which skips are deliberate. +- **`backup-missed`** — no successful Velero backup in 24h. +- **`cnpg-archiving`** — WAL archiving stuck / no recoverability point. **An availability incident, not + just a backup one.** +- **`cnpg-backup-missed`** — no recent CNPG base backup. Velero does not cover these databases. +- **`cnpg-wal-growth`** — `pg_wal` not being recycled. Never delete WAL by hand. +- **`cnpg-exporter`** — the CNPG SQL exporter erroring: a blind spot that looks exactly like health. + +### Identity +- **`keycloak-down`** — the IdP is down. Blast radius, and the break-glass path that still works. +- **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable. +- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that + every green signal missed. + +### Virtualisation +- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running. +- **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps. +- **`kubevirt-crashloop`** — a virt-launcher pod restarting repeatedly. +- **`cdi-import-stuck`** — a DataVolume import not progressing. Check node I/O before restarting it. + +## Editing a runbook + +Each page is `runbooks/.md` in the `home-network-k3s-impl` repo. Edit it, push to `main`, and Flux +re-renders the ConfigMap and rolls the two-replica server; kustomize's content hash on the generated +ConfigMap name is what triggers the roll. + +Adding a **new** alert with a `runbook_url` requires the matching `runbooks/.md` **and** its +`=.md` entry in `runbooks/kustomization.yaml`. Both are enforced by +`conformance/tests/l0-static/alert-runbooks-resolvable`, which runs unconditionally in the merge gate — +"the file exists in git" and "the URL resolves in the cluster" are different claims, and it checks both. + +Every page must carry the four operator sections (*What this means here*, *First commands*, *Likely +causes, most likely first*, *When to escalate or stop*), mention `dev.sh`, and pass `--context k3s-prod`. +The check enforces that too, because a stub that makes the link resolve while helping nobody is worse than +the 404 it replaced — it looks fixed. + +## Why this route has no login + +Deliberately public. Every other exposed UI here declares forward-auth via oauth2-proxy → Keycloak; if +these pages did too, `keycloak-down` would be unreadable exactly when Keycloak is down, +`keycloak-db` unreadable when its database is down, and `keycloak-client-secrets` unreadable during the +precise failure that broke all 12 OIDC clients on 2026-09-06 behind an entirely green cluster. An +operator who cannot log in is this route's primary reader. The content is operational procedure — metric +names, thresholds, kubectl invocations, topology — and no secrets. See +`runbooks/serving/httproute.yaml` for the full reasoning, including the instruction to leave this route +**out** of the Gateway ext-authz attachment when forward-auth enforcement lands. + +The running narrative of real incidents on this cluster is `DECISIONS-LOG.md`. diff --git a/runbooks/backup-failure.md b/runbooks/backup-failure.md new file mode 100644 index 0000000..ef0486f --- /dev/null +++ b/runbooks/backup-failure.md @@ -0,0 +1,169 @@ +# backup-failure — A Velero backup failed + +**Alert:** `VeleroBackupFailure` (critical, 5m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups` +**Expression:** `increase(velero_backup_failure_total[1h]) > 0` + +## What this means here + +A Velero backup in the last hour ended in `phase=Failed`. Off-cluster DR (R-STATE-03) is at risk. Velero +backs up the **cluster's Kubernetes objects and its Longhorn PVC data**; it does **not** provide the DR +path for the three CloudNativePG databases — CNPG owns those via its own Barman archive, because a +Velero snapshot of a live Postgres is not a backup. So a Velero failure and a CNPG failure are separate +incidents with separate runbooks (`/runbooks/cnpg-archiving`). + +## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE + +`kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**. +Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and +`velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified +on prod 2026-09-13, where the fully-qualified queries returned **6 Velero backups** and **27 CNPG +backups** while the short name returned nothing. + +This has already caused two false "the backups are missing" conclusions in one night. **A destructive +recovery action taken on that false negative is how a monitoring quirk becomes real data loss.** Use: + +``` +backups.velero.io backups.postgresql.cnpg.io backups.longhorn.io +schedules.velero.io scheduledbackups.postgresql.cnpg.io clusters.postgresql.cnpg.io +``` + +## Where the backups go + +Cloudflare **R2**, since the operator's 2026-09-05 decision (commit `32d6766`). Prod cannot reach the +on-Proxmox MinIO at all — measured from a prod pod, *connection timed out after 15s* — and that MinIO +ran on the same physical host as dev's VMs, so dev's "off-cluster" backup shared a failure domain. +Live config: endpoint `https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`, region +**`auto`**, bucket `k3s-prod-backup`, prefix `velero`, schedule `0 3 * * *`, retention `720h`. +`backupstoragelocation.velero.io/default` should read **Available**. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which backup failed, and why — fully-qualified, always: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get backups.velero.io -A \ + -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,START:.status.startTimestamp,ERRORS:.status.errors,WARN:.status.warnings' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero get backups.velero.io -o yaml | sed -n "/status:/,\$p"' +``` + +The storage location and the controller's log (the R2 error text is in the log, not the CR): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero get backupstoragelocations.velero.io,schedules.velero.io; echo; \ + kubectl --context k3s-prod -n velero logs deploy/velero --tail=150 2>&1 | grep -iE "error|NotImplemented|InvalidArgument|tagging|checksum" | tail -25' +``` + +Did the **volume data** upload even though the backup failed? (This distinction is the whole point — +see cause 1.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero get podvolumebackups.velero.io \ + -o custom-columns=NAME:.metadata.name,BACKUP:.metadata.labels.velero\\.io/backup-name,PHASE:.status.phase,BYTES:.status.progress.bytesDone' +``` + +The plugin pin (cause 1 depends entirely on this): + +``` +grep -n "velero-plugin-for-aws\|checksumAlgorithm\|region" platform/velero/helmrelease.yaml +``` + +## Likely causes, most likely first + +### 1. The R2 object-tagging incompatibility (the known, specific, repo-documented one) +`velero-plugin-for-aws` **v1.14.2** sets PutObject's Tagging unconditionally — +`Tagging: aws.String(o.tagging)` with `o.tagging == ""` — so the SDK emits `x-amz-tagging: ` (empty) on +**every** PutObject. **R2 does not implement object tagging at all** and answers: + +``` +NotImplemented: Header 'x-amz-tagging' with value '' not implemented +``` + +Observed live on prod **twice**, on the write of `velero/backups//velero-backup.json`. The shape +of the failure is the dangerous part: **kopia uploads the volume DATA successfully (its own S3 client), +then the Backup goes `phase=Failed` with NO metadata object — an unrestorable backup.** +**No configuration avoids it** (`config.tagging` with a real value sends a tag set R2 also rejects). The +fix is upstream PR #309, which landed on release-1.14 *after* v1.14.2 was cut, so the image is pinned to +**`velero/velero-plugin-for-aws:v1.14.3-rc.2`**. +**EXIT CONDITION: move to v1.14.3 the moment it ships GA. Do NOT "clean this up" back to v1.14.2 — that +silently breaks every prod backup's metadata.** If this alert fires and the pin has drifted, that is +your answer. + +Two more mandatory R2 settings in the same file: `config.checksumAlgorithm: ""` (R2 rejects the AWS +SDK's `x-amz-checksum-*` headers with **HTTP 400 InvalidArgument** — harmless against MinIO, fatal +against R2) and `region` exactly **`auto`** (it was once hardcoded `us-east-1`, correct-by-accident for +MinIO). + +### 2. A placeholder or wrong endpoint +`clusters/prod` deliberately carried the **loud** placeholder +`https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` rather than inheriting the base (dev) value, +because *"an omitted op means prod silently inherits a dev address — far worse than a failed apply"*. +Check `S3_BACKUP_ENDPOINT`, `S3_BACKUP_REGION`, `VELERO_BACKUP_BUCKET`, `VELERO_BACKUP_PREFIX` and +`BACKUP_CREDENTIALS_SUFFIX` in `clusters/base/cluster-vars.yaml` and the prod overlay. A +`backupstoragelocation` that is not `Available` points straight here. + +### 3. Credentials +The R2 token lives in a SOPS-encrypted Secret (`velero-cloud-prod.sops.yaml`), and the `velero` tier +carries its own `decryption` stanza. An expired or mis-scoped token gives 403s in the log. + +### 4. A volume could not be read +Check `podvolumebackups.velero.io` for a specific volume. Note that NFS-sourced volumes are +**deliberately skipped** by the `velero-volume-policies` ConfigMap — see `/runbooks/backup-partial`, +which is where that belongs. + +### 5. The node or storage was unhealthy during the run +Velero's daily runs at 03:00 and reads Longhorn PVCs. If a node was under I/O pressure or a volume was +degraded, the read can fail. Cross-read `/runbooks/node-io` and `/runbooks/longhorn-degraded`. + +## When to escalate or stop + +- **Escalate promptly.** This is a `critical` because it is a DR alarm: the cluster may currently have no + recent restorable backup. +- **Do not delete a Failed backup to "clean up".** A Failed Velero backup can leave **orphaned kopia + blobs** in the bucket — prod has two such backups from 2026-09-12 (`prod-proof-20260912`, + `prod-vp-probe-20260912`) whose orphaned blobs are **pending owner consent to delete**. Deleting + objects in R2 is an operator decision, not a tidy-up. +- **Do not assume a Completed backup is restorable** if the plugin pin has drifted — the failure mode is + precisely "data uploaded, metadata missing". +- **Be honest about restore**: `dr-restore` is **authored and has never been executed** + (`conformance.home.k3s/never-executed`), so the R2 restore path is unrehearsed. R2 egress is free, + which is the whole reason restore rehearsals are affordable — but nobody has done one yet. +- The three CNPG databases are **not** covered by Velero. If the concern is a database, go to + `/runbooks/cnpg-archiving`. + +--- +Source of truth: `runbooks/backup-failure.md`. Index: `/runbooks/` +Records: commit `2e259a8` and `platform/velero/helmrelease.yaml` lines ~68-95 (the R2 tagging +incompatibility, `checksumAlgorithm`, `region: auto`); commit `32d6766` (the move to R2); commit +`2d5171e` (Velero shipped with no ServiceMonitor, so these alerts could never fire). diff --git a/runbooks/backup-missed.md b/runbooks/backup-missed.md new file mode 100644 index 0000000..0ba1ae5 --- /dev/null +++ b/runbooks/backup-missed.md @@ -0,0 +1,157 @@ +# backup-missed — No successful Velero backup in 24 hours + +**Alert:** `VeleroNoRecentBackup` (critical, 1h) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups` +**Expression:** `(time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)` + +## What this means here + +A Velero **schedule** has not produced a successful backup in over a day. This is the alert that catches +the quiet failure — not "a backup broke" (that is `/runbooks/backup-failure`) but "backups stopped +happening and nobody noticed". The `{schedule!=""}` filter means it only watches scheduled backups, so +ad-hoc ones do not mask a dead schedule. + +Live shape on prod: two schedules, both `0 3 * * *` — +`schedule.velero.io/velero-daily` and `schedule.velero.io/velero-daily-resources-only` — retention +`720h`, target Cloudflare R2 (bucket `k3s-prod-backup`, prefix `velero`, region `auto`). + +**Why this alert is worth its `critical`:** the precedent is that these rules *could not fire at all*. +Velero shipped with three backup alerts and the chart default `metrics.serviceMonitor.enabled: false`, +confirmed live on prod — `kubectl -n velero get servicemonitor` returned no resources. Recorded verbatim: +*"A nightly backup could fail — or silently stop running — for weeks with nothing to say so."* The +ServiceMonitor now exists (`velero` and `velero/node-agent` are both live scrape jobs as of 2026-09-13), +which is the only reason this alert is real. +## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE + +`kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**. +Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and +`velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified +on prod 2026-09-13: the fully-qualified queries returned **6 Velero backups** and **27 CNPG backups** +while the short name returned nothing. This false negative has already produced two wrong "the backups +are missing" conclusions in one night. **A destructive recovery action taken on it is how a CLI quirk +becomes real data loss.** Always: + +``` +backups.velero.io backups.postgresql.cnpg.io backups.longhorn.io +schedules.velero.io scheduledbackups.postgresql.cnpg.io clusters.postgresql.cnpg.io +``` +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Are the schedules enabled, and when did each last run? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero get schedules.velero.io \ + -o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,PAUSED:.spec.paused,LASTBACKUP:.status.lastBackup,PHASE:.status.phase' +``` + +What backups actually exist (fully-qualified — the short name lies): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get backups.velero.io -A \ + -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,SCHEDULE:.metadata.labels.velero\\.io/schedule-name,START:.status.startTimestamp --sort-by=.status.startTimestamp' +``` + +Is the storage location usable at all? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero get backupstoragelocations.velero.io \ + -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,LASTVALIDATED:.status.lastValidationTime,DEFAULT:.spec.default' +``` + +Is Velero's own Flux tier reconciling, or was it suspended? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \ + -o custom-columns=NAME:.metadata.name,SUSPENDED:.spec.suspend,READY:.status.conditions[?\(@.type==\"Ready\"\)].status | grep -Ei "velero|NAME"' +``` + +And the controller: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero get pods; \ + kubectl --context k3s-prod -n velero logs deploy/velero --tail=120 2>&1 | tail -40' +``` + +## Likely causes, most likely first + +### 1. The Velero Flux tier was suspended and nobody resumed it +This has happened: **Velero was found suspended live, with the suspension never recorded in git**, and +was resumed with `flux resume`. A suspended backup controller is a silent DR gap and is precisely the +class this alert exists for. Remediation (operator): `flux resume kustomization velero`, and see +`/runbooks/flux-suspended` for why an undocumented live suspension is itself a problem. + +### 2. The schedule is paused +`schedules.velero.io` has `spec.paused`. Note from the live listing that +`velero-daily-resources-only` shows an empty PAUSED column while `velero-daily` shows `false` — read the +field, not the column width. A paused schedule is a git change to unpause. + +### 3. Every run is failing rather than not running +Then `backups.velero.io` will show recent `Failed` entries and this alert is a downstream symptom. Go to +`/runbooks/backup-failure` — and in particular check the plugin pin +(`velero/velero-plugin-for-aws:v1.14.3-rc.2`), because the R2 tagging bug produces exactly this: volume +data uploads, the backup goes Failed, and no successful backup ever lands. + +### 4. The BackupStorageLocation is not Available +A wrong endpoint, a bad credential, or the placeholder. `clusters/prod` deliberately carried the loud +placeholder `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` rather than inheriting the base (dev) +value, because *"an omitted op means prod silently inherits a dev address — far worse than a failed +apply"*. Check `S3_BACKUP_ENDPOINT`, `S3_BACKUP_REGION` (`auto`), `VELERO_BACKUP_BUCKET`, +`VELERO_BACKUP_PREFIX`, `BACKUP_CREDENTIALS_SUFFIX` (`-prod`). + +### 5. Velero cannot schedule a pod +Node pressure, or the node-agent DaemonSet unhealthy. Cross-read `/runbooks/node-io`. + +### 6. The metric, not the backup, is missing +If `velero_backup_last_successful_timestamp` has no series at all the rule cannot fire — which is the +historical state. Confirm the `velero` scrape job is up (`/runbooks/targets-down`) before trusting either +its silence or its firing. + +## When to escalate or stop + +- **Escalate: this is a DR alarm and 24h is already a real window of exposure.** +- **Do not create an ad-hoc backup and consider the alert answered.** The expression filters + `{schedule!=""}`, so a manual backup will not clear it, and more importantly it does not fix the + schedule. (An operator's ad-hoc run for reassurance is + `velero backup create --from-schedule velero-daily`.) +- **Do not conclude "no backups exist" from `kubectl get backup`.** See the trap above. +- **Remember the databases are separate.** `keycloak-db`, `nextcloud-db` and `semaphore-db` are covered + by CNPG's Barman archive, not by Velero. This alert being clear says nothing about them, and vice + versa — check `/runbooks/cnpg-backup-missed` too. +- **Be honest about restore:** `dr-restore` is authored and has **never been executed**, so the R2 + restore path is unrehearsed. Free R2 egress makes a rehearsal cheap; it is still outstanding. + +--- +Source of truth: `runbooks/backup-missed.md`. Index: `/runbooks/` +Records: commit `2d5171e` (Velero had no ServiceMonitor, so these alerts could never fire); +`DECISIONS-LOG.md` (Velero found suspended with the suspension never in git; `dr-restore` never +executed); commit `32d6766` (R2 decision and endpoint knobs). diff --git a/runbooks/backup-partial.md b/runbooks/backup-partial.md new file mode 100644 index 0000000..d652a0b --- /dev/null +++ b/runbooks/backup-partial.md @@ -0,0 +1,160 @@ +# backup-partial — A Velero backup completed with partial failures + +**Alert:** `VeleroBackupPartialFailure` (warning, 5m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups` +**Expression:** `increase(velero_backup_partial_failure_total[1h]) > 0` + +## What this means here + +The backup finished, but some resources or volumes were skipped or errored. It is a warning rather than a +critical because *most of it worked* — and that is exactly what makes it dangerous: a +`PartiallyFailed` backup looks like a backup in every listing, and the thing it silently omitted may be +the thing you need. + +**The first question is always: was the skip deliberate?** On this cluster a great deal of skipping is +intentional and correct. +## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE + +`kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**. +Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and +`velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified +on prod 2026-09-13: the fully-qualified queries returned **6 Velero backups** and **27 CNPG backups** +while the short name returned nothing. This false negative has already produced two wrong "the backups +are missing" conclusions in one night. **A destructive recovery action taken on it is how a CLI quirk +becomes real data loss.** Always: + +``` +backups.velero.io backups.postgresql.cnpg.io backups.longhorn.io +schedules.velero.io scheduledbackups.postgresql.cnpg.io clusters.postgresql.cnpg.io +``` + +## The deliberate exclusions (know these before calling anything a failure) + +**NFS-sourced volumes are skipped on purpose, and the reason is a near-miss worth remembering.** The +opt-out file-system-backup schedule (`defaultVolumesToFsBackup: true`, excluding only `monitoring` and +`unifi`) would have read the NFS media library **three times** — `plex-media`, `sonarr-media` and +`radarr-media` are all bound to the same `192.168.2.102:/mnt/Storage/plex` export, **8.32 TiB +measured** — plus `nextcloud-legacy-archive` (270 GiB), pushing roughly **8.6 TiB to R2 nightly with +720h retention**. Namespace exclusion could not express the fix, because those namespaces also hold real +Longhorn PVCs that must be backed up. + +The fix is a **per-volume opt-out by SOURCE TYPE**: the ConfigMap `velero-volume-policies` skips any +volume whose source is NFS, and the schedule references it. The per-pod +`backup-volumes-excludes` annotation was **rejected for failing OPEN** (forget the annotation and you +back up 8 TiB). Proven on prod: a Backup of namespace `nextcloud` produced exactly **one** +PodVolumeBackup (`data`, Longhorn, 875,417,102 bytes, Completed) and the NFS volume produced none. +**Residual gap, stated honestly: the rule matches NFS only.** A future non-NFS bulk source would not be +covered. + +**The three CNPG databases are also out of scope by design.** `keycloak-db`, `nextcloud-db` and +`semaphore-db` have their own DR path (Barman WAL archive + base backups to R2) because a Velero snapshot +of a live Postgres is not a backup. If a `*-db-N` volume is skipped, that is correct — see +`/runbooks/cnpg-archiving` for the path that does cover them. + +**`monitoring` and `unifi` are excluded** from file-system backup as well. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which backup, and what exactly it warned about: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get backups.velero.io -A \ + -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,ERRORS:.status.errors,WARN:.status.warnings,START:.status.startTimestamp' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero get backups.velero.io -o yaml | sed -n "/status:/,\$p"' +``` + +Per-volume results — this tells you whether a skip was the NFS policy or a real failure: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero get podvolumebackups.velero.io \ + -o custom-columns=NAME:.metadata.name,BACKUP:.metadata.labels.velero\\.io/backup-name,POD:.spec.pod.name,VOL:.spec.volume,PHASE:.status.phase,BYTES:.status.progress.bytesDone' +``` + +The volume policy that does the deliberate skipping: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero get cm velero-volume-policies -o yaml' +``` + +The controller's own account of what it skipped: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero logs deploy/velero --tail=200 2>&1 | grep -iE "skip|excluded|warn|error" | tail -30' +``` + +## Likely causes, most likely first + +### 1. The deliberate NFS / CNPG / monitoring exclusions +Most common by far, and correct. Confirm against the `velero-volume-policies` ConfigMap and the +schedule's excluded namespaces. **No action.** If you are tempted to "fix" it, re-read the 8.6 TiB +near-miss above. + +### 2. A resource Velero could not serialise +Usually a CRD whose API was unavailable at the moment of the backup, or an admission webhook refusing a +read. The log names the group/kind. Often transient; if it recurs for the same kind, that kind is +genuinely not backed up. + +### 3. A volume was busy, degraded, or on a node under pressure +Velero's daily runs at 03:00 against Longhorn PVCs. A degraded volume or a node with high I/O-PSI can +fail one volume's read while the rest succeed. Cross-read `/runbooks/longhorn-degraded` and +`/runbooks/node-io`. + +### 4. A partial R2 write +Related to the tagging/checksum incompatibilities in `/runbooks/backup-failure`: those normally produce a +hard `Failed`, but a partially-written backup is worth checking the plugin pin for +(`velero/velero-plugin-for-aws:v1.14.3-rc.2` — do **not** revert to v1.14.2). + +### 5. A namespace was created or deleted mid-backup +Benign race. Leaked test namespaces have existed on this cluster and been deleted; a namespace +disappearing under Velero produces exactly this warning. + +## When to escalate or stop + +- **Decide, and record, whether the skipped items matter.** The honest failure mode here is shrugging at + a `PartiallyFailed` and discovering later that something was never covered. If the skip was not one of + the documented exclusions, treat it as an uncovered resource. +- **Do not add namespaces to the exclusion list to quiet the alert.** Those namespaces hold real + Longhorn PVCs; namespace-level exclusion is the mistake the source-type policy exists to avoid. +- **Do not switch to the per-pod `backup-volumes-excludes` annotation** — it was rejected for failing + OPEN, and the consequence of forgetting it is 8 TiB per night. +- **Do not delete a PartiallyFailed backup** — like a Failed one it can leave orphaned kopia blobs, and + deleting R2 objects is an operator decision. +- Escalate if the same resource or volume is skipped on consecutive nights: that is a coverage gap, which + is an R-STATE-03 matter and not a transient. + +--- +Source of truth: `runbooks/backup-partial.md`. Index: `/runbooks/` +Records: commit `2e259a8` (the 8.6 TiB near-miss, the source-type volume policy, and the rejection of the +fail-open annotation); `docs/BULK-NFS-STORAGE.md`. diff --git a/runbooks/cdi-import-stuck.md b/runbooks/cdi-import-stuck.md new file mode 100644 index 0000000..f8bb8d9 --- /dev/null +++ b/runbooks/cdi-import-stuck.md @@ -0,0 +1,168 @@ +# cdi-import-stuck — A CDI DataVolume import is not progressing + +**Alert:** `CDIDataVolumeImportStuck` (warning, 30m) +**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt` +**Expression:** `sum(kubevirt_cdi_import_pods_high_restart) > 0` + +## What this means here + +A CDI DataVolume import or clone has not completed after 30 minutes, so **the VM disk it backs cannot +start**. `kubevirt_cdi_import_pods_high_restart` counts importer pods stuck restarting; a sustained +positive value means the import is not making progress rather than merely being slow. + +The 30-minute `for` is generous on purpose: on this hardware a genuine large import is legitimately slow. +The reference case is the **UniFi VM's 60 GiB import**, and the `apps-vms` tier is deliberately +**non-gating** (`wait: false`, `dependsOn: platform-config`) precisely because a VM's image import plus +in-guest install is legitimately long-not-Ready. + +**Two things to know before triaging:** + +1. **Imports are the single heaviest write this platform performs**, and the etcd WAL (`/data/k3s`) shares + one partition with every Longhorn replica (`/data/longhorn`). A 60 GiB import is exactly the class of + write load behind the 2026-09-07 `k3s-prod-1` NotReady outage. So "restart the import" is not a free + action — check `/runbooks/node-io` first. +2. **There is a known false-positive precedent in this area.** A broken HTTP CDI ServiceMonitor once caused + a false `CDIOperatorDown` (commit `6a7e9d1`), and the sibling `KubeVirtControlPlaneDown` once used a + non-existent metric with the wrong job label (`b57b3e9`). The metric name in this rule is flagged in + the file as one to confirm against the deployed CDI. `cdi-prometheus-metrics` **is** a live scrape job + on prod (measured 2026-09-13), so the target is up — but verify the series exists before trusting either + this alert's firing or its silence. + +Platform versions: **CDI v1.66.0** with **KubeVirt v1.9.0**, on real nested KVM. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which DataVolume, and how far it got: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get datavolumes.cdi.kubevirt.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PHASE:.status.phase,PROGRESS:.status.progress,RESTARTS:.status.restartCount' +``` + +The importer pod is where the real error is: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get pods -A | grep -E "importer|upload|clone"; echo; \ + kubectl --context k3s-prod -n unifi logs -l app=containerized-data-importer --tail=100 2>&1 | tail -50' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n unifi describe pod -l app=containerized-data-importer 2>&1 | sed -n "/Events:/,\$p"' +``` + +Is there anywhere to put it — and is the scratch class configured? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get pvc -A | grep -Ei "scratch|dv|rootdisk"; echo; \ + kubectl --context k3s-prod get sc; echo; \ + kubectl --context k3s-prod get cdis.cdi.kubevirt.io -o yaml | grep -A5 scratchSpace' +``` + +Is the node being flattened by the import? **Check this before restarting anything:** + +``` +ssh k3s-prod-1 'cat /proc/pressure/io; df -h /data' +``` + +Can the importer reach the image source at all? (Default-deny is the baseline everywhere.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \ + hubble observe --verdict DROPPED --last 100 -o compact | grep -i import' +``` + +Does the metric this alert depends on actually exist? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + for q in "up{job=\"cdi-prometheus-metrics\"}" "count(kubevirt_cdi_import_pods_high_restart)"; do \ + printf "%-50s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \ + | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done' +``` + +## Likely causes, most likely first + +### 1. Not enough schedulable storage for the target or the scratch space +The recorded failure, and the reason a whole StorageClass exists: **the 3-replica default class failed the +UniFi 60Gi import with insufficient storage / `ReplicaSchedulingFailure`.** The fix was a dedicated +single-replica **`longhorn-scratch`** class (`numberOfReplicas: 1`, `dataLocality: best-effort`), +referenced by the CDI CR's `scratchSpaceStorageClass`. If scratch is pointed at the 3-replica default, +this is your cause. Note that a 60 GiB import needing 3x replication plus scratch space is a very +different ask from 60 GiB. + +### 2. An egress or default-deny block on the image source +Import pods are ordinary pods under the cluster's default-deny baseline (R-NET-01). The alert's own +description names this: *"often an egress/default-deny block on the image source, or a slow mirror."* +Remediation is an explicit allow in git; see `/runbooks/denied-flows` — and in particular the +post-Cilium-upgrade finding that `toEntities: world` no longer covers a destination that resolves to an +in-cluster backend. + +### 3. The importer cannot open the raw Block device +`nonroot-devices: true` is set on the k3s servers because the **non-root CDI importer must open a raw +Block device that Kubernetes never chowns for it** (commit `df5979f`). A permission error in the importer +log against a Block-mode PVC is this. `longhorn-vm` is Block-mode only, so VM disks are always in scope. + +### 4. The source is slow, unreachable, or the URL is wrong +A 404/403 or a stalled transfer appears plainly in the importer log. A slow mirror is a genuine cause — the +30m `for` already allows for it, so check throughput rather than assuming a fault. + +### 5. Node I/O pressure is starving the import +Self-reinforcing: the import is heavy, the disk is shared with etcd, and a node under I/O pressure slows +the import which then runs longer. `/runbooks/node-io`. + +### 6. Longhorn is degraded +An import onto a degraded volume competes with the rebuild. `/runbooks/longhorn-degraded`. + +### 7. The alert is wrong +Given the CDI/KubeVirt false-positive precedents, if the DataVolumes all read `Succeeded` this is a +monitoring fault. Fixing the rule is a real change, not a shrug — a permanently-firing alert is a +recorded, expensive failure mode here. + +## When to escalate or stop + +- **Check node I/O pressure BEFORE restarting or recreating an import.** A 60 GiB re-import is hours of + write load on the partition the embedded etcd fsyncs to, and that combination took a node out of the + cluster on 2026-09-07. If I/O-PSI is already elevated, wait or freeze Longhorn rebuilds first + (`/runbooks/longhorn-degraded`). +- **Do not delete the target DataVolume/PVC casually** — that guarantees a full re-import. +- **Do not point `scratchSpaceStorageClass` at the 3-replica default** to "keep things consistent". The + single-replica class exists because the default one demonstrably failed. +- **Do not widen a NetworkPolicy to `world`** to let an import out; author the narrow allow. +- Scope it honestly: the VM tier is non-gating, so a stuck import does not affect the container platform. + The exception is the UniFi VM, which is the home network's controller. +- Escalate for k3s server-flag changes (`nonroot-devices`) or CDI CR changes — Ansible and operator work. + +--- +Source of truth: `runbooks/cdi-import-stuck.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` (the UniFi 60Gi import failing on the 3-replica default; `longhorn-scratch`); +commits `df5979f` (`nonroot-devices`), `6a7e9d1` and `b57b3e9` (false CDI/KubeVirt alerts). diff --git a/runbooks/cert-expiry.md b/runbooks/cert-expiry.md new file mode 100644 index 0000000..a8328d4 --- /dev/null +++ b/runbooks/cert-expiry.md @@ -0,0 +1,142 @@ +# cert-expiry — A certificate is expiring soon, or imminently + +**Alerts:** `CertificateExpiringSoon` (warning, <14d, 1h) and `CertificateExpiryImminent` (critical, <3d, 1h) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.certificates` +**Expression:** `(certmanager_certificate_expiration_timestamp_seconds - time()) < (14 * 24 * 3600)` / `(3 * 24 * 3600)` + +## What this means here + +cert-manager has a Certificate whose expiry is inside 14 days (warning) or 3 days (critical) and it has +not renewed. cert-manager normally renews at two-thirds of lifetime — about 60 days into a 90-day Let's +Encrypt certificate — so **by the time this fires, renewal has already been failing for weeks**. Treat +it as a late symptom of `/runbooks/cert-issuance`, not as a separate problem: the fix is almost always +"why is issuance failing", and expiry is the clock running out on that. + +What is at stake concretely. There is effectively **one certificate that matters**: `wildcard-home` in +the `gateway` namespace, secret `wildcard-home-tls`, requesting **both** `*.${BASE_DOMAIN}` **and** +`${BASE_DOMAIN}` from `${ACME_CLUSTER_ISSUER}` (= `letsencrypt-prod` on prod). Every HTTPS listener on +the shared Gateway terminates with it — the wildcard `https` listener and the `https-apex` listener both +reference `wildcard-home-tls`. If it expires: + +- every public hostname serves an untrusted certificate; +- **SSO breaks in-cluster**, because OIDC clients validate the issuer's TLS. This is not hypothetical: + while the wildcard was still LE-**staging**, in-cluster clients rejected it with + `x509: certificate signed by unknown authority`, which is what forced the move to `letsencrypt-prod`; +- and the page you are reading right now is served over that same certificate. + +**There is no history of an actual expiry on this cluster.** The recorded history is all issuance +failures. Say so rather than inventing precedent — but note the corollary: the renewal path has never +been exercised under stress, so do not assume it will quietly work. + +### Verify the alert can even fire (measured 2026-09-13) + +**This alert is currently INERT.** `count(certmanager_clock_time_seconds)` and +`count(certmanager_certificate_expiration_timestamp_seconds)` both returned **0 series** on prod: +there is no cert-manager scrape job among Prometheus's 19 jobs, so all three certificate rules evaluate +against nothing. **Certificate expiry is unmonitored today.** Check expiry by hand with the commands +below, and see `/runbooks/targets-down` for the full list of inert rule groups. Adding cert-manager's +ServiceMonitor is open work. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Every certificate and its real expiry, straight from the CRs (works with no metrics): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get certificates.cert-manager.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,NOTAFTER:.status.notAfter,RENEWAL:.status.renewalTime,ISSUER:.spec.issuerRef.name' +``` + +What the Gateway is actually serving (the answer that matters), read from outside the cluster with an +explicit SNI — testing HTTPS from **inside** a pod is misleading here, and an IP+SNI probe without the +right SNI returns curl exit 35: + +``` +curl -sv --resolve home.keithjackson.dev:443:192.168.2.20 https://home.keithjackson.dev/runbooks/ 2>&1 | grep -E "expire|subject|issuer|SSL certificate" +``` + +``` +echo | openssl s_client -connect 192.168.2.20:443 -servername grafana.home.keithjackson.dev 2>/dev/null | openssl x509 -noout -subject -issuer -dates +``` + +Is a renewal in flight or stuck? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get certificaterequests.cert-manager.io,orders.acme.cert-manager.io,challenges.acme.cert-manager.io -A; echo; \ + kubectl --context k3s-prod -n cert-manager logs deploy/cert-manager --tail=80 | tail -40' +``` + +## Likely causes, most likely first + +### 1. Issuance has been failing since the last successful renewal +The dominant cause, and the one with real history. Go to `/runbooks/cert-issuance` — it covers the +DNS-01 self-check failure (fixed with `dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"` and +`dns01RecursiveNameserversOnly: true`), the Cloudflare token, and the ACME order/challenge chain. Fixing +issuance fixes expiry; nothing else does. + +### 2. Two clusters racing the same `_acme-challenge` TXT record +A specific, recorded hazard for any future two-cluster window. `infra/PROD-PROVISIONING.md` §10 records +that after the 2026-09-04 incident, `platform-config` **would have** requested a +`*.home.keithjackson.dev` wildcard from letsencrypt-prod for a certificate dev already held; both +clusters would then have raced each other's `_acme-challenge` TXT records on one Cloudflare zone and +**neither would reliably solve DNS-01**. It never fired only because the tier was suspended. If a second +cluster exists, confirm which one owns the zone before renewing. + +### 3. Let's Encrypt rate limits +One wildcard is well inside the limits, so this is unlikely — unless something has been retrying in a +loop. `cert-manager` logs will say `too many certificates already issued`. Remediation: stop the loop, +wait out the window; do **not** switch to letsencrypt-staging as a workaround, because in-cluster OIDC +clients reject the staging chain (that is the bug that forced LE-prod). + +### 4. The issuer or its credential changed +`${ACME_CLUSTER_ISSUER}` is a cluster-vars knob; the Cloudflare API token is a SOPS-encrypted Secret. +Note the standing rule: **non-secret, environment-specific values must not live inside an encrypted +Secret** — `${BASE_DOMAIN}` substitution can never reach inside one, which is exactly how prod's OIDC +clients once ended up pointed at dev's Keycloak. + +### 5. The Certificate is Ready but the Secret is stale +Check the Secret's actual notAfter (the `openssl` command above) rather than the CR's status. The Gateway +reads the Secret. + +## When to escalate or stop + +- **`CertificateExpiryImminent` on `wildcard-home` is a cluster-wide outage in under 3 days**, including + SSO and including this runbook server. Get the operator immediately. +- **Do not switch to `letsencrypt-staging`** to make issuance succeed. It will issue, and it will break + every in-cluster OIDC client with `x509: unknown authority`. +- **Do not delete the `wildcard-home-tls` Secret** to force a reissue while issuance is broken — you + would replace a certificate that still has days left with none at all. +- **Do not hand-edit the Secret.** cert-manager owns it; the durable change is in git. +- Verify DNS at the **authority** (`dig @betty.ns.cloudflare.com _acme-challenge.home.keithjackson.dev TXT`), + never at a resolver: 1.1.1.1 has served a deleted TXT record for minutes here and produced a false + failure diagnosis. + +--- +Source of truth: `runbooks/cert-expiry.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` "Fix 5 — cert-manager DNS-01 self-check via public resolvers" and the +LE-staging → LE-prod switch (Session 7); `infra/PROD-PROVISIONING.md` §10 (the queued ACME race). diff --git a/runbooks/cert-issuance.md b/runbooks/cert-issuance.md new file mode 100644 index 0000000..ca19f42 --- /dev/null +++ b/runbooks/cert-issuance.md @@ -0,0 +1,161 @@ +# cert-issuance — A Certificate is not Ready (ACME issuance is failing) + +**Alert:** `CertificateNotReady` (warning, 1h) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.certificates` +**Expression:** `certmanager_certificate_ready_status{condition="False"} == 1` + +## What this means here + +cert-manager cannot issue or renew a certificate. The one that matters is `wildcard-home` in the +`gateway` namespace — secret `wildcard-home-tls`, requesting **both** `*.${BASE_DOMAIN}` **and** +`${BASE_DOMAIN}` from `${ACME_CLUSTER_ISSUER}` (`letsencrypt-prod` on prod). Every HTTPS listener on the +shared Gateway terminates with it, so issuance failing is a slow-burning cluster-wide outage: it does +nothing today and takes down every hostname plus in-cluster SSO when the existing certificate expires. +The 1h `for` reflects that — a failing order is worth investigating today, not paging at 3am. + +The solver is **DNS-01 against Cloudflare**, which matters because DNS-01 has a failure mode this +cluster has already hit: + +**The self-check, not the challenge, is the usual failure.** On 2026-08-16 the wildcard stalled with the +challenge at `Presented=true` — the Cloudflare TXT record *was* created, so the API token worked — while +cert-manager's propagation self-check looped **"DNS record not yet propagated"**. The record was +publicly resolvable at 1.1.1.1 and 8.8.8.8; the NAT'd cluster's own DNS could not see it, because +split-horizon CoreDNS answers authoritatively for the zone and never forwards. The fix is live in +`platform/cert-manager/helmrelease.yaml`: + +```yaml +dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53" +dns01RecursiveNameserversOnly: true +``` + +The challenge went pending → valid and the certificate Issued. If you find those two lines missing or +changed, that is your answer. + +### Verify the alert can even fire (measured 2026-09-13) + +**This alert is currently INERT.** `count(certmanager_clock_time_seconds)` returned **0 series** on +prod — there is no cert-manager scrape job among Prometheus's 19 jobs, so all three certificate rules +evaluate against nothing. Certificate health must be read from the CRs, as below. See +`/runbooks/targets-down` for the full list of inert rule groups. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Walk the chain top-down — Certificate → CertificateRequest → Order → Challenge. The failure is almost +always visible at one specific level: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get certificates.cert-manager.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,NOTAFTER:.status.notAfter' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get certificaterequests.cert-manager.io -A; echo ==; \ + kubectl --context k3s-prod get orders.acme.cert-manager.io -A; echo ==; \ + kubectl --context k3s-prod get challenges.acme.cert-manager.io -A -o wide' +``` + +The controller log, which carries the actual ACME error text: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n cert-manager logs deploy/cert-manager --tail=150 2>&1 | grep -iE "error|propagat|challenge|order|rate" | tail -30' +``` + +Is the TXT record really there? **Ask the authority, never a resolver** — 1.1.1.1 has served a deleted +TXT record here for minutes and produced a false failure diagnosis: + +``` +dig +short @betty.ns.cloudflare.com _acme-challenge.home.keithjackson.dev TXT +dig +short @1.1.1.1 _acme-challenge.home.keithjackson.dev TXT +``` + +Is the resolver override still configured? + +``` +grep -n "dns01Recursive" platform/cert-manager/helmrelease.yaml +``` + +The issuer and its credential: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusterissuers.cert-manager.io \ + -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message' +``` + +## Likely causes, most likely first + +### 1. The DNS-01 propagation self-check cannot resolve the TXT record +The recorded failure. Signature: Challenge `Presented=true`, status looping on *"DNS record not yet +propagated"*, and `dig @betty.ns.cloudflare.com` **does** return the TXT. Cause: cert-manager is asking +the cluster's own resolver, which is split-horizon CoreDNS answering authoritatively for the zone and +never forwarding. Remediation: the two `dns01Recursive*` values above, in git. + +### 2. The Cloudflare API token is wrong, expired, or scoped too narrowly +Signature: Challenge never reaches `Presented=true`, and the log carries a Cloudflare 403/6003. The +token lives in a SOPS-encrypted Secret. Note the standing rule from the OIDC incident: **non-secret +environment-specific values must never live inside an encrypted Secret** — `${BASE_DOMAIN}` +substitution cannot reach inside one, and that is precisely how prod's OIDC clients once ended up +pointed at dev's Keycloak. Keep hostnames out; keep only the token in. + +### 3. `EXTERNALDNS_DOMAIN_FILTER` / zone confusion +Recorded as finding F-DNS1: the filter **must be the ZONE** (`keithjackson.dev`), not the subdomain. +A zone mismatch means the TXT record is created in the wrong place or not at all. + +### 4. Two clusters racing the same `_acme-challenge` record +`infra/PROD-PROVISIONING.md` §10: after the 2026-09-04 incident `platform-config` would have requested a +`*.home.keithjackson.dev` wildcard from letsencrypt-prod for a certificate dev already held; both +clusters would then have raced each other's `_acme-challenge` TXT records on one Cloudflare zone and +**neither would reliably solve DNS-01**. Only the tier being suspended prevented it. If any second +cluster exists, establish which one owns the zone first. + +### 5. Let's Encrypt rate limit +`too many certificates already issued` in the log. One wildcard is well inside the limits, so this means +something has been retrying in a loop. Stop the loop and wait out the window. + +### 6. A new Certificate for a hostname that does not resolve +A newly added service whose DNS is not yet published cannot solve DNS-01. Check ExternalDNS first. + +## When to escalate or stop + +- **Escalate if `wildcard-home` has been not-Ready for more than a day**, and check + `/runbooks/cert-expiry` for how much runway the existing certificate has. Under 3 days of runway is an + operator page. +- **Never switch `${ACME_CLUSTER_ISSUER}` to letsencrypt-staging** to get a green certificate. Staging + issues fine and then breaks every in-cluster OIDC client with `x509: certificate signed by unknown + authority` — that is the bug that forced LE-prod in the first place. +- **Do not delete the existing `wildcard-home-tls` Secret** while issuance is broken. A stale but valid + certificate is far better than none. +- **Do not create certificates by hand.** cert-manager owns them; the durable change is in git. +- Rotating the Cloudflare token is an operator action (SOPS, DR path). + +--- +Source of truth: `runbooks/cert-issuance.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` "Fix 5 — cert-manager DNS-01 self-check via public resolvers" +(commit ~`59dd3d9`) and the LE-staging → LE-prod switch; `infra/PROD-PROVISIONING.md` §10. diff --git a/runbooks/cnpg-archiving.md b/runbooks/cnpg-archiving.md new file mode 100644 index 0000000..648f9b7 --- /dev/null +++ b/runbooks/cnpg-archiving.md @@ -0,0 +1,194 @@ +# cnpg-archiving — WAL archiving is stuck, or a Postgres cluster has no recoverability point + +**Alerts:** `CNPGWALArchiveBacklog` (warning, 1h) and `CNPGNoRecoverabilityPoint` (warning, 6h) +**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg` +**Expressions:** +``` +max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 0 +max by (namespace, job) (cnpg_collector_first_recoverability_point) == 0 +``` + +## What this means here — and why it is an AVAILABILITY incident, not just a backup one + +`pg_wal/archive_status` holds one marker per WAL segment: `.ready` = written but **not yet archived**, +`.done` = archived and safe to recycle. A healthy archiver drains `.ready` within seconds, so a +persistently non-zero ready count means `barman-cloud-wal-archive` is failing. + +**Two consequences, and the second one is the emergency:** + +1. There is no continuous archive, so **PITR is broken** (R-IDP-05 / R-STATE-03). +2. **`archive_mode=on` means Postgres will NOT recycle an unarchived WAL segment.** `pg_wal` grows + without bound until the data PVC fills and the postmaster **PANICs**. `pg_wal` shares the **data** + PVC on every Cluster here — no `walStorage:` is configured on any of them — so there is no separate + volume to absorb it. + +This is measured, not theoretical. Prod ran **from bootstrap to 2026-09-07** with +`endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`: all three Clusters at `ContinuousArchiving=False` with +`barman-cloud-wal-archive: exit status 4` and `firstRecoverabilityPoint: null`, **for 38+ hours, with no +alert of any kind** — found by hand. On `keycloak-db` that was 17 unarchived segments / ~268 MiB on a +5 GiB PVC, growing **~6.8 MiB/h ≈ 29 days to disk-full** (about 579 MiB already burned by 2026-09-07). +The rule recorded verbatim: *"Never leave a placeholder here, and treat `ContinuousArchiving=False` on +this Cluster as an availability incident, not a backup one."* + +`CNPGWALArchiveBacklog` is *"the one that would have caught the 2026-09-06 incident within the hour +instead of 38 hours later."* `> 0` with `for: 1h` — not a count threshold — because `for` requires the +condition at **every** evaluation for the full hour, so a write burst resets the timer while a genuinely +stuck archiver never returns to zero. A `> 5` threshold would have delayed `semaphore-db` indefinitely: +its backlog sits at 3 because it writes almost nothing. + +## The two alerts are NOT the same statement + +`CNPGWALArchiveBacklog` is about the **mechanism** and is per-**pod** (each instance has its own `pg_wal` +on its own PVC; a `> 0` shape is safe per-pod, and `pod` names the instance to go look at). + +`CNPGNoRecoverabilityPoint` is about **DR coverage** and is per-**CLUSTER**, grouped +`by (namespace, job)`. That grouping is a fix, dated 2026-09-12: only the **primary** reads the backup +catalogue, so a replica reports a hard `0` forever. Grouped by `pod`, this `== 0` fired **permanently on +all six replicas** while all three Clusters were healthy. Verified live: +old `max by (namespace, pod) (...) == 0` → 6 series (`keycloak-db-1/-3`, `nextcloud-db-1/-3`, +`semaphore-db-1/-3`); new `max by (namespace, job) (...) == 0` → **empty** — and not vacuously so: +forcing the metric to 0 for one primary yields exactly one series. `job` is the grouping label because +the metric carries **no `cnpg.io/cluster` label**; the operator creates one PodMonitor per Cluster, so +prometheus-operator derives `job="/"`: `keycloak/keycloak-db`, +`nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`. `namespace` alone happens to work today but +would let two Clusters in one namespace mask each other. + +**Practical note:** if you see this alert firing on *pods* rather than per cluster, the per-cluster +aggregation is not deployed on this cluster yet — check the deployed PrometheusRule before treating it +as a real loss of DR. + +**Severity is `warning` on purpose.** A database heading for disk-full deserves weeks of lead time, and +this repo has been burned by a permanently-firing `critical` (the `VPNTunnelDown` post-mortem). The +`critical` tier for this failure mode is `PersistentVolumeAlmostFull`, which fires only when the outage +is genuinely imminent. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +The three Clusters' archiving and recoverability status — the single most informative command: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint' +``` + +The archiver's actual error text — `exit status 4` and its cause live here: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -i archive | tail -30' +``` + +The backlog on disk, per instance: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c \ + "ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | grep -c ready; du -sh /var/lib/postgresql/data/pgdata/pg_wal; df -h /var/lib/postgresql/data"' +``` + +The backups that exist — **fully-qualified, or you will be told there are none**: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A; echo; \ + kubectl --context k3s-prod get scheduledbackups.postgresql.cnpg.io -A' +``` + +`kubectl get backup -A` prints "No resources found" on this cluster even with 27 CNPG backups present: +three CRDs register the kind `Backup` (`longhorn.io`, `postgresql.cnpg.io`, `velero.io`) and the short +name resolves to Longhorn's, which holds zero. Verified on prod 2026-09-13. **Never decide a recovery +action from the short name.** + +The object-store configuration, which is where the fault usually is: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak keycloak-db -o yaml | sed -n "/backup:/,/^ [a-z]/p"' +``` + +``` +grep -nE "CNPG_BACKUP_PATH|S3_BACKUP_ENDPOINT|S3_BACKUP_REGION|BACKUP_CREDENTIALS_SUFFIX" clusters/base/cluster-vars.yaml clusters/prod/kustomization.yaml +``` + +## Likely causes, most likely first + +### 1. A wrong or placeholder object-store endpoint / path / credential +The recorded cause of the 38-hour incident: prod ran with `endpointURL: +TBD-PROD-S3-BACKUP-ENDPOINT`. The placeholder was **deliberately loud** because *"base defaults are the +DEV values, so an OMITTED op means prod silently inherits a dev address — far worse than a failed +apply"*. Live prod values to expect: `CNPG_BACKUP_PATH = s3://k3s-prod-backup/barman`, +`S3_BACKUP_ENDPOINT = https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`, +`S3_BACKUP_REGION = auto`, `BACKUP_CREDENTIALS_SUFFIX = -prod`, retention `30d`, WAL compression gzip, +`maxParallel: 2`. Remediation: fix it in git and push; Flux applies and the archiver drains on its own. + +### 2. R2-specific S3 incompatibilities +Cloudflare R2 rejects things MinIO tolerates. The two that are mandatory elsewhere in this platform and +worth checking here: `region` must be exactly `auto`, and AWS-SDK checksum headers are rejected with +**HTTP 400 InvalidArgument**. See `/runbooks/backup-failure` for the full R2 story, including the Velero +plugin's `x-amz-tagging` `NotImplemented` failure. + +### 3. Credentials expired or mis-scoped +A SOPS-encrypted Secret selected by `BACKUP_CREDENTIALS_SUFFIX`. Standing rule from the OIDC incident: +**non-secret, environment-specific values must NOT live inside an encrypted Secret** — `${BASE_DOMAIN}` +substitution can never reach inside one. Keep endpoints and paths in cluster-vars, tokens in the Secret. + +### 4. Egress is blocked +Default-deny is the baseline. The database namespaces need outbound HTTPS to R2. Cross-read +`/runbooks/denied-flows`. + +### 5. A fresh Cluster that has not made its first base backup +`firstRecoverabilityPoint` reads null until the first base backup lands; the `for: 6h` exists exactly to +keep a legitimate cold start quiet. Base backups are staggered: keycloak 01:00, nextcloud 02:00, +semaphore 04:00 (Velero at 03:00). If a Cluster is hours old, wait. + +### 6. Scaling out made it worse +Noted honestly: scaling a Cluster while `archive_mode=on` with a failing `archive_command` +**multiplies the disk-full clock**. Do not add instances as a remedy here. + +## When to escalate or stop + +- **Treat `ContinuousArchiving=False` as an availability incident with a countdown.** Compute the runway: + free bytes on the data PVC ÷ WAL growth rate. At ~6.8 MiB/h on a 5 GiB PVC that is weeks; on a busier + database it is days. +- **NEVER delete WAL segments by hand.** It destroys the database. This is the one irreversible mistake + available in this runbook. +- **Do not delete the `.ready` marker files.** Same outcome. +- **Do not expand the PVC and stop there.** It buys time and hides the fault. +- **Do not scale the Cluster** while archiving is failing. +- **Verify before any restore** that `firstRecoverabilityPoint` is non-null and + `backups.postgresql.cnpg.io` shows completed backups. Prod's healthy state on 2026-09-13 was: + all three Clusters `ContinuousArchiving=True`, `LastBackupSucceeded=True`, `firstRecoverabilityPoint` + 2026-09-07, 27 backups (9 per cluster) completing daily, with one historical failure + (`semaphore-db-daily-20260905130918`). That is what "good" looks like. +- Escalate to the operator for credential or endpoint changes, and for any restore. + +--- +Source of truth: `runbooks/cnpg-archiving.md`. Index: `/runbooks/` +Records: `platform/cnpg/alerts.yaml` header (the 2026-09-06 incident, the measured fill rate, and the +2026-09-12 aggregation-scope fix with its live promtool proof); +`platform/keycloak/postgres-cluster.yaml`; commit `32d6766` (the R2 decision). diff --git a/runbooks/cnpg-backup-missed.md b/runbooks/cnpg-backup-missed.md new file mode 100644 index 0000000..8b506bd --- /dev/null +++ b/runbooks/cnpg-backup-missed.md @@ -0,0 +1,159 @@ +# cnpg-backup-missed — No recent CNPG base backup for a Postgres cluster + +**Alert:** `CNPGNoRecentBaseBackup` (warning, 2h) +**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg` +**Expression:** +``` +(time() - max by (namespace, job) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600) +and max by (namespace, job) (cnpg_collector_last_available_backup_timestamp) > 0 +``` + +## What this means here + +The newest completed **base backup** for one Postgres cluster is over 36 hours old — one missed daily run +plus slack. The daily `ScheduledBackup` has silently stopped producing backups, so the PITR window is +ageing out against the `30d` retention policy. + +**Nothing else covers this.** The Velero rules in `platform/monitoring/alerts.yaml` watch Velero's own +schedules, and **Velero does not back up these databases** — CNPG owns their DR path, because a Velero +PVC snapshot of a live Postgres is not a backup. So `/runbooks/backup-missed` being clear says nothing +about the databases, and vice versa. + +The three Clusters and their staggered base-backup times: **keycloak 01:00**, **nextcloud 02:00**, +**semaphore 04:00** (Velero's own daily runs at 03:00, deliberately between them). + +## Two expression details that matter when you read the alert + +**The `> 0` guard is load-bearing.** A cluster that has *never* had a backup reports `0`, and +`time() - 0` is about **57 years** — always true. Without the guard this rule would double-notify with +`CNPGNoRecoverabilityPoint`, which is the correct alert for "never had a backup". So this rule means +specifically: *this cluster has had at least one backup, and the newest one is stale.* + +**It is aggregated `by (namespace, job)`, not `by (namespace, pod)`** — fixed 2026-09-12 alongside +`CNPGNoRecoverabilityPoint`. `cnpg_collector_last_available_backup_timestamp` has the same primary-only +defect: measured on prod the same day, `keycloak-db-2` = 3.1h old, `nextcloud-db-2` = 2.1h, +`semaphore-db-2` = 0.1h, and **all six replicas = 0**, i.e. `time() - 0` ≈ **56.7 years**. Unlike its +sibling this rule was *not* firing — because the `> 0` guard was the single thing standing between it and +six more permanent false positives, which is far too thin a margin for a DR alarm. Aggregating `pod` away +removes the dependence: `max` picks the primary's timestamp as the cluster's answer (a demoted primary +drops to 0 and can never win the `max`; a fresher backup always beats a staler one). +`job` is the grouping label because the metric carries **no `cnpg.io/cluster` label** — the operator +creates one PodMonitor per Cluster, so prometheus-operator derives `job="/"`: +`keycloak/keycloak-db`, `nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`. + +**Severity is `warning` on purpose** — this repo has been burned by a permanently-firing `critical` +(the `VPNTunnelDown` post-mortem), and *"noise on a DR alarm is worse than no alarm at all"*. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +**Fully-qualified names, always.** `kubectl get backup -A` prints **"No resources found"** on this +cluster even with **27 CNPG backups** present: three CRDs register the kind `Backup` +(`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1`, `velero.io/v1`) and the bare short name resolves to +**Longhorn's**, which holds zero objects. Verified on prod 2026-09-13. Concluding "the backups are gone" +from that and acting on it is how a CLI quirk becomes data loss. + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CLUSTER:.spec.cluster.name,PHASE:.status.phase,START:.status.startedAt,STOP:.status.stoppedAt --sort-by=.metadata.creationTimestamp | tail -20' +``` + +Are the schedules there and enabled? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get scheduledbackups.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SCHEDULE:.spec.schedule,SUSPEND:.spec.suspend,CLUSTER:.spec.cluster.name,LAST:.status.lastScheduleTime' +``` + +The Cluster's own verdict (and the condition that tells you whether *archiving* is also broken): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIMARY:.status.currentPrimary,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint' +``` + +Why the last attempt failed — the barman error is in the primary's logs: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -iE "barman|backup|archive" | tail -30; echo ====; \ + kubectl --context k3s-prod -n cnpg-system logs deploy/cnpg-controller-manager --tail=120 2>&1 | grep -iE "backup|error" | tail -20' +``` + +## Likely causes, most likely first + +### 1. The ScheduledBackup is suspended, missing, or unapplied +Check `spec.suspend`, and whether the object exists at all for that Cluster. A `ScheduledBackup` that was +never applied is the "looks done, is absent" failure class this repo keeps finding; verify the Flux tier +that owns it is Ready (`/runbooks/flux-reconcile`) and not suspended (`/runbooks/flux-suspended`). + +### 2. The object store is rejecting the write +Same root cause family as `/runbooks/cnpg-archiving`: endpoint, path, region or credentials. Live prod +values to expect are `CNPG_BACKUP_PATH = s3://k3s-prod-backup/barman`, +`S3_BACKUP_ENDPOINT = https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`, +`S3_BACKUP_REGION = auto`, `BACKUP_CREDENTIALS_SUFFIX = -prod`. **If `ContinuousArchiving` is also +False, fix that first** — a cluster whose archiver is broken has a disk-full countdown running, which is +more urgent than a stale base backup. + +### 3. A failed run that nobody noticed +There is precedent: one historical failure, `semaphore-db-daily-20260905130918`. A single failed run does +not clear itself; the next night's run should. If two consecutive nights fail, the cause is +configuration, not luck. + +### 4. The primary changed and the new primary cannot write +The backup catalogue is the **primary's** job. After a switchover, a credential or path problem scoped to +one instance shows up here. Check which pod is `currentPrimary` and read that pod's logs, not another's. + +### 5. Node or storage trouble during the backup window +Base backups are I/O-heavy and run at 01:00-04:00 alongside Velero. Cross-read `/runbooks/node-io`. + +### 6. The metric is missing, not the backup +If `cnpg_collector_last_available_backup_timestamp` has no series, the rule cannot fire. Note that a +**related** blind spot is real and separate: the CNPG SQL exporter can fail while `up` stays 1 — see +`/runbooks/cnpg-exporter`. + +## When to escalate or stop + +- **A stale base backup narrows, but does not eliminate, recoverability** — provided WAL archiving is + healthy, PITR still works from the older base. That is why this is a warning. If **both** this and + `CNPGNoRecoverabilityPoint` / `CNPGWALArchiveBacklog` are lit, DR for that database is effectively + gone: escalate. +- **Never delete WAL, `.ready` markers, or anything under a Postgres data directory.** +- **Do not delete old CNPG Backup objects to tidy up.** Retention is `30d` and the operator manages it; + deleting a Backup object can orphan objects in R2. +- **Do not trigger a base backup as a reflex** on a cluster whose archiver is failing — it adds write + load to a volume that may already be filling. +- **Be honest about restore:** the CNPG restore path from R2 has not been rehearsed on this cluster + (`dr-restore` is authored and never executed). Free R2 egress makes a rehearsal cheap; it is + outstanding. A restore is an operator operation. + +--- +Source of truth: `runbooks/cnpg-backup-missed.md`. Index: `/runbooks/` +Records: `platform/cnpg/alerts.yaml` (the 2026-09-12 aggregation-scope fix with its live measurements, +the `> 0` guard rationale, and the staggered schedule). diff --git a/runbooks/cnpg-exporter.md b/runbooks/cnpg-exporter.md new file mode 100644 index 0000000..4edcea2 --- /dev/null +++ b/runbooks/cnpg-exporter.md @@ -0,0 +1,164 @@ +# cnpg-exporter — The CNPG metrics collector is erroring (a database is invisible) + +**Alert:** `CNPGMetricsCollectorErroring` (warning, 6h) +**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg` +**Expression:** `max by (namespace, pod) (cnpg_last_error) == 1` + +## What this means here + +**This is the monitor's own blind spot, for databases — a failure that looks exactly like health.** + +CloudNativePG publishes **two independent metric families** and only one of them depends on SQL: + +- `cnpg_collector_*` comes from the instance manager and is **always** present. +- `cnpg_pg_stat_*` (archiver, database, replication) comes from the exporter logging in to the + application database as the `cnpg_metrics_exporter` role. + +When that login fails, `cnpg_last_error` goes to 1 and **the entire `cnpg_pg_stat_*` family blanks for +that instance — while `up` stays 1 and `cnpg_collector_up` stays 1.** Nothing looks broken. The scrape +target is green. The dashboards have gaps that read as "no data" rather than "broken". + +**This is not hypothetical, and it shaped the whole alert file.** Measured live on 2026-09-06 on both +clusters, `cnpg_errors_total` carried: + +``` +pg_stat_archiver on db nextcloud: failed to connect to `user=cnpg_metrics_exporter database=nextcloud`: +FATAL: permission denied for database "nextcloud" (SQLSTATE 42501) +``` + +`nextcloud-db` had been publishing **no** `pg_stat_archiver` / `pg_stat_database` metrics at all. The +consequence recorded verbatim: *an archiving alert built on `pg_stat_archiver` — the obvious choice — +would have been silently blind to one of three prod databases.* That is why every DR rule in +`platform/cnpg/alerts.yaml` keys on `cnpg_collector_*` instead, and why this rule exists to alert on the +blind spot itself. + +**Scope: `pod` is kept here, deliberately and unchanged.** `cnpg_last_error` is genuinely per-instance — +each instance's own exporter reports its own SQL-collection health — and prod proved the metric +discriminates rather than tracking role: all **three** `nextcloud-db` pods (including primary `-2`) +reported 1 while all six `keycloak`/`semaphore` pods reported 0. There is no primary-only defect to fix, +and per-pod is the right granularity because the remedy is a **GRANT applied per database** and you want +to see it clear on every instance. + +`for: 6h` keeps a pod restart or a brief connection blip quiet. Severity stays `warning` so it never +becomes the permanently-firing `critical` this repo has been burned by. + +**Live state, measured 2026-09-13:** `max by (namespace,pod) (cnpg_last_error) == 1` returned **empty** +across the 9 instances that publish `cnpg_last_error` — so the `nextcloud` grant appears to have been +addressed. Re-measure rather than assuming either way. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which instances are erroring: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s -G --data-urlencode "query=max by (namespace,pod) (cnpg_last_error)" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "pod|\"1\""' +``` + +**The message itself** — `cnpg_errors_total` carries it in an `errorUserQueries` label. This is the one +command that tells you the cause outright: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s -G --data-urlencode "query=cnpg_errors_total" localhost:19090/api/v1/query | python3 -m json.tool | head -60' +``` + +Or straight from the instance's metrics endpoint: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n nextcloud-db exec nextcloud-db-2 -c postgres -- \ + curl -s localhost:9187/metrics | grep -E "^cnpg_last_error|^cnpg_errors_total|^cnpg_collector_up" | head -20' +``` + +Confirm which family is actually missing — this is what makes the blind spot visible: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + for q in "count(cnpg_collector_up)" "count(cnpg_last_error)" "count(cnpg_pg_stat_archiver_archived_count)" "count(cnpg_pg_stat_database_xact_commit)"; do \ + printf "%-50s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \ + | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done' +``` + +The exporter role's actual grants: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n nextcloud-db exec nextcloud-db-2 -c postgres -- \ + psql -tAc "select rolname, rolcanlogin from pg_roles where rolname = \"cnpg_metrics_exporter\";" \ + -c "select datname, has_database_privilege(\"cnpg_metrics_exporter\", datname, \"CONNECT\") from pg_database where datistemplate = false;"' +``` + +## Likely causes, most likely first + +### 1. The exporter role lacks CONNECT on the application database +The recorded cause. Signature: `SQLSTATE 42501`, `permission denied for database ""` in +`cnpg_errors_total`. Remedy: **grant the `cnpg_metrics_exporter` role CONNECT on the application +database.** Do it declaratively — CNPG can manage roles and the `managed.roles` / bootstrap surface in the +Cluster spec is where this belongs, so the grant survives a re-seed. A `psql` GRANT by hand fixes the +symptom on one instance and vanishes on the next bootstrap. + +### 2. Custom monitoring queries reference something that does not exist +CNPG lets a Cluster attach extra collector queries via a ConfigMap. A query against a missing table, +extension or column fails the whole collection pass for that instance. `cnpg_errors_total`'s label names +the failing query. + +### 3. The database is not accepting connections +A read-only/recovery state, `max_connections` exhausted, or the instance mid-restart. Cross-read +`/runbooks/keycloak-db` if the instance is `keycloak-db-*`. Note the deliberate setting +`dataDurability: preferred` (never `required`) exists so a degraded standby cannot block writes here. + +### 4. An upgrade changed the exporter's expectations +CNPG **1.30.0** is the verified operator version on both clusters. A major operator or Postgres bump can +change the collector's queries; if this started at an upgrade, read `/runbooks/update-rollback`. + +### 5. The instance is fine and the whole scrape is gone +Then `up` for that `job` would be 0 and this is the wrong runbook — `/runbooks/targets-down`. The precise +value of *this* alert is the case where `up` is **1** and the data is still missing. + +## When to escalate or stop + +- **Understand what you are currently blind to.** While this fires, `cnpg_pg_stat_archiver`, + `cnpg_pg_stat_database` and `cnpg_pg_stat_replication` are missing for that instance. The DR alerts in + `platform/cnpg/alerts.yaml` are deliberately built on `cnpg_collector_*` so they still work — **but do + not build any new alert or dashboard on `pg_stat_*` without checking this first.** That is the trap + this alert exists to mark. +- **Do not silence it.** It is a warning with a 6h `for` precisely so it can be left visible without + becoming noise, and it was a **true positive** for months. +- **Do not grant the exporter role broad privileges** to make it go away. CONNECT on the application + database is the whole requirement; anything more widens the blast radius of a metrics credential. +- **Do not restart the database** to clear it — the error is a grant, not a state. +- Escalate to the operator if the fix requires changing the Cluster's managed roles or bootstrap, since + that touches the database's declarative definition and its DR path. + +--- +Source of truth: `runbooks/cnpg-exporter.md`. Index: `/runbooks/` +Records: `platform/cnpg/alerts.yaml` (the 2026-09-06 `SQLSTATE 42501` measurement on `nextcloud-db`, the +two-metric-families finding, and the 2026-09-12 scope audit that deliberately kept `pod`). diff --git a/runbooks/cnpg-wal-growth.md b/runbooks/cnpg-wal-growth.md new file mode 100644 index 0000000..e3d9db2 --- /dev/null +++ b/runbooks/cnpg-wal-growth.md @@ -0,0 +1,157 @@ +# cnpg-wal-growth — pg_wal is not being recycled + +**Alert:** `CNPGWALVolumeNotRecycling` (warning, 2h) +**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg` +**Expression:** +``` +max by (namespace, pod) (cnpg_collector_pg_wal{value="count"}) + > max by (namespace, pod) (cnpg_collector_pg_wal{value="max"}) +``` + +## What this means here + +`pg_wal` is holding more WAL segments than `max_wal_size` allows for, which means **checkpoints are no +longer reclaiming space**. On CNPG the units are **segments of 16 MiB**: `max` is `max_wal_size` +(**64 = 1 GiB** on this platform) and `count` is what is currently sitting in `pg_wal`. + +This is the **storage escalation** of `/runbooks/cnpg-archiving`, not an independent fault. +`CNPGWALArchiveBacklog` is the day-one signal — it fires within an hour of the archiver getting stuck. +This one fires when the resulting backlog has grown large enough to be a *disk* problem. With +`archive_mode=on` a failing `archive_command` is exactly what this looks like after a while, and because +**`pg_wal` shares the data PVC on every Cluster here** (no `walStorage:` is configured on any of them) +it ends in the volume filling and the postmaster **PANICking**. + +It does **not** fire today and is not meant to: measured, prod `keycloak-db` holds 17 segments against +max 64 (dev held 36 against 64), and across all nine prod instances counts ran 16-37 against 64. At +prod's observed ~6.8 MiB/h, `keycloak-db` would reach 64 segments roughly **40 days before** its 5 GiB +PVC actually ran out — so this alert leads the PVC-fill alerts by weeks. + +**Why `pod` is kept here, and is required.** Every instance reports its own `count` and `max`, `pg_wal` +is per-instance on a per-instance PVC, and the comparison only means anything pod-against-itself — +aggregating `pod` away would compare one instance's `count` with another's `max`. A `>` shape is also +safe per-pod: no instance can sit permanently true. (Contrast the `== 0` and staleness rules in the same +file, which **must** aggregate `pod` away — that is the 2026-09-12 fix.) + +**One metric that looks right and is not:** the CNPG-native +`cnpg_collector_pg_wal{value="volume_size"}` is **NaN on both clusters**, precisely because `pg_wal` +shares the data PVC rather than having its own. Do not build anything on it. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which instance, and how far past `max_wal_size`: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s -G --data-urlencode "query=max by (namespace,pod) (cnpg_collector_pg_wal{value=\"count\"})" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "pod|\"1\""' +``` + +The same thing from inside the instance, which also shows you the archive backlog and the free space in +one view: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c \ + "du -sh /var/lib/postgresql/data/pgdata/pg_wal; \ + ls /var/lib/postgresql/data/pgdata/pg_wal | grep -c \"^0\"; \ + ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | grep -c ready; \ + df -h /var/lib/postgresql/data; \ + psql -tAc \"show max_wal_size; show archive_mode; select pg_walfile_name(pg_current_wal_lsn());\""' +``` + +Is archiving the cause? (It almost always is.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint; echo; \ + kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -i archive | tail -20' +``` + +Is a replication slot or a stale standby holding WAL instead? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- \ + psql -x -c "select slot_name, active, wal_status, safe_wal_size, restart_lsn from pg_replication_slots;" \ + -c "select application_name, state, sync_state, replay_lag from pg_stat_replication;"' +``` + +Compute the runway — free bytes on the PVC divided by the observed growth rate: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s -G --data-urlencode "query=predict_linear(kubelet_volume_stats_available_bytes{job=\"kubelet\",namespace=\"keycloak\"}[24h], 7*24*3600)" localhost:19090/api/v1/query' +``` + +## Likely causes, most likely first + +### 1. WAL archiving is failing (dominant cause) +Go to `/runbooks/cnpg-archiving` — that is the actual fix. Fix the `archive_command` and Postgres recycles +the backlog on its own, without any intervention in `pg_wal`. Historical precedent: prod ran 38+ hours +from bootstrap with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, all three Clusters at +`ContinuousArchiving=False` with `barman-cloud-wal-archive: exit status 4`. + +### 2. An inactive replication slot is pinning WAL +A standby that has been gone long enough leaves a slot holding `restart_lsn`, and Postgres keeps every +segment after it. `pg_replication_slots.wal_status` will read `extended` or `lost`. The healthy fix is to +bring the standby back (CNPG manages the slots), not to drop the slot by hand. Note +`podAntiAffinityType: required` means a lost instance stays **Pending** until a node returns — expected, +and accepted. + +### 3. Genuinely heavy write load +Then `count` exceeding `max_wal_size` is checkpointing lagging, not a leak, and it self-corrects. Confirm +that `archive_status/*.ready` is near zero: a large `pg_wal` with an empty ready queue is a busy +database, not a stuck archiver. + +### 4. `max_wal_size` was lowered +64 segments = 1 GiB is the configured value on both clusters. A reduction makes this rule fire without +anything actually being wrong. Check the Cluster's postgresql parameters — and note that under +`primaryUpdateMethod: switchover` the operator **rejects an update that changes `imageName` and a +postgresql parameter in the same reconcile**. + +### 5. The instance is a replica that fell behind +Replay lag holds WAL on the primary. `pg_stat_replication.replay_lag` shows it. + +## When to escalate or stop + +- **NEVER delete WAL segments or `archive_status` marker files by hand.** It destroys the database. This + is the single irreversible mistake available here, and the alert's own description says so: *"Fix + archiving — do NOT delete WAL by hand."* +- **Do not drop a replication slot** to free space without the operator: you may make a lagging standby + unrecoverable and force a full re-seed. +- **Do not expand the PVC and stop there.** It buys days and hides the fault. +- **Do not scale the Cluster out** while archiving is failing — that **multiplies** the disk-full clock. +- Escalate when the computed runway is under a week, or when `PersistentVolumeAlmostFull` + (`/runbooks/pvc-capacity`) is also firing for the same PVC — at that point an outage is imminent and a + Postgres data volume under 5% free will PANIC the postmaster. + +--- +Source of truth: `runbooks/cnpg-wal-growth.md`. Index: `/runbooks/` +Records: `platform/cnpg/alerts.yaml` (the measured segment counts, the NaN `volume_size` finding, and the +2026-09-12 scope audit that deliberately KEPT `pod` here). diff --git a/runbooks/denied-flows.md b/runbooks/denied-flows.md new file mode 100644 index 0000000..5843b58 --- /dev/null +++ b/runbooks/denied-flows.md @@ -0,0 +1,150 @@ +# denied-flows — A spike in policy-denied network flows + +**Alert:** `HubbleDeniedFlowSpike` (warning, 10m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.networking` +**Expression:** `sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5` + +## What this means here + +Cilium is dropping more than 5 flows/second on policy. East-west traffic is **default-deny** everywhere +(R-NET-01): a bare workload reaches DNS and nothing else, and every allow is declared per +source → destination → port (R-NET-02). So a denied-flow spike is almost always **a NetworkPolicy +regression** — something legitimate that is no longer allowed — and only rarely a probe. + +### Verify the alert can even fire (measured 2026-09-13) + +**This alert is currently INERT.** `count(hubble_drop_total)` returned **0 series** on prod: there is no +Hubble/Cilium scrape job among Prometheus's 19 jobs, so Hubble metrics are not being collected and this +rule evaluates against nothing. **Policy drops are unmonitored today** — you will hear about them as +"service X cannot reach service Y", and you diagnose them with `hubble observe`, below. See +`/runbooks/targets-down` for the full list of inert rule groups. + +### The regression class that has actually bitten this cluster + +**Cilium's CVE-2025-30162 hairpin change (the expensive one).** After the 1.16 → 1.20 upgrade, the +shared Gateway's L7 proxy re-checks a pod's egress policy against the destination's **real backend +identity**. So `toEntities: world` **no longer covers an in-cluster call to a `*.home.keithjackson.dev` +hostname** — the destination resolves to an in-cluster backend, not to `world`. Every app's SSO call +started returning **403 with `server: envoy`**. The fix is an explicit `toEndpoints` allow to the +Keycloak backend, which `catalog/render.py` now emits for every service with +`auth.mode ∈ {oidc, forward-auth}`. + +**The durable rule that follows: any new app → app-by-hostname dependency needs its own explicit allow.** +If a newly added or newly SSO-enabled service is 403ing or timing out, this is the first thing to check. + +A second recorded instance of the same class: Flux's `source-controller` was denied by +`gitea-allow-gateway-ingress`, whose ingress rule was `fromEntities: [ingress]` only — so an in-cluster +`wget` to Gitea simply timed out. And a matching diagnostic trap: **testing HTTPS from inside a pod is +misleading here**; test from outside with `curl --resolve`. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +See the actual drops — this is the tool, since the metric is not collected: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \ + hubble observe --verdict DROPPED --last 200 -o compact' +``` + +Narrow to a suspected pair: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \ + hubble observe --verdict DROPPED --from-namespace nextcloud --last 100 -o compact' +``` + +What policies apply to the source workload, and is the policy actually enforced? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get ciliumnetworkpolicies.cilium.io,networkpolicies.networking.k8s.io -A; echo; \ + kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg endpoint list | head -30' +``` + +Cilium's own drop counters, as a cross-check: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg metrics list | grep -i drop' +``` + +Is the 403 an Envoy/L7 verdict rather than an L3/L4 drop? (The hairpin class above shows up here, not in +`hubble observe`.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n logs --tail=50 | grep -iE "403|envoy|oidc|well-known"' +``` + +## Likely causes, most likely first + +### 1. A newly added or newly SSO-enabled service has no explicit allow to Keycloak +The CVE-2025-30162 hairpin class described above. Signature: HTTP **403 with `server: envoy`** on a call +to `keycloak.${BASE_DOMAIN}` or another `*.home` hostname from inside the cluster, while the same URL +works from outside. Remediation: the allow is **rendered**, not hand-written — set `auth.mode` in +`catalog/.yaml` and re-render so `render.py` emits the `toEndpoints` allow to the keycloak +backend. Do not patch a CiliumNetworkPolicy by hand; it will be overwritten. + +### 2. A platform component was moved or a namespace label changed +`fromEntities: [ingress]`-only ingress rules (the Gitea/source-controller case) break any in-cluster +client. Remediation: add the specific allow in the component's `networkpolicy.yaml`, in git. + +### 3. A missing dependency declaration in the catalog +`dependencies:` in a ServiceDefinition is what generates app→app allows (R-NET-02). A service talking to +something it never declared is denied by design. The declaration is the fix. + +### 4. A scrape target that nothing allows +Prometheus selects **every** ServiceMonitor in the cluster +(`serviceMonitorSelectorNilUsesHelmValues: false`), so a new workload's monitor is scraped immediately +and will be denied until the component allows ingress from `monitoring`. Cross-read +`/runbooks/targets-down`. + +### 5. An actual probe or scan +Possible, and it is why the alert is worded as a security signal (docs/07). Distinguish it by the +**source**: a spike from outside the cluster or from an unexpected identity, rather than one named +workload repeatedly failing one destination. Escalate rather than tuning policy. + +## When to escalate or stop + +- **Do not widen a policy to make drops stop.** Default-deny is a safety-critical requirement here + (R-NET-01 is flagged `safety_critical: true` in the requirement registry). The correct fix is the + narrowest explicit allow, authored in git, usually via the catalog so it is rendered consistently. +- **Do not hand-edit a rendered CiliumNetworkPolicy** — it comes from `catalog/render.py` and will be + reverted on the next reconcile. +- **Escalate if the source is unexpected.** An unexplained spike from outside the cluster is a security + event, not a configuration bug. +- If drops coincide with a Cilium upgrade, assume the hairpin/identity class above before assuming a + policy authoring mistake — the upgrade changed the *evaluation*, not the policy. +- Note the diagnostic trap when verifying a fix: probe from **outside** with `curl --resolve`, because + in-pod HTTPS tests to the platform's own hostnames are misleading. + +--- +Source of truth: `runbooks/denied-flows.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` (the Cilium CVE-2025-30162 world-hairpin / envoy 403 finding and its +`render.py` fix; source-controller denied by `gitea-allow-gateway-ingress`). diff --git a/runbooks/flux-reconcile.md b/runbooks/flux-reconcile.md new file mode 100644 index 0000000..52ec45d --- /dev/null +++ b/runbooks/flux-reconcile.md @@ -0,0 +1,160 @@ +# flux-reconcile — A Flux resource has failed to reconcile for 15 minutes + +**Alert:** `FluxReconciliationFailure` (warning, 15m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux` +**Expression:** `max by (namespace, name, kind) (gotk_reconcile_condition{type="Ready", status="False"}) == 1` + +## What this means here + +Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than +it sounds, because **`git push origin main` IS the deploy** — there is no other apply step. A stuck +Kustomization means git and the cluster have diverged and nothing will close the gap. + +The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` → +`platform-config-cdi` → `apps-wave-0/1/2`, plus the non-gating leaves `apps-vms`, `oauth2-proxy`, +`headlamp`, `homepage`, `crafty`, `runbooks`, `backup`. Which tier is failing decides the blast radius: + +- **`flux-system`, `cilium`, `platform`, `platform-config` are gating (`wait: true`).** One Failed + HelmRelease in the `platform` tier stalls **everything downstream** on "dependency not ready". That + is not theoretical: a crashlooping Headlamp HelmRelease once stalled `platform-config`, + `oauth2-proxy` and every apps wave (the "Headlamp gating stall"), and oauth2-proxy twice deadlocked + by **blocking its own fix** — it sat in the gating tier while needing Keycloak + trusted TLS + + split-horizon DNS, so `platform-config`, which applies the fix, could never reconcile. Both were + resolved by moving the leaf into its own `wait: false` tier. +- **The leaf tiers are `wait: false`** precisely so their unhealth can never gate anything. + +Prod's source is `http://gitea.gitea.svc.cluster.local:3000` — the **in-cluster Service**, on purpose. +It used to be a public hostname, and the apex flip taught the expensive version of that lesson: the +HTTPRoute serving the source's own hostname lives in a downstream tier that cannot reconcile while the +source is down, so `srcReady=False` / *"unable to list remote"* became a **deadlock**. If you are +tempted to point the GitRepository at a public name, don't. +**This alert is currently INERT.** `count(gotk_reconcile_condition)` and `count(gotk_suspend_status)` +both returned **0 series** on prod on 2026-09-13: there is no scrape job for the Flux controllers in +Prometheus's 19 jobs, so every `gotk_*`-based rule (`FluxReconciliationFailure`, +`FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, +`UpdateHealthGateFailed`) evaluates against nothing. **Flux health is therefore unmonitored today** and +must be read from the CRs directly, with the commands below. See `/runbooks/targets-down` for the full +list of inert rule groups. Adding the Flux controllers' ServiceMonitor is open work. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +The whole chain at a glance — this is the command to run first, every time: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \ + -o custom-columns=NAME:.metadata.name,PATH:.spec.path,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,REV:.status.lastAppliedRevision' +``` + +Is the SOURCE healthy? If not, nothing downstream can be: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n flux-system get gitrepositories.source.toolkit.fluxcd.io \ + -o custom-columns=NAME:.metadata.name,URL:.spec.url,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message' +``` + +HelmReleases (the usual culprit inside `platform`): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get helmreleases.helm.toolkit.fluxcd.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message' +``` + +The controller's own view, with the real error text: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n flux-system logs deploy/kustomize-controller --tail=80; echo ====; \ + kubectl --context k3s-prod -n flux-system logs deploy/helm-controller --tail=80' +``` + +## Likely causes, most likely first + +### 1. "no matches for kind X" — a CR applied before its CRD +The recurring structural failure here. Flux server-side-apply fails the **whole** apply, so a tier that +ships both a CRD-installing HelmRelease and CRs of that CRD can never converge — the CRD never gets +created because the apply that would create it also contains the CR. Symptoms: +`no matches for kind "ClusterIssuer"` / `"PrometheusRule"`. The standing rule: **any component shipping +CRs of a CRD it installs must split the CRs into `platform-config`** (`dependsOn: platform`), and +second-order CRs whose CRD only appears after a controller reconciles go in `platform-config-cdi`. +Remediation is a git change moving the resource; `bootstrap/standup.sh crds` is the belt-and-braces +repair for a cluster already wedged. + +### 2. A HelmRelease is genuinely failing its install/upgrade +Read the message. Known instances: the `headlamp-0.40.1` `-session-ttl` chart/image desync; Grafana 13 +OOM-looping after kps 65→88. Note `platform`'s `timeout: 5m` versus Longhorn's HelmRelease +`timeout: 15m` — **the tier can time out while a legitimate Longhorn upgrade is still running**. Give +it another interval before concluding it is broken. + +### 3. A `${TOKEN}` problem in post-build substitution +Substitution **fails open**: an undefined variable becomes the empty string and the Kustomization still +reports Ready — which is how `k8sServiceHost: ""` broke Cilium on 2 of 3 dev nodes on 2026-09-01. A +*parse* error is the loud version and does fail the whole Kustomization +(`unable to parse variable name`). If the message mentions envsubst or a variable, run the L0 check: +`conformance/tests/l0-static/flux-postbuild-substitution/check.sh`. Note **strict substitution became +default-on in kustomize-controller v1.9.0**, so a Flux upgrade flips this behaviour whether or not you +opt in. + +### 4. A remote kustomize base timed out +The kustomize-controller's git fetch has a 27s timeout and remote GitHub bases flake against it. +`external-snapshotter` was vendored into the repo for this reason; **`platform/kubevirt` still uses +remote bases**, so a `BuildFail` there is probably this. Remediation: vendor it the same way. + +### 5. SOPS decryption +A tier that carries an encrypted Secret needs its own `decryption` stanza patched in by the overlay, by +name. A newly split-out tier that inherited a secret but not the patch fails here. + +## Remediation notes + +- Everything durable is a **git change**. `flux reconcile` only re-triggers; it does not fix a bad + manifest, and it is blocked for agents here. +- Flux is **level-triggered**: an interrupted reconcile is retried, so a restart of a controller is + safe and often the fastest way past a wedged in-flight apply (operator action). +- `dependsOn` is **revision-gated**, not merely Ready-gated: a dependent cannot reconcile revision N + until its dependency has applied N. That is what makes the chain safe and also what makes one stuck + tier so expensive. + +## When to escalate or stop + +- **If `flux-system`, `cilium`, `platform` or `platform-config` is the failing tier, treat it as a + platform outage in progress** — everything downstream is frozen, including the fix. Get the operator. +- **Do not `flux suspend` a tier to make the alert stop.** That trades a visible failure for + `/runbooks/flux-suspended` and silent drift, which is how prod's whole `platform` tier ended up + suspended and left that way after the 2026-09-04 external-dns incident. +- **Do not enable `driftDetection.mode: enabled` on a CNI.** During the 2026-09-01 Cilium incident the + Helm storage manifest *was* the bad render, so drift correction would have actively re-asserted the + broken value. +- If the GitRepository itself is not Ready, check Gitea before Flux — and remember prod deliberately + reads from the in-cluster Service so that public DNS can never be in this path. + +--- +Source of truth: `runbooks/flux-reconcile.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` "2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium" (substitution +fails open), the Headlamp gating stall, the oauth2-proxy cold-start deadlock, and the 2026-09-13 +GitOps source deadlock. diff --git a/runbooks/flux-suspended.md b/runbooks/flux-suspended.md new file mode 100644 index 0000000..44addf4 --- /dev/null +++ b/runbooks/flux-suspended.md @@ -0,0 +1,130 @@ +# flux-suspended — A Flux resource has been suspended for an hour + +**Alert:** `FluxResourceSuspended` (warning, 1h) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux` +**Expression:** `max by (namespace, name, kind) (gotk_suspend_status) == 1` + +## What this means here + +Someone turned reconciliation off for a resource and it is still off an hour later. While suspended, +Flux neither applies new commits nor corrects drift — **git stops being the truth** for that subtree, +silently, and everything downstream of a suspended gating tier is frozen with it. + +Suspension is a legitimate emergency valve here and is used deliberately: `cilium` was split into its +own Kustomization partly so that `flux suspend kustomization cilium` becomes an option that does not +freeze the other eleven platform components. The problem is never the suspend; it is the **forgotten** +suspend. Two real cases: + +- After the 2026-09-04 external-dns incident, prod's whole `platform` tier and everything downstream + were suspended **and left suspended** (`infra/PROD-PROVISIONING.md` §10). That suspension is also + the only reason a second bug never fired: `platform-config` would otherwise have requested a + `*.home.keithjackson.dev` wildcard from letsencrypt-prod for a certificate dev already held, and the + two clusters would have raced each other's `_acme-challenge` TXT records on one zone — neither would + reliably solve DNS-01. +- **Velero was found suspended live, with the suspension never recorded in git**, and resumed with + `flux resume`. A backup controller that is not reconciling is the definition of a silent DR gap. + +The 1h `for` is the point: short suspensions during maintenance are expected and should not notify. +**This alert is currently INERT.** `count(gotk_reconcile_condition)` and `count(gotk_suspend_status)` +both returned **0 series** on prod on 2026-09-13: there is no scrape job for the Flux controllers in +Prometheus's 19 jobs, so every `gotk_*`-based rule (`FluxReconciliationFailure`, +`FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, +`UpdateHealthGateFailed`) evaluates against nothing. **Flux health is therefore unmonitored today** and +must be read from the CRs directly, with the commands below. See `/runbooks/targets-down` for the full +list of inert rule groups. Adding the Flux controllers' ServiceMonitor is open work. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +What is suspended, across every Flux kind: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'for k in kustomizations.kustomize.toolkit.fluxcd.io helmreleases.helm.toolkit.fluxcd.io \ + gitrepositories.source.toolkit.fluxcd.io helmrepositories.source.toolkit.fluxcd.io \ + imagerepositories.image.toolkit.fluxcd.io imagepolicies.image.toolkit.fluxcd.io \ + imageupdateautomations.image.toolkit.fluxcd.io; do \ + echo "== $k"; kubectl --context k3s-prod get "$k" -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SUSPENDED:.spec.suspend 2>/dev/null | grep -v ""; done' +``` + +Is the suspension declared in git, or is it live-only drift? (This is the important question.) + +``` +grep -rn "suspend" clusters/ platform/ | grep -v "^Binary" +``` + +If it is not in git, it was applied by hand and nothing will restore it after a controller restart — +which means the resource may un-suspend itself at an unpredictable moment, or stay off forever. + +How far behind is it? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \ + -o custom-columns=NAME:.metadata.name,SUSPENDED:.spec.suspend,APPLIED:.status.lastAppliedRevision; echo; \ + kubectl --context k3s-prod -n flux-system get gitrepositories.source.toolkit.fluxcd.io \ + -o jsonpath="{.items[0].status.artifact.revision}{\"\n\"}"' +``` + +## Likely causes, most likely first + +### 1. A maintenance suspension nobody resumed +By far the most common, and the Velero case above is the archetype. Remediation (operator): +`flux resume kustomization ` — but **only after** reading the next paragraph. + +### 2. It is suspended for a reason that still applies +Before resuming, find out what would be applied. A tier suspended during an incident may be holding +back a change that caused the incident. The 2026-09-04 case is the cautionary one: resuming +`platform-config` while a second cluster still held the same wildcard certificate would have started an +ACME DNS-01 race on a shared Cloudflare zone. **Diff first**: compare +`status.lastAppliedRevision` against the GitRepository's current revision and read the commits in +between. + +### 3. A crashlooping leaf was suspended instead of being moved +The correct fix for "a leaf UI keeps failing and stalls the tier" is a `wait: false` tier of its own — +that is what happened to `oauth2-proxy`, `headlamp`, `homepage`, `crafty`, `velero` and `runbooks`. +Suspension as a permanent substitute leaves drift correction off for that component forever. + +### 4. Image automation +`ImageUpdateAutomation`/`ImagePolicy`/`ImageRepository` are **authored and deliberately disabled** here +(pending a Gitea write credential and `flux install --components-extra`), so if those show up, this is +expected state, not an incident. See `/runbooks/image-automation`. + +## When to escalate or stop + +- **Do not resume a gating tier (`cilium`, `platform`, `platform-config`) without the operator.** + Resuming applies every commit since the suspension in one pass, to the tier everything depends on. +- **Do resume `velero` promptly** if it is the suspended resource — a suspended backup controller is a + DR gap, and it has happened before. +- If the suspension is not in git, record it in git (or resume it). A live-only suspension is + undocumented state, which is the failure class this repo keeps finding. +- If you cannot determine *why* something was suspended, stop and ask. That is a safer outcome than + either resuming blind or leaving it off. + +--- +Source of truth: `runbooks/flux-suspended.md`. Index: `/runbooks/` +Records: `infra/PROD-PROVISIONING.md` §10 (the 2026-09-04 suspension and the queued ACME race); +`DECISIONS-LOG.md` (Velero found suspended with the suspension never in git). diff --git a/runbooks/image-automation.md b/runbooks/image-automation.md new file mode 100644 index 0000000..3d71e7b --- /dev/null +++ b/runbooks/image-automation.md @@ -0,0 +1,125 @@ +# image-automation — Flux's image-update-automation cannot write back to git + +**Alert:** `ImageUpdateAutomationFailing` (warning, 30m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.automated-updates` +**Expression:** `max by (namespace, name) (gotk_reconcile_condition{kind="ImageUpdateAutomation", type="Ready", status="False"}) == 1` + +## What this means here + +`ImageUpdateAutomation` is the write-back half of hands-off updates (ADR-0009): it commits new image +tags into git so Flux then deploys them. When it is Ready=False, **stable image advances silently +stop** — nothing breaks, nothing degrades, the platform just quietly stops receiving updates. That is +the R-UPD-02 failure mode this alert exists for. + +**Read this first: on this cluster, image automation is authored and DELIBERATELY DISABLED.** It has +been that way since 2026-08-24, pending (a) a Gitea write credential for the automation to push with +and (b) `flux install --components-extra=image-reflector-controller,image-automation-controller`. The +line is still commented out in `platform/kustomization.yaml`. So: + +- If this alert fires, the most likely explanation is that **the feature was partially enabled** — + someone uncommented the component or a rendered app started emitting `ImageRepository`/`ImagePolicy` + CRs whose controllers are not installed. +- Rendered apps emit those CRs by default; `catalog/render.py --no-image-automation` is the documented + workaround for exactly this. +- There is consequently **no real incident history** for this alert. Everything below is mechanism, not + post-mortem, and it says so honestly rather than inventing precedent. +**This alert is currently INERT.** `count(gotk_reconcile_condition)` returned **0 series** on prod on +2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every `gotk_*` rule +evaluates against nothing. Read the CRs directly with the commands below, and see +`/runbooks/targets-down` for the full list of inert rule groups. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Are the controllers even installed? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n flux-system get deploy | sort; echo; \ + kubectl --context k3s-prod api-resources --api-group=image.toolkit.fluxcd.io' +``` + +The automation objects and their messages: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get imageupdateautomations.image.toolkit.fluxcd.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,LASTRUN:.status.lastAutomationRunTime' +``` + +The controller log — the git error is always in here: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n flux-system logs deploy/image-automation-controller --tail=100 2>&1 | tail -40' +``` + +Is the repo configured with a write path at all? + +``` +grep -rn "image-automation\|ImageUpdateAutomation\|components-extra" platform/kustomization.yaml bootstrap/ clusters/ 2>/dev/null | head -20 +``` + +## Likely causes, most likely first + +### 1. The controllers are not installed (expected state here) +`api-resources --api-group=image.toolkit.fluxcd.io` returns nothing, or the deployments are absent. +Then the CRs are inert objects and the alert reflects a half-enabled feature. Remediation: either +finish enabling it (an operator runs `flux install --components-extra=...` — note that Flux's own +controllers were installed **imperatively** and carry no Flux ownership labels, so **nothing under +`clusters/**` can install a controller or change a controller flag**), or stop rendering the CRs with +`catalog/render.py --no-image-automation`. + +### 2. No git write credential +The automation must push. Prod's source is `http://gitea.gitea.svc.cluster.local:3000`, and the +read-only source credential is not a write credential. The controller log will say authentication +failed or permission denied. Remediation: a Gitea token in a SOPS-encrypted Secret referenced by the +`GitRepository`/automation, added in git — and note the standing rule from the OIDC incident: +**non-secret environment-specific values must not live inside an encrypted Secret**, because +`${BASE_DOMAIN}` substitution can never reach inside one. + +### 3. It cannot push because the branch moved +Automation commits to a branch; a force-push or a protected branch rejects it. The log names the ref. + +### 4. The commit is being made but the Kustomization is not picking it up +Then this is not the right runbook — go to `/runbooks/flux-reconcile`. + +## When to escalate or stop + +- **This is a warning with a 30m `for` for a reason: nothing is broken.** Do not treat it as an + outage. The consequence is "no new image versions", which is measured in days. +- **Do not create a Gitea write credential ad hoc.** It is a new secret in the DR path and belongs to + the operator, SOPS-encrypted, in git. +- **Do not enable the extra Flux controllers to make the alert stop.** Enabling automation on a + cluster that has never run it will start proposing image bumps across every workload at once; that is + a planned change, with `/runbooks/update-rollback` as its companion. +- If the alert is firing because the feature is half-enabled, the honest fix is to finish or to revert + — both are git changes and both want the operator's decision. + +--- +Source of truth: `runbooks/image-automation.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` (image automation authored-and-disabled since 2026-08-24; the Flux +controllers are outside Flux's own inventory). diff --git a/runbooks/image-policy-stalled.md b/runbooks/image-policy-stalled.md new file mode 100644 index 0000000..d61fd34 --- /dev/null +++ b/runbooks/image-policy-stalled.md @@ -0,0 +1,133 @@ +# image-policy-stalled — An ImagePolicy or ImageRepository cannot resolve its tag + +**Alert:** `ImagePolicyStalled` (warning, 1h) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.automated-updates` +**Expression:** `max by (namespace, name, kind) (gotk_reconcile_condition{kind=~"ImagePolicy|ImageRepository", type="Ready", status="False"}) == 1` + +## What this means here + +An `ImageRepository` cannot scan a registry, or an `ImagePolicy` cannot pick a tag out of what was +scanned. The effect is narrower than `/runbooks/image-automation`: automation itself is healthy, but +**this one service stops receiving updates** (R-UPD-01), while everything else continues. It is +therefore easy to miss, which is what the 1h `for` and the per-object labels are for. + +The same framing applies as for image-automation: **this feature is authored and deliberately disabled +on this cluster** and has been since 2026-08-24 (no Gitea write credential, and the +`image-reflector-controller` / `image-automation-controller` components are not installed — the line is +still commented out in `platform/kustomization.yaml`). Rendered apps emit `ImageRepository` and +`ImagePolicy` CRs anyway, so the objects can exist with no controller to reconcile them; +`catalog/render.py --no-image-automation` is the documented workaround. **There is no incident history +for this alert** — the section below is mechanism, honestly labelled. + +One live constraint worth knowing while reading registry errors: the images in this platform come from +`docker.io`, `ghcr.io`, `lscr.io`, `quay.io`, `registry.k8s.io` and `registry.gitlab.com`. Docker +Hub's anonymous rate limit is the most likely real-world cause of a scan failure once automation is +switched on. +**This alert is currently INERT.** `count(gotk_reconcile_condition)` returned **0 series** on prod on +2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every `gotk_*` rule +evaluates against nothing. Read the CRs directly with the commands below, and see +`/runbooks/targets-down` for the full list of inert rule groups. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which object, and the message (the message is nearly always the whole answer): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get imagerepositories.image.toolkit.fluxcd.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,IMAGE:.spec.image,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,TAGS:.status.lastScanResult.tagCount; echo; \ + kubectl --context k3s-prod get imagepolicies.image.toolkit.fluxcd.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,LATEST:.status.latestImage,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message' +``` + +Are the controllers installed at all? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n flux-system get deploy | grep -i image || echo "image controllers NOT installed (expected on this cluster)"' +``` + +The reflector's log: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n flux-system logs deploy/image-reflector-controller --tail=80 2>&1 | tail -30' +``` + +Can the cluster reach the registry at all? (Default-deny is the baseline; a new namespace may simply +not be allowed out.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get pods -A --field-selector status.phase!=Running | head; echo; \ + kubectl --context k3s-prod get events -A --field-selector reason=Failed --sort-by=.lastTimestamp | grep -i "pull\|registry" | tail -10' +``` + +## Likely causes, most likely first + +### 1. The image controllers are not installed (expected state here) +The CRs exist because `catalog/render.py` emits them; nothing reconciles them. Remediation: render with +`--no-image-automation`, or have the operator finish enabling automation. Note that the Flux +controllers were installed **imperatively** and are outside Flux's own inventory, so no git change can +install them. + +### 2. Registry rate limit +Docker Hub anonymous pulls/scans are limited per IP, and all three nodes share one WAN address. The +message says `toomanyrequests`. Remediation: a registry credential (SOPS-encrypted, in git), or scan +less often via `spec.interval`. + +### 3. Registry auth +A private repository with no `secretRef`, or an expired token. The message names it. + +### 4. The policy's range matches nothing +`ImagePolicy` semver ranges come from each ServiceDefinition's `update.range` (for example +`">=2024.0.0"` for pihole, `">=24.0.0"` for keycloak). A range that no published tag satisfies — or a +repository that publishes only `latest` (this platform has `lscr.io/linuxserver/jackett:latest` and +`plexinc/pms-docker:latest`) — leaves the policy with nothing to select. Remediation: fix the range in +`catalog/.yaml` and re-render; a `latest`-only image cannot participate in automation at all +and should not carry a policy. + +### 5. Egress is blocked for that namespace +Default-deny everywhere (R-NET-01) means outbound registry access is an explicit allow. Cross-read +`/runbooks/denied-flows`, and in particular the post-Cilium-upgrade finding that `toEntities: world` no +longer covers a call whose destination resolves to an in-cluster backend. + +## When to escalate or stop + +- **Warning, 1h, single service: this does not page.** The consequence is one workload pinned at its + current version. +- **Do not widen an `update.range` to make the alert clear.** The range is the service's update policy + (R-UPD-01); loosening it to satisfy a scanner is how an unintended major version arrives. +- **Do not add registry credentials ad hoc** — new secret, DR path, operator's call. +- If the same object has been stalled for days, the honest outcome is to remove its automation CRs + (`--no-image-automation`) rather than leave a permanently-failing object training everyone to ignore + the alert. Alert fatigue is a recorded, expensive failure mode on this cluster. + +--- +Source of truth: `runbooks/image-policy-stalled.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` (image automation authored-and-disabled; `--no-image-automation`); +`catalog/*.yaml` (`update.channel` / `update.range` per service). diff --git a/runbooks/keycloak-client-secrets.md b/runbooks/keycloak-client-secrets.md new file mode 100644 index 0000000..2cc2d64 --- /dev/null +++ b/runbooks/keycloak-client-secrets.md @@ -0,0 +1,197 @@ +# keycloak-client-secrets — The OIDC client reconciler is failing or has gone silent + +**Alerts:** `KeycloakClientSecretReconcilerFailing` (warning, 15m) and +`KeycloakClientSecretReconcilerStale` (warning, 2h) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity` +**Expressions:** +``` +max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"}) > 0 +absent(kube_job_status_succeeded{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"} == 1) +``` + +## What this means here + +These two rules watch the CronJob (every 15 minutes) that repairs OIDC client **secrets** and +environment-derived client **URLs** in the Keycloak realm. They exist because of the single worst +green-looking outage this cluster has had, and they are best understood through it. + +### The 2026-09-06 lockout — why "everything is green" means nothing here + +The Keycloak Operator resolves each OIDC client's secret from an `(env:NAME)` placeholder in the realm +import. **When it cannot resolve one, it stores the LITERAL placeholder text as the client secret and the +import Job still reports SUCCESS.** On prod, **all 12 clients** ended up holding the literal placeholder +and **every OIDC login on the cluster failed** with *"Invalid client or Invalid client credentials"*. +Meanwhile: the import Job succeeded, the CR rested at `Done=True / Started=False`, +`KeycloakDown` and `KeycloakDatabaseDown` were quiet, and **the conformance test that asserts +`Done=True` passed throughout the outage.** The only observable was a human failing to log in. Dev had +exactly one broken client (oauth2-proxy — its key was added *after* dev's import completed, and +**a completed import is never re-run**). + +The realm import is a **ONE-SHOT that fails open**: the operator logs *"Job already executed - not +recreating"*, so a git change to `keycloak-realm-import.yaml` does **not** re-import. + +### The second half: redirect URIs, same one-shot, one field over + +The realm import carries **26 `${BASE_DOMAIN}` references** (39 files under `platform/` and `apps/` +reference the knob). Substitution happened **once**, at the old hostname, and then froze. After the apex +flip, *"every OIDC login fails with a `redirect_uri` mismatch while the CR still reads `Done=True` and +nothing alerts."* The reconciler was extended with a second pass over `redirectUris`, `webOrigins`, +`rootUrl`, `baseUrl` and `adminUrl`, with desired state **generated** by +`ci/render-keycloak-client-urls.py` (gated by an L0 `--check`). + +Semantics worth knowing before you read a Job log: +- URLs are copied **verbatim, including `${BASE_DOMAIN}`**, and rendered by the *same* Flux + Kustomization as the import. +- **List fields compare order-insensitively (`sorted`, not `set`)** — measured, not defensive: Keycloak + genuinely reorders lists, and an order-sensitive compare would have rewritten `kubernetes` (8 URIs) and + `oauth2-proxy` (2) **every 15 minutes forever while reporting success**. +- A field the desired state omits is **UNMANAGED, never forced empty**. +- **Public clients' URLs are managed even though their secrets are not** (`kubernetes` is public, with + PKCE, so no kubeconfig holds a secret). +- The two passes **fail closed independently** — demonstrated live: pass 1 failed on an empty secrets + directory while pass 2 completed. + +### Why there are exactly two rules, and what each catches + +`...Failing` sees a run that *ran and failed* — unrepairable drift, or the admin API unreachable. It +fires alongside `KeycloakDown` during an IdP outage, deliberately: the reconciler genuinely is not +running, so treat `KeycloakDown` as the cause when both are lit. Self-clears when failed Jobs age out +(TTL 1h). + +`...Stale` sees the **healer being quietly absent** — the CronJob suspended, never applied by Flux, or +unable to schedule a pod at all. That is the "looks applied and isn't" failure this whole change exists +to end. It is also the rule that would catch the healer missing during a domain migration. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Does the CronJob exist, is it suspended, and when did it last succeed? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak get cronjobs.batch \ + -o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,SUSPEND:.spec.suspend,LAST:.status.lastScheduleTime,LASTSUCCESS:.status.lastSuccessfulTime' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak get jobs.batch --sort-by=.metadata.creationTimestamp \ + -o custom-columns=NAME:.metadata.name,SUCCEEDED:.status.succeeded,FAILED:.status.failed,START:.status.startTime | tail -15' +``` + +**Read the Job log — it names the pass, the client, and for a URL the exact before/after, and it never +prints a secret** (that is a design property, asserted by the L0 check): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak logs job/$(kubectl --context k3s-prod -n keycloak get jobs.batch \ + --sort-by=.metadata.creationTimestamp -o jsonpath="{.items[-1:].metadata.name}") --tail=200' +``` + +Is the IdP itself up? (If not, that is the cause; come back here after.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak get pods; echo; \ + kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak' +``` + +Confirm the declaration is self-consistent **offline**, before touching the cluster — this is the same +check the merge gate runs unconditionally (Stage 3d): + +``` +bash conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh +python3 ci/render-keycloak-client-urls.py --check +``` + +Confirm what a user actually experiences: + +``` +curl -s -o /dev/null -w '%{http_code}\n' --resolve keycloak.home.keithjackson.dev:443:192.168.2.20 \ + https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration +``` + +``` +infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod auth whoami' +``` + +(Expected on a working OIDC path: `oidc:` with `[oidc:platform-admins]`. From this workstation the +OIDC context hangs on a browser prompt — that is not evidence of breakage.) + +## Likely causes, most likely first + +### 1. Keycloak or its database is down, so the reconciler cannot reach the admin API +`...Failing` fires alongside `KeycloakDown` by design. Fix the IdP first: `/runbooks/keycloak-down`, +`/runbooks/keycloak-db`. Nothing here is actionable until the admin API answers. + +### 2. Unresolvable drift on one client +The log names the client and the pass. Most often a secret key missing from the SOPS Secret, or a client +in the realm that git does not describe. The contract: keys are **`-client-secret`**, and the +public `kubernetes` client declares **no** secret. Remediation is a git change to the declaration, then +let the CronJob heal it — the reconciler is compare-then-write and idempotent. + +### 3. A hostname migration left the realm behind +The `redirect_uri` mismatch case. Regenerate desired state with `ci/render-keycloak-client-urls.py` and +push; the reconciler's second pass applies it. Historical scale: re-rendering at the post-flip apex +produced **24 drifted fields across 12 clients**, with `kubernetes` correctly `ok` because its localhost +callbacks are not domain-derived. + +### 4. The CronJob is suspended or was never applied (`...Stale`) +Check `spec.suspend` and the Flux tier. The ordering requirement is a **real Flux `dependsOn` edge with +`wait: true`**, not a comment — the L0 check asserts that, because a comment is not an ordering guarantee. + +### 5. The pod cannot read its own secrets +A specific, recorded trap: the reconciler needs **`fsGroup: 65532`** with the client-secrets volume at +**`defaultMode: 0440`**. With `0400` and no fsGroup the container dies with `PermissionError` on **every** +run. If the log shows that, this is it. + +### 6. Everything is green and logins still fail +Then trust the human, not the dashboard — that is the entire lesson of 2026-09-06. Audit the live realm +directly with `kcadm.sh` inside `keycloak-0`. + +## When to escalate or stop + +- **Do not "fix" the realm by re-applying the import.** It is a one-shot: the operator logs *"Job already + executed - not recreating"* and nothing happens. The documented path is `kcadm.sh` inside `keycloak-0` + (admin credentials in Secret `keycloak-admin-credentials`; `kcadm` is present, `curl` and `python3` are + **not**) **and** fixing git so the change is reproducible. Doing only the live half creates drift; doing + only the git half changes nothing. +- **Never print or paste a client secret.** The reconciler deliberately logs URL values in full (they are + public hostnames) and never logs secrets. Keep that property. +- **Do not widen the reconciler's endpoint allow-list.** It is hard-scoped so that `/users`, `/groups`, + `/roles` and `/authentication` are unreachable — a blast-radius bound, not an oversight. +- **Do not disable or suspend the CronJob to stop the alert.** That is the healer, and `...Stale` exists + precisely because a suspended healer is invisible. +- Escalate for: rotating client secrets, editing the live realm, or anything touching MFA (a + conditional-OTP misconfiguration once 403'd every platform-admin login with + `"Invalid role name: null"`, fixed to `condUserRole: platform-admin`). + +--- +Source of truth: `runbooks/keycloak-client-secrets.md`. Index: `/runbooks/` +Records: `ci/l0.sh` Stage 3d and +`conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh` (the 2026-09-06 incident and the +reconciler's contract — 212 L0 assertions, an 84-assertion selftest, 12/12 injected regressions killed); +`DECISIONS-LOG.md` "Keycloak client REDIRECT URIs reconcile too — the same one-shot, one field over". diff --git a/runbooks/keycloak-db.md b/runbooks/keycloak-db.md new file mode 100644 index 0000000..6bfee83 --- /dev/null +++ b/runbooks/keycloak-db.md @@ -0,0 +1,165 @@ +# keycloak-db — Keycloak's PostgreSQL is unavailable + +**Alert:** `KeycloakDatabaseDown` (critical, 5m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity` +**Expression:** +``` +max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*", condition="true"}) == 0 +or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*"}) +``` + +## What this means here + +No ready `keycloak-db-*` pod for five minutes. `keycloak-db` is a **CloudNativePG Cluster** with +**3 instances** on Longhorn (3 replicas each), and it is the database the IdP cannot live without. The +`pod=~"keycloak-db-.*"` scoping matters for the same reason as in `/runbooks/keycloak-down`: the +namespace also holds the Keycloak server and operator pods, and an unscoped `max()` would be masked by +those. + +**Blast radius is the whole platform's identity:** Keycloak fails → kubectl OIDC, Grafana, Headlamp, +Gitea and forward-auth all fail. The certificate-based kubeconfig is an independent break-glass path and +still works. This page is deliberately served **without** forward-auth so it stays readable during +exactly this outage. + +## The HA properties you are relying on, and why they are set the way they are + +- **`instances: 3`.** All three CNPG Clusters used to run `instances: 1`, and all three primaries had + drifted onto one node — **on 2026-09-07 that node took SSO, Nextcloud and Semaphore down together.** + Scale-out to 3 is what gives you a standby to fail over to. +- **`podAntiAffinityType: required`**, not CNPG's default `preferred`. `preferred` is a scheduler + *hint* and on the path that matters — recreating an instance after node loss — it co-locates. + Recorded conclusion: *"`preferred` is how a 3-instance cluster decays back into the single-node + failure domain."* The accepted cost is that an instance stays **Pending** until a node returns. +- **`primaryUpdateMethod: switchover`.** The CRD default is `restart`, which is a delete-and-recreate of + the **primary** — tens of seconds with no Postgres, i.e. an SSO outage. Constraint to remember: under + `switchover` the operator **rejects an update that changes `imageName` and a postgresql parameter in + the same reconcile**. +- **Synchronous replication `method: any`, `number: 1`, `dataDurability: preferred` — never `required`.** + `required` is a strict quorum: with fewer than `number` healthy standbys, **writes BLOCK**. On this + database one degraded standby (a node reboot, a Longhorn rebuild, a Cilium restart) would become a + total write outage. `dataDurability` is only honoured while `standbyNamesPre/Post` are unset. + +**And the thing that is easy to get wrong:** *"Standbys replicate corruption and deletions faithfully; +only the Barman archive is a backup."* Three ready replicas are not a recovery plan. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +The Cluster's own verdict first — fully-qualified, because `cluster` is an ambiguous short name: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase' +``` + +Where the instances are, and whether they are all on one node (the 2026-09-07 defect): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak get pods -l cnpg.io/cluster=keycloak-db -o wide' +``` + +Why a pod is not ready: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak describe pod keycloak-db-2 | sed -n "/Events:/,\$p"; echo ====; \ + kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=80 2>&1 | tail -40' +``` + +The conditions that matter for recoverability — check these **before** any recovery action: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak keycloak-db \ + -o custom-columns=ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get backups.postgresql.cnpg.io -n keycloak' +``` + +`kubectl get backup -A` prints "No resources found" here even with 27 CNPG backups present — three CRDs +register the kind `Backup` and the short name resolves to Longhorn's, which has none. Never decide +anything from the short name. + +Storage underneath: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak get pvc; \ + kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy' +``` + +## Likely causes, most likely first + +### 1. A node is down or was rebooted +The instances live on Longhorn with hard anti-affinity, so a node loss takes one instance. With +`instances: 3` the cluster should survive it and promote; with `podAntiAffinityType: required` the lost +instance stays **Pending** until a node returns, which is expected and accepted. Go to +`/runbooks/node-down`. + +### 2. The data PVC filled up +The most dangerous cause, because the mechanism is a *backup* fault presenting as a *storage* failure. +With `archive_mode=on` and a failing `archive_command`, Postgres **refuses to recycle** unarchived WAL; +`pg_wal` shares the data PVC on every Cluster here (no `walStorage:` configured) and grows until the +volume is full and the postmaster **PANICs**. Measured on this very database: 17 unarchived segments / +~268 MiB on a 5 GiB PVC, growing ~6.8 MiB/h — about 29 days from 93% free to dead. Go to +`/runbooks/cnpg-archiving` and `/runbooks/pvc-capacity`. **Never delete WAL by hand.** + +### 3. A Longhorn volume is degraded or faulted +`/runbooks/longhorn-degraded`, `/runbooks/longhorn-faulted`. + +### 4. An upgrade is stuck mid-switchover +Signature: the operator refusing an update that changes `imageName` and a postgresql parameter together. +Split the change in git. See `/runbooks/update-rollback`. + +### 5. The database is up and Keycloak still fails +Then read `/runbooks/keycloak-down` and, more likely, `/runbooks/keycloak-client-secrets` — the +2026-09-06 lockout had all 12 OIDC clients holding literal placeholder secrets behind an entirely green +cluster. + +## When to escalate or stop + +- **Escalate immediately.** This is SSO's foundation; a 5m `for` on a `critical` is already generous. +- **Never change `dataDurability` to `required`** while trying to "make replication safer" — it converts + one degraded standby into a total write outage on the SSO database. +- **Do not delete a CNPG pod or PVC to force a rebuild.** The operator manages instance lifecycle; + deleting a PVC destroys that instance's data and, if it is the primary, may lose transactions that the + archive does not have. +- **Do not attempt a restore before checking `firstRecoverabilityPoint` and `ContinuousArchiving`.** Prod + ran 38+ hours from bootstrap with `ContinuousArchiving=False` and `firstRecoverabilityPoint: null` on + all three Clusters and nothing said so. Know what you actually have before you destroy what you have. +- **Standbys are not backups.** Recovery is from the Barman archive in R2 (`CNPG_BACKUP_PATH = + s3://k3s-prod-backup/barman`, 30d retention), which is an operator operation. + +--- +Source of truth: `runbooks/keycloak-db.md`. Index: `/runbooks/` +Records: commits `f338e91` (instances 3), `e8d8601` (required anti-affinity + switchover), `cac413e` +(sync replication, `dataDurability: preferred`); `platform/cnpg/alerts.yaml` header (the 2026-09-06 +archiving incident and the measured WAL fill rate). diff --git a/runbooks/keycloak-down.md b/runbooks/keycloak-down.md new file mode 100644 index 0000000..a758bb1 --- /dev/null +++ b/runbooks/keycloak-down.md @@ -0,0 +1,162 @@ +# keycloak-down — The Keycloak IdP is down (SSO is unavailable) + +**Alert:** `KeycloakDown` (critical, 5m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity` +**Expression:** +``` +max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+", condition="true"}) == 0 +or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+"}) +``` + +## What this means here + +No ready Keycloak **server** pod for five minutes. The regex is deliberately scoped to +`keycloak-[0-9]+` (the StatefulSet pods `keycloak-0`, `keycloak-1`) because the `keycloak` namespace also +holds the operator (`keycloak-operator-*`) and the database (`keycloak-db-*`); an unscoped `max()` would +be satisfied by those being ready and would never fire for the IdP itself. That scoping was a fix — the +original matched `.*postgres.*`. + +## Blast radius — read this before triaging anything else + +Keycloak is the root of trust. With SSO down, all of these are down or degraded **at once**: + +- **kubectl OIDC login** (the public `kubernetes` client with PKCE, groups → RBAC) +- **Grafana** (OIDC), **Headlamp** (forwards the user's id_token to the apiserver), **Gitea** +- **forward-auth** via oauth2-proxy, and the five per-app oauth2-proxy sidecars (pihole, jackett, + qbittorrent, radarr, sonarr) — these **fail closed**, 302ing to a Keycloak that is not there + +**You can still get in.** The certificate-based kubeconfig is untouched by the OIDC work and remains a +fully independent break-glass path — that is a deliberate design property, not an accident. From this +workstation the working non-interactive path is the dev-container kubeconfig; the host `k3s-oidc` +context will hang on a browser prompt, which is not the cluster being down. + +**And you can still read this page.** The runbook route is deliberately **public, with no +forward-auth**, precisely so that `keycloak-down` is readable by an operator who cannot log in. If that +ever changes, this runbook becomes unreachable exactly when it is needed. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Everything in the namespace, in one view — server, operator and database together: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak get pods -o wide; echo; \ + kubectl --context k3s-prod -n keycloak get statefulsets,deployments' +``` + +Is the database the real cause? (It usually is. Fully-qualified: `clusters.postgresql.cnpg.io`.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak \ + -o custom-columns=NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase' +``` + +The server's own complaint: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak logs keycloak-0 --tail=120 2>&1 | tail -50; echo ====; \ + kubectl --context k3s-prod -n keycloak describe pod keycloak-0 | sed -n "/Events:/,\$p"' +``` + +The operator's view of the CR: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak get keycloaks.k8s.keycloak.org -o yaml | grep -A20 "status:"' +``` + +Is it reachable at all, from outside, with an explicit SNI (in-pod HTTPS tests to these hostnames are +misleading here): + +``` +curl -s -o /dev/null -w '%{http_code}\n' --resolve keycloak.home.keithjackson.dev:443:192.168.2.20 \ + https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration +``` + +## Likely causes, most likely first + +### 1. The CNPG database is not available +Keycloak cannot start without Postgres. This is the highest-probability cause and it has its own runbook: +`/runbooks/keycloak-db`. Note that `keycloak-db` is a 3-instance CNPG Cluster with +`primaryUpdateMethod: switchover` and synchronous replication at `dataDurability: preferred` — +`preferred` specifically so that one degraded standby does **not** block writes, which on this database +would be a total SSO write outage. + +### 2. A node that held the Keycloak pods went away +Both Keycloak pods and all three `keycloak-db` instances previously ran on **one node** — all three CNPG +primaries had drifted onto `k3s-prod-2` — and **on 2026-09-07 that node took SSO, Nextcloud and +Semaphore down together**. The fixes were `instances: 3` on every Cluster and +`podAntiAffinityType: required` (CNPG's default `preferred` is only a scheduler hint and co-locates on +the path that matters — recreating an instance after node loss). Check `-o wide` above: if everything is +on one node, that is a live re-occurrence of the same defect. Go to `/runbooks/node-down`. + +### 3. A HelmRelease / operator upgrade went bad +Read `/runbooks/update-rollback`. Under `primaryUpdateMethod: switchover` the CNPG operator **rejects an +update that changes `imageName` and a postgresql parameter in the same reconcile** — a stuck reconcile +with that message is this. + +### 4. TLS to the issuer is broken +Keycloak's clients validate the issuer's certificate. While the wildcard was LE-**staging**, in-cluster +clients rejected it with `x509: certificate signed by unknown authority`. If `/runbooks/cert-issuance` or +`/runbooks/cert-expiry` is also firing, fix that first. + +### 5. Logins fail but Keycloak is UP +Then this is the **wrong alert and the wrong runbook**, and the distinction has cost this cluster dearly. +On 2026-09-06 every OIDC login on prod failed — *"Invalid client or Invalid client credentials"* — with +**all 12 clients holding the literal `(env:...)` placeholder as their secret**, while +`KeycloakDown`/`KeycloakDatabaseDown` were quiet, the import Job reported success, the CR read +`Done=True`, and the conformance test that asserts `Done=True` **passed throughout**. The only observable +was a human failing to log in. Go to `/runbooks/keycloak-client-secrets`. +Similarly, a `redirect_uri` mismatch after a hostname change is not an outage — same runbook. +And once, MFA broke every platform-admin login on its own: conditional-OTP used a non-existent config +key, giving `"Invalid role name: null"` → 403, fixed to `condUserRole: platform-admin`. + +## When to escalate or stop + +- **Escalate immediately.** SSO down is a platform-wide outage; treat it as such. +- **Use the certificate-based kubeconfig** (the dev-container path) rather than trying to fix OIDC login + first. Do not burn the outage on getting `kubectl` to authenticate through the thing that is down. +- **Do not edit the realm to "reset" anything.** The realm import is a **ONE-SHOT that fails open**: the + operator logs *"Job already executed - not recreating"*, so a git change to + `keycloak-realm-import.yaml` does **not** re-import. The documented path is `kcadm.sh` inside + `keycloak-0` (admin credentials in the Secret `keycloak-admin-credentials`; `kcadm` is present, + `curl`/`python3` are **not**) **and** fixing git for reproducibility. Doing only one of the two is how + drift is created. +- **Do not delete the Keycloak CR or the StatefulSet.** Recreating the IdP is a restore operation, not a + restart. +- **Do not touch `keycloak-db` volumes.** If the database is the cause, read `/runbooks/keycloak-db` and + `/runbooks/cnpg-archiving`; only the Barman archive is a backup, standbys replicate deletions + faithfully. +- Break-glass local admins exist for gitea, nextcloud and semaphore. Nextcloud has **no native + group→admin mapping**, so platform-admins there are promoted by hand. + +--- +Source of truth: `runbooks/keycloak-down.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` (the 2026-09-06 client-secret lockout; the 2026-09-07 single-node SSO +outage); commits `f338e91`, `e8d8601`, `cac413e` (CNPG instances/anti-affinity/switchover/sync). diff --git a/runbooks/kubevirt-control-plane.md b/runbooks/kubevirt-control-plane.md new file mode 100644 index 0000000..d4d55bb --- /dev/null +++ b/runbooks/kubevirt-control-plane.md @@ -0,0 +1,152 @@ +# kubevirt-control-plane — The KubeVirt control plane is degraded + +**Alert:** `KubeVirtControlPlaneDown` (critical, 10m) +**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt` +**Expression:** +``` +(sum(kubevirt_virt_controller_ready_status) == 0) +or (sum(kubevirt_virt_api_ready_status) == 0) +or (absent(up{job="kubevirt-prometheus-metrics"} == 1)) +``` + +## What this means here + +No `virt-controller` or no `virt-api` replica reports ready, so **VM lifecycle operations — create, start, +stop, migrate — will fail** (docs/06). Note carefully what this does *not* mean: **already-running VMs +keep running.** `virt-launcher` pods and their qemu processes are independent of the control plane. So +this is an "I cannot manage VMs" alert, not "the VMs are down". + +The `absent()` clause also fires if the whole KubeVirt metrics target disappears. The scrape job label is +**`kubevirt-prometheus-metrics`** (the Service name), **not** `kubevirt` — that precision is a fix: this +very alert once used a non-existent metric with the wrong job label and fired falsely (commit `b57b3e9`). +There is a sibling precedent too: a broken HTTP CDI ServiceMonitor caused a false `CDIOperatorDown` +(commit `6a7e9d1`). **So the first question with this alert is always whether it is telling the truth.** + +Live state, measured 2026-09-13: `kubevirt-prometheus-metrics` **is** among Prometheus's 19 scrape jobs +and `kubevirt_virt_api_ready_status` returns **2 series**, so the first two clauses are evaluable and +quiet. (`kubevirt_vmi_phase_count` was absent at the same moment — that metric only exists when VMIs do, +which affects `/runbooks/kubevirt-launch-failed`, not this rule.) + +Platform context: **KubeVirt v1.9.0 + CDI v1.66.0**, on real nested KVM (`/dev/kvm` present, vmx/svm +verified; the Proxmox nodes use `cpu_type=host`). The `kubevirt` and `cdi` namespaces deliberately carry +**no `app-namespace` label**, so Kyverno does not generate the default-deny baseline into them — it would +sever the virt control plane. Do not "fix" that by adding the label. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Is the control plane actually unhealthy, or is this a metrics artefact? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n kubevirt get pods -o wide; echo; \ + kubectl --context k3s-prod get kubevirts.kubevirt.io -A \ + -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,READY:.status.conditions[?\(@.type==\"Available\"\)].status' +``` + +Are the VMs themselves still running? (This is the reassuring half.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get vmi -A -o wide; echo; \ + kubectl --context k3s-prod get vm -A' +``` + +Is the scrape target up, and do the two metrics exist? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + for q in "up{job=\"kubevirt-prometheus-metrics\"}" "sum(kubevirt_virt_controller_ready_status)" "sum(kubevirt_virt_api_ready_status)"; do \ + printf "%-55s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \ + | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done' +``` + +Why a component is unready: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n kubevirt logs deploy/virt-controller --tail=80 2>&1 | tail -30; echo ====; \ + kubectl --context k3s-prod -n kubevirt logs deploy/virt-api --tail=80 2>&1 | tail -30; echo ====; \ + kubectl --context k3s-prod -n kubevirt get events --sort-by=.lastTimestamp | tail -20' +``` + +Is the Flux tier that owns KubeVirt healthy? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \ + -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message' +``` + +## Likely causes, most likely first + +### 1. The metrics target, not the control plane +Check the pods first. If `virt-api` and `virt-controller` are Running and Ready, this is a monitoring +fault — and there is direct precedent for false positives on this exact alert (`b57b3e9`) and its CDI +sibling (`6a7e9d1`). Remediation: fix the ServiceMonitor/metric in git. Cross-read +`/runbooks/targets-down`. + +### 2. A node that hosted the control-plane pods went away +`virt-controller` and `virt-api` are ordinary Deployments. Go to `/runbooks/node-down`. + +### 3. The KubeVirt Flux tier failed to reconcile +Specific known hazard: **`platform/kubevirt` still uses remote kustomize bases**, and the +kustomize-controller's git fetch has a **27s timeout** that remote GitHub bases flake against — a +`BuildFail` there is probably this. `external-snapshotter` was vendored into the repo for exactly this +reason; vendoring KubeVirt the same way is the durable fix. See `/runbooks/flux-reconcile`. + +### 4. Node pressure +Heavy sustained node load — TCG-emulated VMs are the recorded example — can degrade the Cilium datapath +into intermittent LB-VIP reachability problems, which looks like control-plane flapping. A restart +cleared it. Cross-read `/runbooks/node-io` and `/runbooks/node-cpu`. + +### 5. A bad operator upgrade +`/runbooks/update-rollback`. + +### 6. A NetworkPolicy severed the control plane +Only if someone added `homelab.keithjackson.dev/app-namespace=true` to `kubevirt` or `cdi`. Those +namespaces are deliberately unlabelled so Kyverno does not inject a default-deny baseline that would cut +virt-handler off. Reverting that label is the fix. + +## When to escalate or stop + +- **Prioritise honestly.** VM *lifecycle* is broken; running VMs are not. The UniFi VM keeps serving. + Treat this as urgent-but-not-catastrophic unless a VM also needs to start. +- **Do not drain a node to move virt pods.** This cluster does not drain, and a VM live-migrates on drain + **only** with per-VM `evictionStrategy: LiveMigrate` — a blanket setting would wedge drains for + RWO-disk VMs. +- **Do not delete a VMI or `virt-launcher` pod** while the control plane is down: nothing will bring the + VM back until `virt-controller` is ready again. +- **Do not add the `app-namespace` label to `kubevirt` or `cdi`** to "harden" them. +- Escalate for vendoring the remote bases or for any operator upgrade. If the alert is a false positive, + fixing the rule is itself worth a change — a permanently-firing critical is a recorded, expensive + failure mode here. + +--- +Source of truth: `runbooks/kubevirt-control-plane.md`. Index: `/runbooks/` +Records: commits `b57b3e9` (this alert's wrong metric/job label) and `6a7e9d1` (false `CDIOperatorDown`); +`DECISIONS-LOG.md` (KubeVirt v1.9.0 + CDI v1.66.0 on nested KVM; the remote-base 27s fetch timeout). diff --git a/runbooks/kubevirt-crashloop.md b/runbooks/kubevirt-crashloop.md new file mode 100644 index 0000000..97d7158 --- /dev/null +++ b/runbooks/kubevirt-crashloop.md @@ -0,0 +1,159 @@ +# kubevirt-crashloop — A VM's virt-launcher pod is restarting repeatedly + +**Alert:** `KubeVirtVMCrashLooping` (warning, 10m) +**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt` +**Expression:** `increase(kube_pod_container_status_restarts_total{pod=~"virt-launcher-.*"}[15m]) > 3` + +## What this means here + +More than three restarts of a `virt-launcher` pod in 15 minutes. The threshold is chosen to distinguish a +crash-**loop** from a one-off reboot: a guest that restarts once is not interesting, a guest that restarts +four times in a quarter of an hour is either panicking, being OOM-killed, or crashing qemu. + +Note what layer this watches: the **pod**, not the guest. `virt-launcher` wraps qemu, so a guest kernel +panic, an OOM kill of the launcher, and a qemu crash all present the same way here. The distinction comes +from the logs and the events, not from the alert. + +This is a `warning`, correctly: the VM tier is deliberately non-gating (`apps-vms` is `wait: false`) and a +crash-looping VM does not affect the container platform. The one caveat is that on this network the UniFi +VM is the home network's controller, so *that* VM crash-looping is user-visible. + +Two platform facts that shape the likely causes: + +- These are **HP EliteDesk 800 G5 Minis with ~22 GiB usable each**, running the whole platform. A VM is + the largest discretionary memory consumer on a node, so OOM is a live possibility rather than a + theoretical one. +- **`useEmulation: false` is set deliberately** — on KubeVirt v1.9, `true` silently forces TCG. A VM + running under TCG burns a whole core per vCPU, and heavy sustained TCG load has been observed to degrade + the Cilium datapath into intermittent LB-VIP unreachability (cleared by a restart). So "the VM is + unstable **and** the network is flaky" is usually one cause, not two. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which launcher, how many restarts, on which node: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get pods -A -l kubevirt.io=virt-launcher -o wide; echo; \ + kubectl --context k3s-prod get vmi -A -o wide' +``` + +Why it died — the previous container's log is the one that matters: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n unifi logs -l kubevirt.io=virt-launcher --previous --tail=120 2>&1 | tail -60' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n unifi describe pod -l kubevirt.io=virt-launcher 2>&1 | sed -n "/Last State/,\$p" | head -40' +``` + +Was it OOM-killed? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get events -A --field-selector reason=OOMKilling --sort-by=.lastTimestamp | tail -10; echo; \ + kubectl --context k3s-prod top pods -A --sort-by=memory | head -10' +``` + +``` +ssh k3s-prod-1 'dmesg -T | grep -iE "oom|killed process" | tail -10; free -h; swapon --show' +``` + +(An empty `swapon --show` is what you want. On 2026-09-07 all three prod nodes came back from a reboot with +977 MiB of swap **active** while the role reported success — `SwapTotal == 0` is not evidence swap is off. +See `/runbooks/node-memory`.) + +Is it really TCG rather than KVM? + +``` +ssh k3s-prod-1 'ls -l /dev/kvm; grep -c -E "vmx|svm" /proc/cpuinfo; ps -eo pid,pcpu,comm | grep -i qemu' +``` + +Is the disk healthy underneath? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ + -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state | grep -v healthy' +``` + +## Likely causes, most likely first + +### 1. The guest is panicking or the guest OS is broken +Read the serial console via the previous container's log. A guest that panics at boot after a change is +usually a bad cloud-init or a bad image. Two precise traps here: **KubeVirt caps inline `userData` at 2048 +bytes** (the shipped payload is a deliberately minimal ~1.1 KB inline blob; `userDataSecretRef` was tried +and reverted), and **`apps-vms` must NOT carry `postBuild.substituteFrom`** — Flux's envsubst rewrote +`${...}` inside the cloud-init and corrupted it (commit `da1f833`). Do not restore substitution there for +consistency with the app waves. + +### 2. The launcher is being OOM-killed +On ~22 GiB nodes running the whole platform, a VM with a generous memory request plus Prometheus (3Gi) and +Grafana (768Mi) is tight. Remediation: lower the VM's memory in git, or address node memory +(`/runbooks/node-memory`). Check swap is genuinely off first — active swap makes the kubelet's accounting +lie and eviction misbehave. + +### 3. qemu is crashing +The launcher log will carry the qemu error. Most often a device or firmware mismatch: a UEFI image needs +`firmware.efi`, and `containerDisk` does not work here at all (it requires the ALPHA `ImageVolume` gate, +off on k3s — use a CDI DataVolume). See `/runbooks/kubevirt-launch-failed`. + +### 4. The disk is unavailable or degraded mid-flight +A `longhorn-vm` volume that went degraded or faulted takes the guest with it. Cross-read +`/runbooks/longhorn-degraded` and `/runbooks/longhorn-faulted`. Remember `longhorn-vm` is **Block mode +only** and `migratable: true`; a VM on plain `longhorn` cannot live-migrate. + +### 5. Node I/O pressure is starving the guest +A guest whose virtual disk cannot complete writes will wedge and be restarted. `/runbooks/node-io` — and +note the guest's writes land on the same partition as the embedded etcd WAL, so a thrashing VM is a +control-plane risk, not just its own problem. + +### 6. It is running under TCG +`useEmulation: true` forces it on v1.9. Confirm with `/dev/kvm` presence and qemu CPU usage. Fix in git. + +## When to escalate or stop + +- **Bound the damage first.** If the VM is thrashing the disk, its restarts are loading the node that + etcd writes to. If `/runbooks/node-io` is also firing, stopping the VM (an operator action on the `VM` + object, `spec.running: false` in git) is the right move — protecting the control plane beats keeping one + VM trying. +- **Do not delete the DataVolume or the PVC.** A re-import is hours of I/O on the shared partition. +- **Do not `kubectl drain`** the node to relocate it. This cluster does not drain, and a VM live-migrates + on drain **only** with per-VM `evictionStrategy: LiveMigrate`; a blanket setting would wedge drains for + RWO-disk VMs. +- Escalate if the guest needs in-guest repair — that is console work, and for the UniFi VM it is also + home-network-visible. +- If the alert is firing for a `virt-launcher` in a leaked test namespace, deleting that namespace is the + fix; leaked TCG test namespaces have caused API blips here before. + +--- +Source of truth: `runbooks/kubevirt-crashloop.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` Session 7 (`useEmulation`, `firmware.efi`, `containerDisk`/ImageVolume); +commit `da1f833` (cloud-init corrupted by envsubst); the 2026-09-07 swap-persistence finding. diff --git a/runbooks/kubevirt-launch-failed.md b/runbooks/kubevirt-launch-failed.md new file mode 100644 index 0000000..06e3c6b --- /dev/null +++ b/runbooks/kubevirt-launch-failed.md @@ -0,0 +1,169 @@ +# kubevirt-launch-failed — One or more VirtualMachineInstances are in the Failed phase + +**Alert:** `KubeVirtVMLaunchFailing` (critical, 5m) +**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt` +**Expression:** `sum by (namespace) (kubevirt_vmi_phase_count{phase="Failed"}) > 0` + +## What this means here + +A VMI reached the `Failed` phase — a scheduling, launch or qemu failure. The IaaS path is broken for that +workload, which is why it is page-worthy even though the VM tier is deliberately **non-gating** +(`apps-vms` is `wait: false`, `dependsOn: platform-config`, because a VM's image import and in-guest +install are legitimately long-not-Ready). + +What actually runs here: the `unifi` namespace holds the UniFi VM (rootdisk 40 GiB, Block mode on +`longhorn-vm`). **UniFi OS Server is not yet installed in the guest** — that is known, documented state, +not a launch failure. + +**Live note, measured 2026-09-13:** `kubevirt_vmi_phase_count` returned **0 series** on prod. That metric +only exists while VMIs do, so this alert is currently evaluating against nothing. Confirm the real state +from `kubectl get vmi -A`, not from this alert's silence. (`kubevirt-prometheus-metrics` *is* a live scrape +job and `kubevirt_virt_api_ready_status` does publish, so the target is up — see +`/runbooks/kubevirt-control-plane`.) + +### The three k3s-specific launch failures this platform already hit and fixed + +Recorded from bring-up, and worth checking against any new VM definition because all three were +non-obvious: + +1. **`containerDisk` requires the ALPHA `ImageVolume` feature gate**, which is off on k3s. The fix was to + switch to a **CDI DataVolume on Longhorn** instead. A VM authored with `containerDisk` will not launch + here. +2. **A UEFI image needs `firmware.efi`** set on the VM. Without it, boot fails. +3. **`useEmulation: true` silently forces TCG on KubeVirt v1.9** — it is set to `false` deliberately. A VM + that appears to launch but burns a whole core per vCPU is this; it also degrades the Cilium datapath + under sustained load. + +Also recorded: a `virt-handler` certificate error seen during that work was **unrelated and cosmetic** — +do not chase it. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which VMI, in which namespace, and what phase: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get vmi -A -o wide; echo; \ + kubectl --context k3s-prod get vm -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,STATUS:.status.printableStatus,READY:.status.ready' +``` + +The reason — the VMI status conditions and the launcher pod's events are where it is stated: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n unifi describe vmi unifi 2>&1 | sed -n "/Status:/,\$p" | head -60' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n unifi get pods -l kubevirt.io=virt-launcher -o wide; echo; \ + kubectl --context k3s-prod -n unifi describe pod -l kubevirt.io=virt-launcher 2>&1 | sed -n "/Events:/,\$p"' +``` + +Is the disk there? (A missing or still-importing DataVolume is the most common cause.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get datavolumes.cdi.kubevirt.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PHASE:.status.phase,PROGRESS:.status.progress; echo; \ + kubectl --context k3s-prod get pvc -A | grep -Ei "rootdisk|dv|unifi"' +``` + +Is nested virtualisation actually available on the node it was scheduled to? + +``` +ssh k3s-prod-1 'ls -l /dev/kvm; grep -c -E "vmx|svm" /proc/cpuinfo' +``` + +Is the storage class right for a live-migratable VM? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get sc; echo; \ + kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy' +``` + +And whether the KubeVirt control plane can act at all: + +``` +infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kubevirt get pods' +``` + +## Likely causes, most likely first + +### 1. The disk is not ready — a CDI import that has not completed or has stalled +The VM cannot start without its DataVolume. Go to `/runbooks/cdi-import-stuck`. The specific recorded +failure: the **3-replica default class failed the UniFi 60Gi import with insufficient storage / +`ReplicaSchedulingFailure`**, which is why a dedicated single-replica **`longhorn-scratch`** class exists +and is referenced by the CDI CR's `scratchSpaceStorageClass`. + +### 2. The VM is authored in a way k3s does not support +The three items above: `containerDisk` without the ALPHA `ImageVolume` gate (use a CDI DataVolume), +missing `firmware.efi` for a UEFI image, or `useEmulation: true` forcing TCG. All are git changes to the +VM definition. + +### 3. The cloud-init payload is too large or corrupted +Two precise traps. **KubeVirt caps inline `userData` at 2048 bytes** — the original rich cloud-init +exceeded it; `userDataSecretRef` was tried and reverted, and the shipped state is a minimal ~1.1 KB inline +payload. And: **`apps-vms` must NOT carry `postBuild.substituteFrom`** — Flux's envsubst rewrote `${...}` +inside the cloud-init and corrupted it (commit `da1f833`). *Do not restore substitution there for +consistency with the app waves.* If a VM stopped booting right after someone "tidied up" the Flux +Kustomizations, this is it. + +### 4. It cannot be scheduled +No node with room, or a storage constraint. A live-migratable VM needs **`longhorn-vm`**: RWX, +**Block mode only** (migratable RWX is a raw block device, not the share-manager/NFS path), annotated +`storageclass.kubevirt.io/is-default-virt-class: "true"` — which is *not* the cluster default class. +A VM pointed at plain `longhorn` will not live-migrate. + +### 5. The CDI importer cannot open the raw Block device +`nonroot-devices: true` is set on the k3s servers because the non-root CDI importer must open a raw Block +device that Kubernetes never chowns for it (commit `df5979f`). A permission error in the importer on a +Block PVC is this. + +### 6. The KubeVirt control plane is down +Then nothing can launch. `/runbooks/kubevirt-control-plane`. + +## When to escalate or stop + +- **Scope it before escalating.** The VM tier is non-gating and does not affect the container platform. + One failed VM is not a platform outage — but the UniFi VM is the home network's controller, so treat + that one as user-visible. +- **Do not delete the VM or its DataVolume to "start clean".** A 60 GiB re-import is hours of I/O on disks + that share a partition with the embedded etcd WAL, and that is the mechanism behind the 2026-09-07 node + outage. Fix the definition, not the data. +- **Do not set `evictionStrategy: LiveMigrate` cluster-wide.** A blanket setting would **wedge drains** + for RWO-disk VMs; it is per-VM on purpose. +- **Do not add `postBuild.substituteFrom` to `apps-vms`.** It corrupts cloud-init. +- Escalate for anything that needs a fresh image import during a storage incident, and for changes to the + k3s server flags (`nonroot-devices`) — those are Ansible/operator work. + +--- +Source of truth: `runbooks/kubevirt-launch-failed.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` Session 7 (the three k3s-specific launch failures); commits `da1f833` +(cloud-init corrupted by envsubst) and `df5979f` (`nonroot-devices`); the `longhorn-scratch` / +UniFi 60Gi import finding. diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml new file mode 100644 index 0000000..7554231 --- /dev/null +++ b/runbooks/kustomization.yaml @@ -0,0 +1,85 @@ +--- +# Runbooks tier — reconciled by its OWN non-gating post-platform Flux Kustomization +# (clusters/base/runbooks.yaml, dependsOn platform-config, wait:false). Same placement and rationale as +# platform/homepage: a leaf web server must never sit in the gating `platform` tier, where a UI's +# unhealth once stalled the whole chain (DECISIONS-LOG: "Headlamp gating stall"). +# +# WHY THE MANIFESTS LIVE HERE, ALONGSIDE THE DOCS, RATHER THAN IN platform/runbooks/. +# The markdown is the source of truth and the task is to serve it verbatim, which means the ConfigMap must +# be GENERATED from these files — there must not be a second hand-maintained copy of 34 documents that can +# drift from the originals. `configMapGenerator` does that, and its `files:` paths are resolved relative to +# THIS directory. A `platform/runbooks/kustomization.yaml` would therefore need `../../runbooks/.md`, +# which depends on kustomize load restrictions being relaxed — true for Flux's kustomize-controller, +# refused by a plain `kustomize build`, and so a divergence between what the gate can verify and what the +# cluster does. Making this directory the Flux path removes the question entirely: no `../`, identical +# behaviour locally and in-cluster, one copy of every document. The serving manifests are tucked under +# `serving/` so `runbooks/*.md` stays exactly the flat set of pages the URLs name. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: runbooks +resources: + - serving/namespace.yaml + - serving/nginx-config.yaml + - serving/deployment.yaml + - serving/service.yaml + # default-deny baseline + Cilium gateway-ingress allow (DNS-only egress from the baseline). + - serving/networkpolicy.yaml + - serving/httproute.yaml # apex `/runbooks` prefix (binds the Gateway `https-apex` listener) + +configMapGenerator: + # ONE ConfigMap holding every runbook, keyed by the EXACT name the alert annotations use. + # + # THE KEYS ARE EXTENSIONLESS ON PURPOSE. The annotations are `https://${BASE_DOMAIN}/runbooks/` + # with NO `.md`, and this repo must not change them (they are spread across four alerts.yaml files that + # other work is editing). `=` maps `node-down.md` on disk to the key `node-down`, the volume + # mount turns each key into a file under the nginx root, and `/runbooks/node-down` becomes a plain static + # lookup — no rewrite rule, nothing to get subtly wrong at 3am. Drop the `=` and every key would + # carry `.md`, so every alert link would 404 while this file still looked correct. + # + # The generated name carries kustomize's content hash, so editing any runbook rolls the server + # automatically and a stale projection is impossible. + # + # ADDING A RUNBOOK: add the file AND its line here. The L0 check + # conformance/tests/l0-static/alert-runbooks-resolvable asserts both halves, because a file in git with + # no generator entry is reviewed, merged, apparently done — and still a 404 in the cluster. + - name: runbooks-content + options: + labels: + app.kubernetes.io/name: runbooks + files: + # The index, served at /runbooks/ (nginx `try_files /runbooks/README`). + - README=README.md + - backup-failure=backup-failure.md + - backup-missed=backup-missed.md + - backup-partial=backup-partial.md + - cdi-import-stuck=cdi-import-stuck.md + - cert-expiry=cert-expiry.md + - cert-issuance=cert-issuance.md + - cnpg-archiving=cnpg-archiving.md + - cnpg-backup-missed=cnpg-backup-missed.md + - cnpg-exporter=cnpg-exporter.md + - cnpg-wal-growth=cnpg-wal-growth.md + - denied-flows=denied-flows.md + - flux-reconcile=flux-reconcile.md + - flux-suspended=flux-suspended.md + - image-automation=image-automation.md + - image-policy-stalled=image-policy-stalled.md + - keycloak-client-secrets=keycloak-client-secrets.md + - keycloak-db=keycloak-db.md + - keycloak-down=keycloak-down.md + - kubevirt-control-plane=kubevirt-control-plane.md + - kubevirt-crashloop=kubevirt-crashloop.md + - kubevirt-launch-failed=kubevirt-launch-failed.md + - longhorn-capacity=longhorn-capacity.md + - longhorn-degraded=longhorn-degraded.md + - longhorn-faulted=longhorn-faulted.md + - monitoring-capacity=monitoring-capacity.md + - node-cpu=node-cpu.md + - node-disk=node-disk.md + - node-down=node-down.md + - node-io=node-io.md + - node-memory=node-memory.md + - pvc-capacity=pvc-capacity.md + - targets-down=targets-down.md + - update-rollback=update-rollback.md + - vpn-down=vpn-down.md diff --git a/runbooks/longhorn-capacity.md b/runbooks/longhorn-capacity.md new file mode 100644 index 0000000..7b20a1c --- /dev/null +++ b/runbooks/longhorn-capacity.md @@ -0,0 +1,154 @@ +# longhorn-capacity — A Longhorn volume or node is near its storage capacity + +**Alerts:** `LonghornVolumeCapacityHigh` (warning, 30m) and `LonghornNodeStorageHigh` (warning, 30m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn` +**Expressions:** +`(longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85` +`(longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85` + +## What this means here + +Either one volume is 85% full, or one node's Longhorn disk is 85% allocated. They are different +problems with different fixes, so read the labels: `volume` means a workload is filling its own PVC; +`node` means the cluster is running out of places to put replicas. + +The node case is bounded by settings that are already conservative: +`storageReservedPercentageForDefaultDisk: 20`, `storageMinimalAvailablePercentage: 25`, +`storageOverProvisioningPercentage: 200`, `defaultDataLocality: disabled`, hard node anti-affinity. +Measured headroom on prod is roughly **269 GiB scheduled against an 829 GiB ceiling**, so the node +alert firing means something has changed materially, not that the platform has slowly grown. + +The severe consequence of running out of node storage is that a degraded volume **cannot finish +rebuilding** — hard anti-affinity needs three distinct nodes with room, so one full node pins a +volume at 2 replicas indefinitely. Cross-read `/runbooks/longhorn-degraded`. +### Verify the alert can even fire (measured 2026-09-13) + +Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`, +`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`, +`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`, +`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`, +`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`. +There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this +rule depends on before you trust its silence: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \ + sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query' +``` + +**These alerts are currently INERT.** `count(longhorn_volume_capacity_bytes)` and +`count(longhorn_node_storage_capacity_bytes)` both returned **0 series** on prod on 2026-09-13 — +Longhorn ships no ServiceMonitor here. Volume-fill coverage actually comes from +`kubelet_volume_stats_*` instead, i.e. `/runbooks/pvc-capacity`, which is where the working alerts +are. Use this runbook for the Longhorn-side (node allocation) half, and expect to reach it by hand. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Node-level allocation (the half that has no substitute alert): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o yaml | grep -E "name:|storageAvailable|storageMaximum|storageScheduled|allowScheduling"' +``` + +Volume sizes versus actual usage: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ + -o custom-columns=NAME:.metadata.name,SPEC:.spec.size,ACTUAL:.status.actualSize,ROBUST:.status.robustness' +``` + +Which PVC each one is, so you know who is filling it: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SIZE:.spec.capacity.storage,SC:.spec.storageClassName' +``` + +The filesystem view, which is the alert that actually works: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s -G --data-urlencode "query=topk(10, 1 - kubelet_volume_stats_available_bytes{job=\"kubelet\"} / kubelet_volume_stats_capacity_bytes{job=\"kubelet\"})" localhost:19090/api/v1/query' +``` + +On the node itself: + +``` +ssh k3s-prod-1 'df -h /data; du -xh --max-depth=1 /data/longhorn/replicas 2>/dev/null | sort -h | tail' +``` + +## Likely causes, most likely first + +### 1. A CNPG database is not recycling WAL (check this FIRST) +The only cause here that grows without bound. `archive_mode=on` plus a failing `archive_command` means +Postgres will not recycle unarchived WAL; measured on prod `keycloak-db` at ~6.8 MiB/h with `pg_wal` +sharing the data PVC (no `walStorage:` is configured on any Cluster) — about 29 days from 93% free to +disk-full, ending in a postmaster PANIC. Go to `/runbooks/cnpg-archiving`. **Never delete WAL by +hand.** + +### 2. Snapshots and orphaned replicas accumulating +Longhorn snapshots are not free, and cleanup here is **manual**: `orphanAutoDeletion: true` in +`platform/longhorn/helmrelease.yaml` has been **silently inert since Longhorn 1.12** — it was renamed +`orphanResourceAutoDeletion` (now a list) and the chart drops unknown `defaultSettings` keys with no +error. Verify the key's current state in that file before citing it; then prune orphans and stale +snapshots from the Longhorn UI. + +### 3. A workload genuinely grew +Plex/Sonarr/Radarr **media** does not live on Longhorn — it is static NFS from +`192.168.2.102:/mnt/Storage/plex`, via the `bulk` tier. If a media volume is on `longhorn`, that is a +misconfiguration worth fixing, not a capacity problem: the `bulk` StorageClass is a stub +(`provisioner: nfs.csi.k8s.io`, `server:/share: REPLACE-ME`) and provisions nothing, so real bulk +storage is static NFS PVs. Otherwise expand the PVC — all Longhorn classes have +`allowVolumeExpansion: true`; change the size in git and push. + +### 4. Replicas piled onto one node +`replicaAutoBalance: best-effort` should prevent this, and hard anti-affinity bounds it to one replica +per volume per node. If one node is much fuller than the others, check whether a rebuild placed +replicas there while another node was down. + +### 5. The cluster has outgrown its disks +Prod's ceiling is ~829 GiB across three nodes. `docs/plans/longhorn-ssd-tiering.md` is the planned +answer (a ~51 GB hot tier: Prometheus, CNPG, Loki, Gitea). Operator planning work. + +## When to escalate or stop + +- **A node over 85% while any volume is degraded is urgent**, because the rebuild cannot complete. + Free space or restore the other node before anything else. +- **Never delete files under `/data/longhorn`** or inside a Postgres `pg_wal`. Both are irreversible. +- **Do not expand a PVC as a reflex** if the cause is unrecycled WAL — you are buying days and hiding + the real fault. +- Escalate for adding disks, the SSD tiering plan, or anything that needs a node reboot (each reboot + degrades all ~18 volumes and triggers a rebuild storm). + +--- +Source of truth: `runbooks/longhorn-capacity.md`. Index: `/runbooks/` +Records: `platform/longhorn/helmrelease.yaml` (the settings above and the inert `orphanAutoDeletion`); +`platform/cnpg/alerts.yaml` header (the WAL disk-full clock); `docs/BULK-NFS-STORAGE.md`. diff --git a/runbooks/longhorn-degraded.md b/runbooks/longhorn-degraded.md new file mode 100644 index 0000000..bd410b3 --- /dev/null +++ b/runbooks/longhorn-degraded.md @@ -0,0 +1,166 @@ +# longhorn-degraded — A Longhorn volume has been degraded for 10 minutes + +**Alert:** `LonghornVolumeDegraded` (warning, 10m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn` +**Expression:** `longhorn_volume_robustness == 2` (0=unknown 1=healthy 2=degraded 3=faulted) + +## What this means here + +A volume has fewer healthy replicas than it should. It is **still serving I/O** — that is the +difference between degraded and faulted (`/runbooks/longhorn-faulted`). What you have lost is +redundancy, and on a 3-node cluster with hard node anti-affinity +(`replicaSoftAntiAffinity: false`, `numberOfReplicas: 3`) one degraded volume means one more node +loss away from data loss. + +**Degraded is the NORMAL state after any node reboot on this cluster, and it is expensive.** +Recorded plainly in the stabilization notes: *"Longhorn on the HDD: every node reboot degrades all 18 +volumes → full rebuild (slow; 30%+ IO-delay). Roll reboots one node at a time; wait +`robustness=healthy` before the next."* The rebuild is the thing that hurts, not the degradation: +rebuild writes land on the same partition as the embedded etcd WAL (`/data/k3s` and `/data/longhorn` +share one device), and that is the mechanism behind the 2026-09-07 `k3s-prod-1` NotReady outage. So +the question to answer is never just "why is it degraded" but "is the rebuild about to take a node +out". Cross-read `/runbooks/node-io`. +### Verify the alert can even fire (measured 2026-09-13) + +Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`, +`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`, +`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`, +`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`, +`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`. +There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this +rule depends on before you trust its silence: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \ + sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query' +``` + +**This alert is currently INERT.** `count(longhorn_volume_robustness)` returned **0 series** on prod +on 2026-09-13: the whole `platform-health.longhorn` rule group has no data, because Longhorn ships no +ServiceMonitor here. That is a separate open defect, and the practical consequence is that **you will +probably learn about a degraded volume from an application symptom or from `volumes.longhorn.io`, not +from this alert**. Do not infer "no alert, so volumes are fine". +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +The truth about robustness, straight from the CRs (this works whether or not the metric does): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ + -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state,NODE:.status.currentNodeID,SIZE:.spec.size' +``` + +Which replicas are missing, and where: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get replicas.longhorn.io \ + -o custom-columns=NAME:.metadata.name,VOL:.spec.volumeName,NODE:.spec.nodeID,RUNNING:.status.currentState | sort -k2' +``` + +Which PVC/workload is affected: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SC:.spec.storageClassName' +``` + +Are the nodes and their disks schedulable, and is a rebuild in flight? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o wide; echo; \ + kubectl --context k3s-prod -n longhorn-system get settings.longhorn.io concurrent-replica-rebuild-per-node-limit -o custom-columns=NAME:.metadata.name,VALUE:.value' +``` + +And the pressure that rebuild is causing — the reason this alert matters: + +``` +ssh k3s-prod-1 'cat /proc/pressure/io' +``` + +## Likely causes, most likely first + +### 1. A node rebooted or was briefly NotReady (overwhelmingly the most common) +Expected, and it self-heals by rebuilding. Your job is to keep the rebuild from taking a node down. +Remediation (operator, mutating) — freeze rebuilds while replicas will **reattach** (an in-place +bounce): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \ + concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"' +``` + +Thaw to `"1"` — never higher — once every volume reads `healthy`. Use `1` rather than `0` when the +replicas are genuinely gone (an OS re-provision), so replenishment can actually proceed. +`concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and +held**; do not "fix" this by setting a knob that is already set. `replicaRebuildingBandwidthLimit` is +**v2/SPDK-only** and does nothing here; `rebuildConcurrentSyncLimit` is already at its floor of 1. +What was actually tightened after that incident was every *other* Longhorn I/O source: +`concurrentVolumeBackupRestorePerNodeLimit` and `snapshotHeavyTaskConcurrentLimit` 5 → 1. + +### 2. A node is still down +Then the rebuild cannot complete and this will not clear. Go to `/runbooks/node-down` first; there is +nothing to fix in Longhorn. + +### 3. A disk was replaced and Longhorn refuses it +Signature: `DiskFilesystemChanged` — *"record diskUUID doesn't match the one on the disk"*. +Remediation, in exactly this order (the validating webhook enforces it): **disable the disk → remove +the disk entry → re-add it**. Seen during the 2026-09-07 drive swap. + +### 4. No node can hold the replica (anti-affinity or space) +Hard anti-affinity means a 3-replica volume needs three distinct schedulable nodes. With one node +cordoned or its disk over `storageMinimalAvailablePercentage: 25`, the third replica cannot be +placed and the volume stays degraded indefinitely. Check `nodes.longhorn.io` scheduling and +`/runbooks/longhorn-capacity`. + +### 5. `replicaAutoBalance: best-effort` is shuffling replicas +Normal background behaviour, transient. If degradation follows a pattern of replicas moving and never +settling, that is worth an operator's attention. + +## When to escalate or stop + +- **If more than one volume is degraded and I/O-PSI on any node is climbing, freeze rebuilds first + and then get the operator.** That combination is the 2026-09-07 failure mode in progress. +- **Never reboot, cordon or re-provision a second node while any volume is degraded.** This is the + single most important rule in this runbook. +- **Do not `kubectl drain`.** This cluster deliberately does not drain: there is no Longhorn + `nodeDrainPolicy` configured anywhere in the repo, the upgrade plan rejects + system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD Longhorn", and the + node-loss conformance test evicts only its own writer pod. +- **Do not delete a degraded volume's replicas to "force a clean rebuild"** unless the operator agrees + — with 3 replicas and one already missing you are one mistake from `/runbooks/longhorn-faulted`. +- Escalate for: a node that will not come back, a disk replacement, or a volume degraded for more than + a few hours with all three nodes Ready (that is a Longhorn bug, not an operational state). + +--- +Source of truth: `runbooks/longhorn-degraded.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` "2026-08-31 — Post-upgrade stabilization" (the reboot tax) and +"2026-09-07 — k3s-prod-1 NotReady"; `docs/plans/cluster-upgrade-plan.md` §6.2/§7 (the freeze/thaw). diff --git a/runbooks/longhorn-faulted.md b/runbooks/longhorn-faulted.md new file mode 100644 index 0000000..3c475df --- /dev/null +++ b/runbooks/longhorn-faulted.md @@ -0,0 +1,160 @@ +# longhorn-faulted — A Longhorn volume is FAULTED (data at risk, volume offline) + +**Alert:** `LonghornVolumeFaulted` (critical, 1m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn` +**Expression:** `longhorn_volume_robustness == 3` + +## What this means here + +Faulted means Longhorn has **no usable replica**. The volume is offline; the workload's writes are +failing. This is the most serious storage state on the cluster and the `for: 1m` is deliberate — there +is no benign transient version of this. + +Understand what is and is not recoverable before you touch anything: + +- The volumes on this cluster are 3-replica with **hard** node anti-affinity, so faulted normally + means two or three replicas are gone at once — two nodes lost, or one node lost while another + volume's rebuild had not finished, or a disk that ate its replica. +- **Standbys and replicas are not backups.** Recorded verbatim for the databases and true for all of + this: *"Standbys replicate corruption and deletions faithfully; only the Barman archive is a + backup."* For a CNPG volume (`keycloak-db`, `nextcloud-db`, `semaphore-db`) the real recovery path + is the object-store archive, not Longhorn. For everything else it is Velero. +- If this is a **`longhorn-vm`** volume (RWX Block, `migratable: true`) the VM is down, not just the + disk. If it is **`longhorn-scratch`** (single replica, CDI scratch space) losing it is expected and + harmless — re-run the import. +### Verify the alert can even fire (measured 2026-09-13) + +Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`, +`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`, +`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`, +`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`, +`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`. +There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this +rule depends on before you trust its silence: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \ + sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query' +``` + +**This alert is currently INERT.** `count(longhorn_volume_robustness)` returned **0 series** on prod +on 2026-09-13 — Longhorn ships no ServiceMonitor here, so the whole `platform-health.longhorn` group +has no data. A faulted volume today announces itself as an application outage. That is a known open +defect; treat `volumes.longhorn.io` as the source of truth, not this alert's silence. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Stop and take inventory before changing anything. Which volume, which workload, which replicas exist: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ + -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state,NODE:.status.currentNodeID | grep -Ev "healthy"' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get replicas.longhorn.io \ + -o custom-columns=NAME:.metadata.name,VOL:.spec.volumeName,NODE:.spec.nodeID,STATE:.status.currentState,FAILED:.spec.failedAt | sort -k2' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SC:.spec.storageClassName | grep ' +``` + +What recovery material exists — **fully-qualified names, or you will be told there is nothing**: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get backups.velero.io -A; echo; \ + kubectl --context k3s-prod get backups.postgresql.cnpg.io -A; echo; \ + kubectl --context k3s-prod -n longhorn-system get backups.longhorn.io' +``` + +`kubectl get backup -A` prints "No resources found" on this cluster even when six Velero and 27 CNPG +backups exist. Never make a recovery decision from the short name. + +Node and disk state: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get nodes; kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o wide' +``` + +## Likely causes, most likely first + +### 1. Two nodes were unavailable at once +Usually a second reboot or cordon while the first node's rebuilds were still running. Remediation: +bring the nodes back. Longhorn will reattach if the replica data survived on disk — which is why you +must not delete replicas before checking. + +### 2. A disk failed and took its replica with it +`k3s-prod-1`'s original 2011 SSD had 61,440 reallocated sectors before it was replaced; the dev-era +Seagate had 448 current-pending plus 448 offline-uncorrectable. Check SMART on the node holding the +failed replica (`/runbooks/node-io` has the command). If the disk is bad, the replica is gone — the +path forward is restore, not repair. + +### 3. All replicas were evicted or deleted +Check whether a disk was disabled, a node removed from Longhorn, or a `replicas.longhorn.io` object +deleted. `staleReplicaTimeout: 30` (minutes) means Longhorn eventually discards replicas it considers +stale. + +### 4. It is a single-replica volume by design +`longhorn-scratch` is `numberOfReplicas: 1`. Losing it is not an incident — delete the DataVolume and +re-import. + +## Remediation, in order + +1. **Do not delete the volume, the PVC, or any replica.** A faulted volume with a stale replica is + recoverable; a deleted one is not. +2. If a node is down, restore the node (`/runbooks/node-down`). That is the highest-probability fix. +3. If Longhorn shows a replica with data but marked failed, an operator can salvage it from the + Longhorn UI (*Volume → Salvage*) — this is the documented path for "all replicas failed but the + data is still there". +4. If the volume backs a **CNPG database**, do not try to salvage Postgres files. Recover the Cluster + from the Barman archive (`/runbooks/cnpg-archiving` for how to confirm the archive is real first). +5. Otherwise restore from Velero. Note honestly: `dr-restore` is **authored and has never been + executed** (`conformance.home.k3s/never-executed`), and the R2 restore path has not been rehearsed. + Expect to be the first person to run it, and read `/runbooks/backup-failure` for the R2 plugin + caveats before you rely on a backup's metadata. + +## When to escalate or stop + +- **Get the operator now.** This is a critical, data-at-risk alert; it is not a runbook-and-carry-on + situation. Everything above is diagnosis, and steps 3–5 are operator actions. +- **Stop before any destructive command.** Specifically: no `delete pvc`, no `delete + volumes.longhorn.io`, no `delete replicas.longhorn.io`, no touching files under `/data/longhorn`. +- **Do not reboot anything** while a volume is faulted. +- If the affected volume is `keycloak-db`, expect a full SSO outage as a knock-on: kubectl OIDC, + Grafana, Headlamp, Gitea and forward-auth all depend on it. The certificate-based kubeconfig is an + independent break-glass path and still works. + +--- +Source of truth: `runbooks/longhorn-faulted.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` (drive failures 2026-08-22 and 2026-09-07; `dr-restore` never executed). diff --git a/runbooks/monitoring-capacity.md b/runbooks/monitoring-capacity.md new file mode 100644 index 0000000..440063c --- /dev/null +++ b/runbooks/monitoring-capacity.md @@ -0,0 +1,129 @@ +# monitoring-capacity — The monitoring stack's own PVC is filling up + +**Alert:** `MonitoringVolumeFillingUp` (warning, 15m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.monitoring-self` +**Expression:** `kubelet_volume_stats_available_bytes{namespace="monitoring"} / kubelet_volume_stats_capacity_bytes{namespace="monitoring"} < 0.15` + +## What this means here + +"Don't let the monitor blind itself." Prometheus or Loki is running out of disk. If it fills, you lose +the ability to see every other alert in this set — which is why `monitoring` gets its own rule instead +of sharing `/runbooks/pvc-capacity` (the two files partition the namespace space between them; the +thresholds are deliberately identical, 15% free for 15m). + +Live shape on prod: + +- `prometheus-...-prometheus-0` PVC **50Gi**, `retention: 30d`, `retentionSize: 45GiB` +- `storage-loki-0` PVC **50Gi** +- both on the `longhorn` class, 3 replicas + +`retentionSize: 45GiB` against a 50Gi PVC is intentional: Prometheus is *supposed* to plateau near the +high-water mark and then stay there. **So a Prometheus TSDB approaching 45 GiB is normal, not a leak.** +That is also why the predictive rule in `pvc-capacity-alerts.yaml` deliberately **excludes** the +`monitoring` namespace: a TSDB filling toward its retention limit looks exactly like an unbounded leak +to `predict_linear` until it plateaus. + +So the real question is *which* PVC and *whether it should be plateauing*: read the +`persistentvolumeclaim` label first. Loki has no size cap analogous to `retentionSize`, so a log +volume growing past 85% is the more likely genuine problem of the two. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring get pvc; echo; \ + kubectl --context k3s-prod -n monitoring get prometheus -o jsonpath="{.items[0].spec.retention} {.items[0].spec.retentionSize}{\"\n\"}"' +``` + +Actual on-disk usage from inside the pods: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- df -h /prometheus; \ + kubectl --context k3s-prod -n monitoring exec loki-0 -- df -h /var/loki' +``` + +Is Prometheus at its retentionSize (normal) or past it (a problem)? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + for q in "prometheus_tsdb_storage_blocks_bytes" "prometheus_tsdb_head_series" "rate(prometheus_tsdb_compactions_failed_total[1h])"; do \ + printf "%s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]])"; done' +``` + +Series growth is the usual driver — find the biggest contributors: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s "localhost:19090/api/v1/status/tsdb" | python3 -m json.tool | head -40' +``` + +## Likely causes, most likely first + +### 1. Prometheus has reached its retention high-water mark (expected) +`retentionSize: 45GiB` on a 50Gi PVC leaves 10% headroom, which is close to the 15% trigger. If +`prometheus_tsdb_storage_blocks_bytes` is near 45 GiB and steady, **this is the design working** and +the correct response is either to accept it or to raise both the PVC and `retentionSize` together in +`platform/monitoring/kube-prometheus-stack.yaml`. Raising retention without raising the PVC is how you +get a genuinely full disk. + +### 2. Loki log volume growth +Loki has no retentionSize equivalent configured. A noisy workload (a crashlooping pod, a chatty +reconciler) can grow it quickly. Remediation: find the noisy stream in Grafana → Explore, fix the +source, and set/lower Loki retention in git. + +### 3. Series cardinality grew after a chart or workload change +`serviceMonitorSelectorNilUsesHelmValues` and `podMonitorSelectorNilUsesHelmValues` are both `false` +here, which means **Prometheus selects every ServiceMonitor/PodMonitor in the cluster regardless of +labels**. That is convenient (no `additionalLabels` needed anywhere) and it also means a new +component's monitor is picked up automatically, cardinality and all. Check +`/api/v1/status/tsdb` for the top label pairs. + +### 4. Compaction is failing +`rate(prometheus_tsdb_compactions_failed_total[1h]) > 0` means blocks are not being merged or expired. +Usually an I/O problem underneath — cross-read `/runbooks/node-io`. + +### 5. The volume is on a node that is itself short of space +`/runbooks/longhorn-capacity` and `/runbooks/node-disk`. + +## When to escalate or stop + +- **Do not delete Prometheus TSDB blocks or Loki chunks by hand.** Lower retention and let the + component expire them; hand-deleting blocks can leave the TSDB unopenable. +- **Do not raise `retention` or `retentionSize` without raising the PVC in the same change.** +- If Prometheus is already full and crashlooping, you are blind: check the alerts you care about + directly against the CRs (`clusters.postgresql.cnpg.io`, `volumes.longhorn.io`, `kubectl get nodes`) + rather than waiting for alerts that cannot be evaluated. +- Escalate for PVC expansion — it needs node headroom, and this cluster has been memory- and + disk-constrained before (Prometheus was already raised 2 → 3Gi of RAM after the kps 65→88 upgrade). + +--- +Source of truth: `runbooks/monitoring-capacity.md`. Index: `/runbooks/` +Records: `platform/monitoring/pvc-capacity-alerts.yaml` (why monitoring is excluded from the +predictive rule); `DECISIONS-LOG.md` (Grafana OOM / Prometheus 2→3Gi during kps 65→88). diff --git a/runbooks/node-cpu.md b/runbooks/node-cpu.md new file mode 100644 index 0000000..6fdfd6c --- /dev/null +++ b/runbooks/node-cpu.md @@ -0,0 +1,121 @@ +# node-cpu — CPU pressure or run-queue depth high on a node + +**Alerts:** `NodeCPUPressureHigh` (warning, 15m) and `NodeCPULoadHigh` (warning, 15m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes` +**Expressions:** +`rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.15` +`node_load5 / count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1` + +## What this means here + +`NodeCPUPressureHigh` is PSI "some" for CPU: the fraction of wall-clock time during which at least +one task was stalled waiting for a CPU. Under 5% is healthy. Averaged CPU% — what `kubectl top` +shows — **hides this entirely**, which is why the alert uses PSI: it is the metric that surfaces the +"spikes cause blips of delay" symptom operators actually notice. + +`NodeCPULoadHigh` is 5-minute load average divided by core count. Sustained above 1 means the run +queue is longer than the number of CPUs. + +**Read them together.** CPU-PSI high *and* load high = genuine CPU contention. Load high *and* CPU-PSI +normal = the queue is full of tasks blocked on **I/O**, not CPU — go to `/runbooks/node-io`, which on +this cluster is the more dangerous of the two (etcd WAL and Longhorn replicas share one partition). + +Measured history: steady CPU PSI ran **15–23%** with load at or above core count, which is what drove +the node resize from **6 to 8 vCPU**; afterwards PSI measured ~4.5–7.7%. So 0.15 is not an arbitrary +threshold, it is the level the cluster actually lived at before the resize. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Distinguish CPU contention from I/O contention — do this first: + +``` +ssh k3s-prod-1 'cat /proc/pressure/cpu; echo; cat /proc/pressure/io; echo; uptime; nproc' +``` + +Who is burning it: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod top nodes; echo; kubectl --context k3s-prod top pods -A --sort-by=cpu | head -20' +``` + +``` +ssh k3s-prod-1 'top -b -n1 | head -20' +``` + +Are there VMs, and are any of them emulated (the worst case)? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get vmi -A -o wide' +``` + +``` +ssh k3s-prod-1 'grep -c -E "vmx|svm" /proc/cpuinfo; ls -l /dev/kvm' +``` + +## Likely causes, most likely first + +### 1. It is actually I/O, not CPU +Most common misread. A node blocked on disk has a long run queue and normal CPU-PSI. Confirm with +`/proc/pressure/io` and go to `/runbooks/node-io`. On 2026-09-07 the node that died had I/O-PSI full +at 11.36 while CPU looked unremarkable. + +### 2. A KubeVirt VM running under TCG emulation +`useEmulation: true` **silently forces TCG on KubeVirt v1.9** — it is set to `false` on purpose. A VM +that lost `/dev/kvm` or was authored with emulation burns a whole core per vCPU. Heavy sustained TCG +load has also degraded the Cilium datapath into intermittent LB-VIP unreachability (cleared by a +restart), so an "intermittent networking" report alongside this alert is probably one cause, not two. +Check `grep -c -E "vmx|svm" /proc/cpuinfo` (nested KVM is verified working; Proxmox nodes use +`cpu_type=host`). + +### 3. A Longhorn rebuild's checksumming +Rebuilds are I/O-heavy but not free on CPU. Same remediation as `/runbooks/longhorn-degraded`: freeze +with `concurrent-replica-rebuild-per-node-limit` `0`, thaw to `1`. + +### 4. Prometheus rule evaluation or a Loki query +`PersistentVolumeFillingUpPredicted` runs `predict_linear` over a 24h window across every PVC; a +Grafana panel doing the same interactively is heavier still. Bursty, self-limiting, usually harmless. + +### 5. Genuine capacity +If nothing is anomalous the node is simply undersized again. The 6 → 8 vCPU resize is the precedent. +A bpg cores change **reboots the VM**, so roll **one node at a time, waiting for Longhorn +`robustness=healthy` between each**. + +## When to escalate or stop + +- **If I/O-PSI is also elevated, stop treating this as a CPU problem** and switch to + `/runbooks/node-io`. Wrong runbook is the main risk here. +- Alone and steady, this alert is a capacity signal — schedule it, do not page on it. That is why it + is a warning with a 15m `for`. +- **Do not drain or cordon** to shed CPU load; this cluster does not drain. +- Escalate to the operator for a vCPU change (it reboots the node) or to kill a VM you did not create. + +--- +Source of truth: `runbooks/node-cpu.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` (node vCPU 6→8 and the post-resize PSI measurements; KubeVirt +`useEmulation` finding). diff --git a/runbooks/node-disk.md b/runbooks/node-disk.md new file mode 100644 index 0000000..d7419b5 --- /dev/null +++ b/runbooks/node-disk.md @@ -0,0 +1,127 @@ +# node-disk — A node filesystem is filling up + +**Alert:** `NodeDiskFillingUp` (warning, 15m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes` +**Expression:** `node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{...} < 0.10` + +## What this means here + +Less than 10% free on a real filesystem on a node, for 15 minutes. Read the `mountpoint` label +first, because on these nodes it tells you which of two very different problems you have. + +**`/data` is the one that matters.** Both the embedded **etcd WAL** (`/data/k3s`) and **every +Longhorn replica** (`/data/longhorn`) live on that single partition — +`node_baseline_etcd_source: shared-data-partition`, not a dedicated device, and the exit condition +for splitting them is still open (`infra/PROD-PROVISIONING.md` §2). If `/data` fills: + +- etcd cannot write its WAL. A control-plane member with a full WAL partition does not degrade + gracefully; it stops, and this is a 3-member cluster. +- Longhorn stops being able to hold replicas and volumes go degraded, which triggers rebuilds, which + need *more* space. + +So `/data` at 10% free is not a warning, it is the last comfortable moment to act. + +**`/` filling up** is usually logs, container images or a stuck journal. Lower stakes, but there is +history: in the dev era the host root filled to ~99% because VM disks sat under a snapper-snapshotted +subvolume, and the hypervisor **paused the k3s VMs on I/O error**. Recovery was deleting 11 snapper +timeline snapshots plus `virsh destroy` on the paused VMs (which were holding rsync-unlinked inodes): +99% → 30%. + +Longhorn's own guard rails are already tight and are why this alert usually fires before anything +breaks: `storageReservedPercentageForDefaultDisk: 20`, +`storageMinimalAvailablePercentage: 25`, `storageOverProvisioningPercentage: 200`. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which filesystem, on which node: + +``` +ssh k3s-prod-1 'df -h -x tmpfs -x overlay; echo; du -xh --max-depth=1 /data 2>/dev/null | sort -h | tail -10; echo; du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -10' +``` + +What Longhorn thinks it has: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,SCHED:.spec.allowScheduling; echo; \ + kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,SIZE:.spec.size,ACTUAL:.status.actualSize,ROBUST:.status.robustness' +``` + +Journal and image bloat on `/`: + +``` +ssh k3s-prod-1 'sudo journalctl --disk-usage; sudo crictl images | wc -l; sudo du -sh /var/lib/rancher/k3s/agent/containerd 2>/dev/null' +``` + +## Likely causes, most likely first + +### 1. Longhorn replicas grew, or replicas landed here after a rebuild +Hard node anti-affinity (`replicaSoftAntiAffinity: false`) means one replica per node per volume, but +`replicaAutoBalance: best-effort` moves them. Prod sits around 269 GiB scheduled against an 829 GiB +ceiling, so this is rarely tight — but a rebuild storm plus snapshots can spike it. +Remediation: check for orphaned replicas and stale snapshots in the Longhorn UI. Note +`orphanAutoDeletion: true` in `platform/longhorn/helmrelease.yaml` **has been silently inert since +Longhorn 1.12** — it was renamed `orphanResourceAutoDeletion` (a list) and the chart drops unknown +`defaultSettings` keys with no error. So orphan cleanup is manual. Verify the key's current state in +the file before you trust it. + +### 2. A CNPG database is not recycling WAL +The single most dangerous cause, because it is unbounded. With `archive_mode=on` and a failing +`archive_command`, Postgres **refuses to recycle** unarchived WAL segments and `pg_wal` grows forever +— on prod's `keycloak-db` that was measured at ~6.8 MiB/h, 17 segments / ~268 MiB, about 29 days to +fill a 5 GiB PVC. `pg_wal` shares the data PVC (no `walStorage:` is configured on any Cluster). +Check `/runbooks/cnpg-archiving` **before** you free any space, and **never delete WAL by hand** — +that destroys the database. + +### 3. Journal / container images on `/` +`journalctl --vacuum-size=200M` and pruning unused images are safe. Do not prune images for +workloads that may need to restart while the registry is unreachable. + +### 4. A Velero or CDI staging write +Velero's kopia uploads and CDI's scratch space both stage locally. `longhorn-scratch` +(single-replica) exists precisely because the 3-replica default failed the UniFi 60Gi import on +insufficient storage. + +### 5. The disk is smaller than the platform now needs +`docs/plans/longhorn-ssd-tiering.md` holds the planned hot tier (~51 GB: Prometheus, CNPG, Loki, +Gitea). Working set measured ~79 GB against ~323 GB free on SSD, so the "won't fit" blocker was a +measurement artifact. This is operator planning work, not an incident fix. + +## When to escalate or stop + +- **If the mountpoint is `/data` and free space is under 5%, treat it as an imminent control-plane + outage** and get the operator. etcd and Longhorn are both on it. +- **Never delete anything under `/data/longhorn` or a Postgres `pg_wal` directory by hand.** Both are + irreversible. Free space elsewhere, or fix the cause. +- **Do not `kubectl drain`** the node to move load off it — this cluster does not drain. +- If the cause is a CNPG archive failure, this runbook is the wrong one: go to + `/runbooks/cnpg-archiving`, which is the actual fix. + +--- +Source of truth: `runbooks/node-disk.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` "Disk/drive incident + k3s relocated to NVMe (2026-08-22)"; +`platform/cnpg/alerts.yaml` header (the WAL disk-full clock). diff --git a/runbooks/node-down.md b/runbooks/node-down.md new file mode 100644 index 0000000..732684c --- /dev/null +++ b/runbooks/node-down.md @@ -0,0 +1,152 @@ +# node-down — A node's node-exporter is unscrapable (the node may be gone) + +**Alert:** `NodeExporterDown` (critical, 5m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes` +**Expression:** `up{job="node-exporter"} == 0` + +## What this means here + +Prometheus has not scraped node-exporter on that instance for 5 minutes. The alert is worded +carefully — it says the *exporter* is down — because on this cluster the two cases look identical +from Prometheus and have opposite urgency: + +- **The node is genuinely gone** (powered off, kernel panic, network loss). +- **The node is up and the kubelet stopped posting status.** This is what actually happened on + **2026-09-07 at 06:05:21Z**: `k3s-prod-1` went NotReady with "Kubelet stopped posting node status" + while the host was up, pingable and SSH-able. Root cause was disk-I/O starvation of the embedded + etcd, not a dead machine. If this alert is firing alongside `node-io`, read `/runbooks/node-io` + first — that is the cause and this is the symptom. + +**Why it is critical rather than a warning.** All three nodes are control-plane + etcd members +(`k3s-prod-1` .10, `k3s-prod-2` .11, `k3s-prod-3` .12, kube-vip API VIP `192.168.2.9`). Losing one +leaves a 2-member etcd with no fault tolerance: the next loss ends quorum and the API server with it. +Losing a node also degrades every Longhorn volume that had a replica there, and every degraded volume +starts rebuilding — which loads the same disks etcd fsyncs to. One node down is not a routine event +on a 3-node cluster; it is one step from an outage. + +Two node-specific roles matter when you read which node it is: +- `k3s-prod-3` is the **sole VPN egress gateway**. Cilium OSS has **no egress-gateway failover**, so + while that node is down, VPN-classed egress is an outage, not a degradation. See `/runbooks/vpn-down`. +- `k3s-prod-2` is the cluster **founder** (`[k3s_prod_primary]` in `inventory/hosts.ini`). Do not + re-provision it from the inventory without reading `/runbooks/node-io` — re-provisioning a node + still listed as primary founds a **second cluster**. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Is the node actually NotReady, or is only the exporter unscrapable? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get nodes -o wide; echo; kubectl --context k3s-prod -n monitoring get pods -l app.kubernetes.io/name=prometheus-node-exporter -o wide' +``` + +Is it the API server or the node? (If this hangs, you have a bigger problem than one node.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get --raw /readyz?verbose 2>&1 | tail -20' +``` + +Is the host alive at all? Prod is directly on `192.168.2.0/24` — no NAT hop: + +``` +ping -c3 192.168.2.10 ; ssh k3s-prod-1 'uptime; systemctl is-active k3s; cat /proc/pressure/io' +``` + +The three lines that distinguish "etcd starved" from "machine dead": + +``` +ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|Kubelet stopped|leader"' +``` + +Etcd membership and leader — do this before touching anything: + +``` +ssh k3s-prod-2 'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"' +``` + +What did the node take with it? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ + -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy; \ + kubectl --context k3s-prod get pods -A --field-selector spec.nodeName=k3s-prod-1 -o wide | head -30' +``` + +## Likely causes, most likely first + +### 1. Disk I/O starvation wedged the kubelet (host still up) +The signature is `slow fdatasync took=3.56s` / `apply request took too long took=8.30s` / +`etcdserver: too many requests` in `journalctl -u k3s`, with `/proc/pressure/io` full avg300 well +above 1. On 2026-09-07 node 1 read **11.36** against 0.75 on node 2 and 1.29 on node 3. +Remediation: follow `/runbooks/node-io`. Recovery that day required a manual +`systemctl restart k3s` on the node (operator, on-host), after which 18 Longhorn volumes rebuilt and +pressure only fell when the rebuilds finished. + +### 2. The node rebooted (planned or otherwise) +Check `uptime`. A reboot is not harmless here: **every node reboot degrades all ~18 Longhorn volumes +and triggers a full rebuild**. If a reboot is intentional, roll one node at a time and wait for +`robustness=healthy` on every volume before the next, and freeze rebuilds first (see +`/runbooks/longhorn-degraded`). Note two hardware quirks recorded in `infra/PROD-PROVISIONING.md` +§0b: `k3s-prod-2` **hangs on a warm reboot while staying powered**, and all three nodes carry the EFI +memattr kernel bug that once presented `k3s-prod-1` as having 2.6 GiB usable of 16 GB (needs kernel +≥ 6.13). + +### 3. node-exporter itself is unhealthy, node fine +If `kubectl get nodes` says Ready and the pod is CrashLoopBackOff or Pending, this is a monitoring +fault, not a node fault. Downgrade urgency, but do not ignore it — while that exporter is down you +are blind to `node-io`, `node-memory`, `node-disk` and `node-cpu` on that node, which is exactly the +blindness that let 2026-09-07 arrive as a surprise. Related: `/runbooks/targets-down`. + +### 4. Network loss to the node +`ping` fails, SSH fails. Check the switch and the node's `eno1` (prod-3's +`/etc/network/interfaces` also holds a commented-out wireless stanza — inert, but worth scrubbing). +A node reachable by ping but not by the API is usually Cilium, not the network: heavy sustained node +load has been observed to degrade the Cilium datapath into intermittent LB-VIP unreachability, +cleared by a restart. + +### 5. Hardware +`k3s-prod-1`'s original 2011 SSD had 61,440 reallocated sectors and was replaced 2026-09-07. Check +SMART (`/runbooks/node-io` has the command). Hardware is an operator job. + +## When to escalate or stop + +- **STOP if two nodes are affected.** Two of three etcd members down means no quorum, no API server, + and no GitOps. Nothing in this runbook applies; get the operator. +- **Do not reboot, drain or cordon a second node** while any Longhorn volume is not `healthy`. This + cluster deliberately does not `kubectl drain` — the auto-drain of system-upgrade-controller was + rejected precisely because it fights slow-disk Longhorn. +- **Do not delete the Node object** to "clean up". That triggers replica deletion in Longhorn. +- Escalate to the operator for: a power-cycle, a disk replacement, or re-provisioning a node (the + inventory's `[k3s_prod_primary]` trap above makes re-provisioning a two-person job). +- If the node returns but volumes stay degraded for more than an hour, go to + `/runbooks/longhorn-degraded`. + +--- +Source of truth: `runbooks/node-down.md` in `home-network-k3s-impl`. Index: `/runbooks/` +Incident record: `DECISIONS-LOG.md` "2026-09-07 — k3s-prod-1 NotReady"; `infra/PROD-PROVISIONING.md` §0b/§0c. diff --git a/runbooks/node-io.md b/runbooks/node-io.md new file mode 100644 index 0000000..a696393 --- /dev/null +++ b/runbooks/node-io.md @@ -0,0 +1,179 @@ +# node-io — Disk I/O pressure / starvation on a node + +**Alerts:** `NodeIOPressureHigh` (warning, 10m) and `NodeIOPressureCritical` (critical, 5m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes` +**Expression:** `rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05` (warning) / `> 0.10` (critical) + +## What this means here + +PSI **full** for I/O: the fraction of wall-clock time in which *every* runnable task on the node was +blocked on disk, i.e. the machine did no useful work. It is not "the disk is busy" — it is "the node +is stalled". + +This alert exists because of a real outage it would have caught and could not. On **2026-09-07 at +06:05:21Z, `k3s-prod-1` went NotReady** ("Kubelet stopped posting node status") while the host stayed +up, pingable and SSH-able. The k3s journal showed embedded etcd starved of disk: + +``` +slow fdatasync took=3.56s / 6.71s / 4.95s / 3.40s (etcd expects 1s) +apply request took too long took=8.30s (expects 100ms) +etcdserver: too many requests +Failed to check local etcd status for learner management: context deadline exceeded +``` + +`/proc/pressure/io` **full avg300** at the time: `k3s-prod-1` **11.36**, `k3s-prod-2` (PM9A1 NVMe) +**0.75**, `k3s-prod-3` (WD Blue) **1.29**, against a ~1% steady state. The stack alerted on CPU-PSI +and memory and had **no I/O alert at all**, so the first signal the operator got was a node already +out of the cluster. The thresholds here are those measurements, not guesses: 0.05 is 5x steady +state, 0.10 is the level node 1 actually held while etcd stalled. + +**Why this is self-reinforcing, and why it is urgent.** On every node the etcd WAL (`/data/k3s`) and +all Longhorn replicas (`/data/longhorn`) sit on **one partition** — `node_baseline_etcd_source: +shared-data-partition`, not a dedicated device, and §2's exit condition in +`infra/PROD-PROVISIONING.md` is still **open**. So: node stalls → drops out → its volumes go degraded +→ Longhorn rebuilds them → more writes to the same spindle etcd fsyncs to → worse. Recovery on +2026-09-07 needed a manual `systemctl restart k3s`, and I/O pressure only fell when the 18 rebuilds +finished. + +**There is no etcd fsync metric to check.** `etcd_disk_wal_fsync_duration_seconds` is not collected — +`kubeEtcd` is deliberately disabled in `platform/monitoring/kube-prometheus-stack.yaml` for k3s. PSI +is the only signal that exists. Do not go looking for the etcd dashboard; it is not there. + +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which node, and how bad: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get nodes -o wide' +``` + +Read PSI and per-device queue depth on the node itself (SSH direct — prod is on +`192.168.2.0/24` with no NAT hop; `.10` = prod-1, `.11` = prod-2, `.12` = prod-3): + +``` +ssh k3s-prod-1 'cat /proc/pressure/io; echo; iostat -x 1 3; echo; lsblk -o NAME,MOUNTPOINT,ROTA' +``` + +Is etcd already stalling? This is the line that decides urgency: + +``` +ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|leader"' +``` + +Is Longhorn rebuilding (the usual writer)? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ + -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,NODE:.status.currentNodeID | grep -v healthy' +``` + +Is the disk itself dying? (This is what it was in September.) + +``` +ssh k3s-prod-1 'sudo smartctl -a /dev/sda | grep -Ei "Reallocated|Pending|Uncorrectable|Wear|Power_On_Hours|Model"' +``` + +## Likely causes, most likely first + +### 1. A Longhorn replica rebuild storm (most common by far) +Every node reboot degrades all ~18 volumes and triggers a full rebuild. That is the normal cost of +Longhorn on these disks, and it is why reboots are rolled **one node at a time, waiting for +`robustness=healthy` on every volume before the next**. + +Remediation — freeze rebuilds while the node recovers (operator, mutating): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \ + concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"' +``` + +Thaw to `"1"` — never higher — once all volumes read `healthy`. Use `0` when replicas will +**reattach** (an in-place bounce); use `1` when they are genuinely gone (an OS re-provision). +`concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and +held** — do not "fix" this by setting a knob that is already set. + +### 2. The node's disk is failing +`k3s-prod-1`'s original 2011 `C400-MTFDDAC512MAM` had **61,440 reallocated sectors**, 31% endurance +consumed and no power-loss protection. The operator replaced it with a Samsung 850 EVO on 2026-09-07 +and the same 18-replica rebuild measured **3.79** instead of 11.36 — ~3x better, node stayed Ready. +The earlier dev-era drive (Seagate `ST4000DM000`: 29,408 reallocated, 448 pending, 448 offline +uncorrectable) drove etcd to 48% iowait. If SMART shows reallocated or pending sectors climbing, +**the fix is hardware**; nothing in software recovers it. + +When replacing a disk, two traps recorded from that rebuild: +- Longhorn refuses the new disk with `DiskFilesystemChanged` ("record diskUUID doesn't match the one + on the disk"). Remedy: **disable → remove the disk entry → re-add, in that order** (the validating + webhook enforces it). +- `inventory/hosts.ini` decides `--cluster-init` via `[k3s_prod_primary]`. Re-provisioning a node + that is still listed there **founds a second cluster**. The founder now sits on `k3s-prod-2`. + +### 3. A Velero backup or restore running against the same spindle +Velero's daily runs at `0 3 * * *` and reads Longhorn PVCs through kopia. Check +`backups.velero.io` (fully qualified — see above) for one in flight, and the Longhorn limits +`concurrentVolumeBackupRestorePerNodeLimit` / `snapshotHeavyTaskConcurrentLimit`, both pinned to 1 +for exactly this reason. + +### 4. A CDI image import or a TCG-emulated VM +A CDI import writes a whole disk image; heavy sustained node load from TCG-emulated VMs has also been +observed to degrade the Cilium datapath into intermittent LB-VIP unreachability (cleared by a +restart). Check `kubectl --context k3s-prod get vmi -A` and importer pods in `cdi`. + +### 5. Structural: etcd and Longhorn share one partition +This is the standing cause underneath the other four, and it is open work. `roles/etcd_io_priority` ++ `playbooks/etcd-io-priority.yml` apply BFQ on the shared disk, `nr_requests` 64→32 and +`IOWeight=10000` on `system.slice` and `k3s.service`, reading every value back. **It applies nothing +until an operator runs it** — run `k3s-prod-3` first, with `--limit` and `--list-hosts`. + +Four mechanisms are **proven inert on these nodes** — do not re-try them: +`IOSchedulingClass=realtime` on `k3s.service` is a footgun (ioprio is inherited across fork/exec and +containerd is a child of the k3s server process, so realtime for k3s means realtime for Longhorn +too); `IOSchedulingPriority` is a no-op under `mq-deadline`; `io.latency` is unavailable +(`CONFIG_BLK_CGROUP_IOLATENCY` is not built in Debian trixie); a bare `IOWeight=` drop-in is +decoration (blk-iocost compiled-but-disabled, BFQ not loaded). And no cgroup or scheduler control +arbitrates the **jbd2 journal** that etcd's `fdatasync` actually waits on. + +## When to escalate or stop + +- **Stop and get the operator** if `NodeIOPressureCritical` is firing *and* the journal shows + `slow fdatasync` or `apply request took too long`. The node is minutes from leaving the cluster. + With one of three etcd members already gone, a second loss ends quorum. +- **Do not reboot or drain a second node** while any volume is not `healthy`. That is how a + single-node problem becomes a data problem on a 3-replica cluster. +- **Do not `kubectl drain`.** This cluster does not drain: the posture is deliberate (the upgrade + plan rejects system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD + Longhorn"), and the node-loss conformance test evicts only its own writer pod. +- Hand over to the operator for anything physical (disk replacement) or for the Ansible role — both + are outside what can be done from git. + +--- +Source of truth: `runbooks/node-io.md` in `home-network-k3s-impl`. Index of all runbooks: `/runbooks/` +Incident record: `DECISIONS-LOG.md` "2026-09-07 — k3s-prod-1 NotReady: etcd starved of disk I/O by +Longhorn", and `infra/PROD-PROVISIONING.md` §0c / §2. diff --git a/runbooks/node-memory.md b/runbooks/node-memory.md new file mode 100644 index 0000000..c7bf90a --- /dev/null +++ b/runbooks/node-memory.md @@ -0,0 +1,119 @@ +# node-memory — Memory pressure high on a node + +**Alert:** `NodeMemoryPressureHigh` (warning, 10m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes` +**Expression:** `(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90` + +## What this means here + +Available memory has been under 10% for ten minutes. These are HP EliteDesk 800 G5 Minis with +~22 GiB usable, running the whole platform plus KubeVirt VMs on three nodes, so the headroom is +genuinely thin and this alert is a capacity signal rather than a bug signal most of the time. + +Two cluster-specific facts change how you read it: + +**`SwapTotal == 0` is not evidence swap is off.** On 2026-09-07 all three prod nodes came back from a +reboot with **977 MiB of swap ACTIVE** (`/dev/sda4` on prod-1 and prod-3, `/dev/nvme0n1p4` on prod-2) +while the Ansible role's commented-out fstab line reported success. +`systemd-gpt-auto-generator` claims any GPT partition typed +`0657fd6d-a4ab-43c4-84e5-0933c84b4f4f` straight from the partition table and synthesises a `.swap` +unit into `/run/systemd/generator.late/` — it never reads `/etc/fstab`. The fix masks `swap.target` +(the single choke point for gpt-auto, fstab-generator and cryptsetup-generator). If a node is +swapping, the kubelet's memory accounting is lying to you and eviction will misbehave. Check +explicitly with `swapon --show`, never `SwapTotal`. + +**There is a kernel memory-reporting bug on all three nodes.** `infra/PROD-PROVISIONING.md` §0b +records an EFI memattr defect that once presented `k3s-prod-1` as having **2.6 GiB usable of 16 GB**. +It needs kernel ≥ 6.13 and all three nodes are exposed (prod-3 publishes `MEMATTR=0xbeece018` too). +If `MemTotal` looks absurd rather than merely tight, suspect this before suspecting a leak. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod top nodes; echo; kubectl --context k3s-prod get nodes -o custom-columns=NAME:.metadata.name,MEM:.status.capacity.memory,PRESSURE:.status.conditions[?\(@.type==\"MemoryPressure\"\)].status' +``` + +Who is using it — and is swap actually off? + +``` +ssh k3s-prod-1 'free -h; echo; swapon --show; echo "(empty output above = swap really off)"; echo; cat /proc/pressure/memory; echo; ps -eo pid,rss,comm --sort=-rss | head -15' +``` + +Biggest consumers in the cluster, and anything that has been OOM-killed: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod top pods -A --sort-by=memory | head -20; echo; \ + kubectl --context k3s-prod get events -A --field-selector reason=OOMKilling --sort-by=.lastTimestamp | tail -20' +``` + +## Likely causes, most likely first + +### 1. A monitoring component grew past its limit +This is the one with real history. When kube-prometheus-stack went 65 → 88 it shipped Grafana 13 and +**Grafana OOM-looped** — "the hidden blocker that stalled the whole kps upgrade and the platform +tier". Limits went 256 → 512 → **768Mi** for Grafana and Prometheus 2 → **3Gi** (commit `b2b1aa5`). +Prod keeps 30d retention / 45GiB retentionSize on a 50Gi PVC, so Prometheus is legitimately the +largest single consumer. Remediation: raise the limit in +`platform/monitoring/kube-prometheus-stack.yaml` and push; do not restart it in a loop. + +### 2. A KubeVirt VM, especially a TCG-emulated one +VMs are the largest discretionary consumer. `useEmulation: false` is set deliberately (on v1.9 +`true` silently forces TCG), but leaked test namespaces running emulated VMs have caused API blips +before and were deleted. Check `kubectl --context k3s-prod get vmi -A` and the `virt-launcher-*` pods' +memory. + +### 3. Swap silently back on after a reboot +See above. Verify with `swapon --show` and `systemctl is-enabled swap.target` (should be `masked`). +`node_health` now reports `memory.swap_persistence` for this. The fix is the Ansible role; an +operator runs it. + +### 4. Genuine capacity exhaustion +If nothing is anomalous, the node is simply full. The node vCPU count was already raised 6 → 8 for +CPU pressure; memory is next. A bpg/OpenTofu memory change **reboots the VM**, so roll **one node at +a time and wait for Longhorn `robustness=healthy` between each** — the same discipline as every other +reboot here. + +### 5. A leak in an app-tier workload +Nextcloud, Plex and the *arr stack are the usual suspects. Compare `top pods` against the previous +day in Grafana rather than guessing. + +## When to escalate or stop + +- **Escalate immediately if this is firing together with `node-io` or `node-down`** on the same node. + Memory pressure plus I/O pressure on an etcd member is the shape of the 2026-09-07 outage. +- **Do not add swap** to relieve this. Kubernetes does not want it here and the platform has already + been bitten by swap re-enabling itself. +- **Do not evict or drain** to free memory: this cluster does not drain (Longhorn rebuild cost). Kill + the specific offending pod instead, and only if you know what it is. +- Hand to the operator for anything requiring a VM resize or an Ansible run. + +--- +Source of truth: `runbooks/node-memory.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` (Grafana OOM, kps 65→88); commits `0e92d25`/`9d00a3f` (swap persistence); +`infra/PROD-PROVISIONING.md` §0b (EFI memattr). diff --git a/runbooks/pvc-capacity.md b/runbooks/pvc-capacity.md new file mode 100644 index 0000000..3137fe9 --- /dev/null +++ b/runbooks/pvc-capacity.md @@ -0,0 +1,140 @@ +# pvc-capacity — A PersistentVolumeClaim is filling up, nearly full, or predicted to fill + +**Alerts:** `PersistentVolumeFillingUp` (warning, 15m), `PersistentVolumeAlmostFull` (critical, 15m), +`PersistentVolumeFillingUpPredicted` (warning, 1h) +**File:** `platform/monitoring/pvc-capacity-alerts.yaml`, group `platform-health.pvc-capacity` +**Expressions:** `< 0.15` free / `< 0.05` free / `< 0.50` free **and** +`predict_linear(kubelet_volume_stats_available_bytes[24h], 7*24*3600) < 0`, each `and +kubelet_volume_stats_inodes{job="kubelet"} > 0` + +## What this means here + +These are the PVC fill alerts for everything **outside** the `monitoring` namespace +(`/runbooks/monitoring-capacity` owns that half). Before this file existed, **no PVC outside +`monitoring` had any fill alert at all**. Unlike the Longhorn rule group, these work: they are built +on `kubelet_volume_stats_*`, which is live (56 series on prod). + +Three design facts you need in order to read the labels correctly: + +- **Every rule pins `job="kubelet"`.** `kubelet_volume_stats_*` is published by *two* jobs with + different label shapes: `job="kubelet"` carries the PVC's real namespace, while `job="apiserver"` + reports `namespace="default"` with the real one in `exported_namespace`. Without the pin, + `namespace!="monitoring"` double-fires on monitoring's own PVCs and names the wrong namespace. +- **Every rule carries `and kubelet_volume_stats_inodes > 0`.** Block-mode PVCs — KubeVirt VM disks + on `longhorn-vm` — report available=0, used=0, inodes=0 against a real capacity. Without the guard, + `unifi/unifi-rootdisk` (40 GiB Block) would fire both warning **and** critical permanently. If you + are ever tempted to remove that clause, this is why it is there. +- **The predicted rule is the point of the exercise.** A level threshold is useless for a slow leak on + a mostly-empty volume: prod's `keycloak-db` sat at **93% free** while leaking ~6.8 MiB/h of + unrecyclable WAL, so `<15%` was ~29 days away and would have arrived as a surprise. The `< 0.50` + gate keeps it quiet on volumes that are not plausibly about to fill, and the 24h fit / 7d horizon is + chosen so a daily backup or log rotation does not make it flap. + +**If the PVC name matches `*-db-N`, read `/runbooks/cnpg-archiving` before doing anything else.** +That is by far the most likely cause on this cluster and the one where the obvious action is fatal. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which PVC, how full, on which node: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s -G --data-urlencode "query=topk(10, 1 - kubelet_volume_stats_available_bytes{job=\"kubelet\"} / kubelet_volume_stats_capacity_bytes{job=\"kubelet\"})" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "persistentvolumeclaim|namespace|\"1\"" ' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get pvc -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SIZE:.spec.resources.requests.storage,SC:.spec.storageClassName,STATUS:.status.phase' +``` + +Is it a CNPG data volume? (Fully-qualified `clusters.postgresql.cnpg.io` — the short name `cluster` is +ambiguous on clusters carrying more than one `Cluster` kind.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A' +``` + +If yes, the one question that decides everything — is WAL archiving working? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint' +``` + +What is actually consuming the space, from inside the pod: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c "df -h /var/lib/postgresql/data; du -sh /var/lib/postgresql/data/pgdata/pg_wal; ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | wc -l"' +``` + +## Likely causes, most likely first + +### 1. A CNPG database is not recycling WAL +The dominant cause. With `archive_mode=on` and a failing `archive_command`, Postgres **refuses to +recycle** unarchived WAL segments, so `pg_wal` grows until the data PVC is full and the postmaster +**PANICs** — taking the database, and for `keycloak-db` all of SSO, down. `pg_wal` shares the data +PVC on every Cluster here (no `walStorage:` is configured), which is why a *backup* fault presents as +a *capacity* alert. This actually happened: prod ran from bootstrap to 2026-09-07 with +`endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, all three Clusters at `ContinuousArchiving=False` with +`barman-cloud-wal-archive: exit status 4`, unnoticed for 38+ hours. +**Remediation: fix archiving (`/runbooks/cnpg-archiving`). Do NOT delete WAL by hand — that destroys +the database. Do NOT expand the PVC and call it done; that buys days.** + +### 2. Log or cache growth in an app-tier workload +Nextcloud, the *arr apps, Gitea. Prune inside the app, or expand the PVC in git (every Longhorn class +has `allowVolumeExpansion: true`). + +### 3. Media on the wrong storage class +Media is meant to be static NFS PVs from `192.168.2.102:/mnt/Storage/plex` (8.32 TiB measured), not +Longhorn. A media volume on `longhorn` is a misconfiguration; fix the class, do not grow the disk. + +### 4. A Velero or CDI staging write +Transient. CDI uses `longhorn-scratch` deliberately (single replica) because the 3-replica default +failed the UniFi 60Gi import on insufficient storage. + +### 5. The predicted rule caught a genuine slow leak +This is the rule working as designed and giving you a week. Find what is growing before the level +thresholds arrive. + +## When to escalate or stop + +- **`PersistentVolumeAlmostFull` on a `*-db-N` volume is an imminent database outage.** Get the + operator. A Postgres data volume at under 5% free will PANIC the postmaster. +- **Never delete WAL segments, and never delete anything under a Postgres data directory.** +- **Do not delete the PVC** to reclaim space. +- If `unifi-rootdisk` or another Block-mode volume appears in these alerts, the inodes guard has been + removed or broken — that is a monitoring bug, not a storage incident. +- Escalate for PVC expansion beyond what the node can hold (see `/runbooks/longhorn-capacity`). + +--- +Source of truth: `runbooks/pvc-capacity.md`. Index: `/runbooks/` +Records: `platform/monitoring/pvc-capacity-alerts.yaml` (the two-job and inodes findings); +`platform/cnpg/alerts.yaml` header (the 2026-09-06 archiving incident and the measured fill rate). diff --git a/runbooks/serving/deployment.yaml b/runbooks/serving/deployment.yaml new file mode 100644 index 0000000..bf8c0d5 --- /dev/null +++ b/runbooks/serving/deployment.yaml @@ -0,0 +1,151 @@ +--- +# The runbook static server. Deliberately the dumbest thing that can work. +# +# WHY A POD AT ALL, AND NOT A REDIRECT TO GITEA. The obvious cheaper option is an HTTPRoute whose +# only job is a RequestRedirect into the in-cluster Gitea web UI, which already renders these files. +# It was rejected for three reasons, in order of weight: +# 1. DEPENDENCY DIRECTION. A runbook is read when the platform is broken. Gitea is an app-tier +# workload on Longhorn; its own DR path and several of these runbooks' subjects sit UNDER it. +# `runbooks/longhorn-faulted` must be readable while Longhorn is faulted, and +# `runbooks/keycloak-db` while a CNPG database is down. Pointing the operator at Gitea makes +# the documentation fail in exactly the conditions that produce the alert. +# 2. NO SUFFIX REWRITE EXISTS. Gateway API's RequestRedirect path modifier offers ReplaceFullPath +# and ReplacePrefixMatch only. There is no "append .md", so mapping 34 extensionless names onto +# 34 `.md` blobs needs 34 hand-maintained ReplaceFullPath rules — MORE surface than this pod, +# and a silent 404 for every runbook someone forgets to add a rule for. +# 3. A 302 is not a document. The annotation's contract is that the URL returns the runbook; a +# redirect chain through a second app's auth and rendering is a worse contract. +# This pod reads only from a ConfigMap projected out of git. It depends on the CNI, the kubelet and +# the Gateway, and on nothing else in the platform. +# +# WHY NOT THE SERVICE CATALOG (catalog/*.yaml + render.py). The catalog is the idiomatic onboarding +# surface for a SERVICE, and it was the first thing checked — but its schema cannot express this +# route. `ingressRoutes[].host` is pattern-bound to `^[a-z0-9-]+\.(\$\{BASE_DOMAIN\}|home\.keithjackson\.dev)$`: +# a single DNS label plus the apex, so the BARE apex is unrepresentable, and there is no `path` field +# at all, so a path-prefix route cannot be authored. Bending the schema for one docs server would be +# the tail wagging the dog. The repo's OTHER established pattern is exactly right instead: a small +# hand-authored platform component reconciled by its own non-gating Flux Kustomization — what +# platform/homepage does for the apex `/` prefix, which this mirrors file for file. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: runbooks + namespace: runbooks + labels: + app.kubernetes.io/name: runbooks +spec: + # TWO replicas with anti-affinity, unlike homepage's one. This is the only workload in the + # cluster whose whole purpose is to be available during an incident, and a single replica is + # unavailable for the minutes a node reboot or a drain takes — which is when it is wanted. It + # costs 2 x 16Mi. + replicas: 2 + revisionHistoryLimit: 3 + strategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 0 + maxSurge: 1 + selector: + matchLabels: + app.kubernetes.io/name: runbooks + template: + metadata: + labels: + app.kubernetes.io/name: runbooks + spec: + automountServiceAccountToken: false + # Spread across nodes so losing one node never takes the runbooks with it. `preferred`, not + # `required`: on a 3-node cluster with one node cordoned for the very incident being debugged, + # a hard rule would leave a replica Pending instead of running somewhere. + affinity: + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + topologyKey: kubernetes.io/hostname + labelSelector: + matchLabels: + app.kubernetes.io/name: runbooks + # Tolerate a node that is under memory/disk pressure or unreachable for a while. An operator + # reading `runbooks/node-disk` is by definition dealing with a node under pressure, and the + # default 300s unreachable toleration is long enough; these exist so eviction pressure does + # not silently remove the last replica while the cluster is sick. + tolerations: + - key: node.kubernetes.io/unreachable + operator: Exists + effect: NoExecute + tolerationSeconds: 900 + securityContext: + runAsNonRoot: true + runAsUser: 101 # `nginx` in nginxinc/nginx-unprivileged + runAsGroup: 101 + fsGroup: 101 + seccompProfile: + type: RuntimeDefault + containers: + - name: nginx + # Renovate maintains this pin. nginx-unprivileged (not nginx) so the container can honour + # runAsNonRoot: the stock nginx image's entrypoint needs root to drop privileges and to + # bind :80. `stable-alpine` branch, pinned exactly. + image: nginxinc/nginx-unprivileged:1.30.4-alpine + imagePullPolicy: IfNotPresent + ports: + - name: http + containerPort: 8080 + protocol: TCP + resources: + requests: + cpu: 10m + memory: 16Mi + limits: + memory: 64Mi + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + # Achievable here, unlike homepage/headlamp: nginx only needs /tmp (its pid file, on the + # unprivileged image) and /var/cache/nginx writable, and both are emptyDirs below. + readOnlyRootFilesystem: true + livenessProbe: + httpGet: { path: /healthz, port: http } + initialDelaySeconds: 3 + periodSeconds: 15 + readinessProbe: + # Probe a REAL runbook, not just /healthz. nginx answers /healthz from its own config + # even if the ConfigMap volume failed to project, so /healthz alone would report a pod + # that serves 404 for every runbook as Ready. `node-down` is the canonical alert link + # (NodeExporterDown, platform/monitoring/alerts.yaml), so if this path 404s the pod is + # useless and must not take traffic. + httpGet: { path: /runbooks/node-down, port: http } + initialDelaySeconds: 3 + periodSeconds: 15 + volumeMounts: + - name: nginx-config + mountPath: /etc/nginx/conf.d/default.conf + subPath: default.conf + readOnly: true + # The runbooks themselves, one extensionless file per key. Whole-volume mount (NOT + # subPath): subPath mounts do not update when the ConfigMap changes, and here the + # ConfigMap name carries kustomize's content hash so a content change rolls the pod + # anyway — the whole-volume mount just means a stale projection is impossible. + - name: runbooks + mountPath: /usr/share/nginx/html/runbooks + readOnly: true + - name: tmp + mountPath: /tmp + - name: cache + mountPath: /var/cache/nginx + volumes: + - name: nginx-config + configMap: + name: runbooks-nginx + - name: runbooks + configMap: + # Generated by ../kustomization.yaml's configMapGenerator from runbooks/*.md. kustomize + # rewrites this reference to the hashed name, so the pods roll whenever a runbook changes. + name: runbooks-content + - name: tmp + emptyDir: {} + - name: cache + emptyDir: {} diff --git a/runbooks/serving/httproute.yaml b/runbooks/serving/httproute.yaml new file mode 100644 index 0000000..a016563 --- /dev/null +++ b/runbooks/serving/httproute.yaml @@ -0,0 +1,77 @@ +--- +# North-south ingress for the runbooks at https:///runbooks/ — the exact prefix the alert +# annotations already point at (`runbook_url: "https://${BASE_DOMAIN}/runbooks/"`, 42 of them +# across platform/monitoring/alerts.yaml, platform/monitoring/pvc-capacity-alerts.yaml, +# platform/cnpg/alerts.yaml and platform/kubevirt/alerts.yaml). Nothing about those annotations +# changes; this route is what finally makes them resolve. +# +# NO `hostnames:` — DELIBERATE, and the reason this manifest needs no Flux variable substitution. +# An HTTPRoute with no hostnames inherits the hostname of every listener it attaches to, and this +# route attaches to exactly one: `https-apex`, whose hostname is `${BASE_DOMAIN}` +# (platform/gateway/gateway.yaml). So the route is correct on dev and on prod with no +# `postBuild.substituteFrom` anywhere in this tier — which matters a great deal here, because the +# SAME Flux Kustomization renders the runbook markdown into a ConfigMap. With substitution enabled, +# Flux's envsubst would run over the runbook prose and silently EMPTY every `${...}` in every shell +# example a runbook contains (it fails open — DECISIONS-LOG 2026-09-01, the Cilium k8sServiceHost +# incident). Declaring no hostname removes the last reason to substitute, so the docs ship verbatim. +# See clusters/base/runbooks.yaml. +# +# PATH PRECEDENCE AGAINST HOMEPAGE. platform/homepage/httproute.yaml already binds `https-apex` with +# PathPrefix `/`. Gateway API resolves a conflict between rules on the same hostname by preferring +# the LONGEST path match, so `/runbooks` wins over `/` for these requests and homepage keeps +# everything else. The two routes coexist by specification, not by luck — and the live 200s recorded +# in the commit prove it on this Gateway implementation (Cilium). +# +# ============================ AUTH: THIS ROUTE IS PUBLIC ============================== +# auth-mode=none, deliberately, and it is the one route in this repo where that is the SAFE choice. +# +# THE CIRCULARITY. Every other exposed UI here declares forward-auth (oauth2-proxy -> Keycloak). +# Put these pages behind that and `runbooks/keycloak-down` becomes unreadable precisely when +# Keycloak is down; `runbooks/keycloak-db` unreadable when its CNPG database is down; +# `runbooks/keycloak-client-secrets` unreadable during the exact failure mode that broke all 12 +# OIDC clients on 2026-09-06 while every health signal stayed green. An on-call who cannot log in +# is the primary reader of this route. Gating documentation on the identity provider it documents +# is a loop with no exit. +# +# WHAT IS ACTUALLY EXPOSED. Operational procedure: metric names, thresholds, kubectl invocations, +# component topology. No credentials, no secrets, no tokens, no internal IPs beyond the LAN ranges +# already written in the public spec repo. The content is the same text any reader of the git repo +# can see. The real cost is reconnaissance value — it names what runs here and how it fails — and +# that is accepted knowingly, against the certainty of a locked-out operator staring at a login page +# during an outage. If that trade is ever revisited, the correct fix is NOT forward-auth on this +# route; it is to keep it public and move sensitive specifics out of the runbooks. +# +# HONEST NOTE ON TODAY'S POSTURE. The Gateway ext-authz attachment that would ENFORCE forward-auth +# does not exist anywhere in this repo yet (platform/oauth2-proxy is the authenticator only; the one +# CiliumEnvoyConfig on the cluster is the Gateway's own). So every `forward-auth` route — homepage, +# pihole, jackett, qbittorrent, radarr, sonarr — is in practice reachable unauthenticated today. +# This route is therefore no more open than its neighbours right now; the difference is that when +# ext-authz does land, this one must be left OUT of it ON PURPOSE, and this comment is the record of +# that decision so nobody "fixes" it later and re-closes the loop. +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: runbooks + namespace: runbooks + labels: + app.kubernetes.io/name: runbooks + annotations: + homelab.keithjackson.dev/auth-mode: none + homelab.keithjackson.dev/auth-rationale: >- + Public on purpose. An operator locked out of Keycloak is this route's primary reader; + gating it behind Keycloak would make runbooks/keycloak-down unreadable exactly when it is + needed. Content is operational procedure only — no secrets. Do NOT add this route to the + Gateway ext-authz attachment when forward-auth enforcement lands. +spec: + parentRefs: + - name: shared + namespace: gateway + sectionName: https-apex # the apex listener; the wildcard `https` listener won't match it + rules: + - matches: + - path: + type: PathPrefix + value: /runbooks + backendRefs: + - name: runbooks + port: 80 diff --git a/runbooks/serving/namespace.yaml b/runbooks/serving/namespace.yaml new file mode 100644 index 0000000..85ec480 --- /dev/null +++ b/runbooks/serving/namespace.yaml @@ -0,0 +1,16 @@ +--- +apiVersion: v1 +kind: Namespace +metadata: + name: runbooks + labels: + # Serve the runbooks at https:///runbooks/ — the APEX host, on a `/runbooks` path + # prefix. The apex has its own `https-apex` listener on the shared Gateway + # (platform/gateway/gateway.yaml); this label opts the namespace in to attach HTTPRoutes to it + # (serving/httproute.yaml). Same opt-in as homepage, which owns the apex `/` prefix. + gateway-access: "shared" + # NOT homelab.keithjackson.dev/app-namespace=true. This is a PLATFORM component (like homepage and + # headlamp), so Kyverno does not generate the app default-deny baseline here and the app-scoped + # auth-mode / egress-class policies do not apply. The default-deny NetworkPolicy and the + # Gateway-only CiliumNetworkPolicy are shipped explicitly in serving/networkpolicy.yaml instead. + app.kubernetes.io/name: runbooks diff --git a/runbooks/serving/networkpolicy.yaml b/runbooks/serving/networkpolicy.yaml new file mode 100644 index 0000000..9a249ad --- /dev/null +++ b/runbooks/serving/networkpolicy.yaml @@ -0,0 +1,57 @@ +--- +# Default-deny baseline (R-NET-01): deny all ingress + egress, allow only DNS. `runbooks` is a +# PLATFORM namespace (no app-namespace label) so Kyverno does not auto-generate this — it is shipped +# explicitly, defence in depth, same shape as the Kyverno-generated app baseline and as +# platform/homepage/networkpolicy.yaml. Cilium enforces the UNION of this NetworkPolicy and the +# CiliumNetworkPolicy below (allow if ANY policy allows). +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-deny + namespace: runbooks + labels: + homelab.keithjackson.dev/baseline: default-deny +spec: + podSelector: {} + policyTypes: + - Ingress + - Egress + ingress: [] + egress: + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 +--- +# Gateway ingress allow (R-NET-01/02): only the shared Gateway (the Cilium `ingress` entity) may +# reach the runbook server, on container port 8080. The server makes NO outbound calls whatsoever — +# every byte it serves comes from a ConfigMap projected by the kubelet — so the baseline's DNS-only +# egress is already more than it needs and no egress CNP is added. +# +# CONSEQUENCE WORTH STATING: because egress is DNS-only, these pages cannot pull a stylesheet, a +# font or a markdown renderer from anywhere. That is why the content is served as text/plain +# (serving/nginx-config.yaml) rather than as HTML with a CDN dependency, and it is a feature: the +# runbooks render with zero network beyond the request that fetched them. +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: runbooks-allow-gateway-ingress + namespace: runbooks + labels: + app.kubernetes.io/name: runbooks +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: runbooks + ingress: + - fromEntities: + - ingress + toPorts: + - ports: + - port: "8080" + protocol: TCP diff --git a/runbooks/serving/nginx-config.yaml b/runbooks/serving/nginx-config.yaml new file mode 100644 index 0000000..6ca8b9f --- /dev/null +++ b/runbooks/serving/nginx-config.yaml @@ -0,0 +1,81 @@ +--- +# nginx server config for the runbook static server. Hand-authored ConfigMap (same plain-manifest style +# as platform/homepage), mounted over /etc/nginx/conf.d/default.conf. +# +# WHY THE URI NEEDS NO REWRITE. The alert annotations are `.../runbooks/` with NO `.md` suffix, and +# this repo must not change them (they are spread across four alerts.yaml files that other work is +# editing). Rather than rewriting the URI, the ConfigMap generated from runbooks/*.md maps each file to an +# EXTENSIONLESS key (`node-down=node-down.md` in ../kustomization.yaml) and is mounted at /runbooks/. +# So `/runbooks/node-down` is a plain static file lookup: no rewrite, no fallback chain, nothing to get +# subtly wrong at 3am. +# +# CONTENT TYPE. An extensionless file has no MIME mapping, so nginx would serve it as +# application/octet-stream and a browser would DOWNLOAD it instead of showing it. The empty `types {}` +# block removes the mapping table entirely inside these locations and `default_type` forces +# text/plain;charset=utf-8 — markdown read as plain text, which is what markdown is for, and which works +# identically in a browser, in `curl`, and in a terminal with no JavaScript, no CDN and no egress. This +# namespace has DNS-only egress, so a client-side markdown renderer fetched from a CDN could not load +# anyway; plain text is the honest choice here, not a shortcut. +apiVersion: v1 +kind: ConfigMap +metadata: + name: runbooks-nginx + namespace: runbooks + labels: + app.kubernetes.io/name: runbooks +data: + default.conf: | + # Replaces the image's stock conf.d/default.conf. Port 8080 because the base image is + # nginx-unprivileged (runs as uid 101 and cannot bind 80). + server { + listen 8080; + server_name _; + root /usr/share/nginx/html; + + # Never leak the server version on an error page. + server_tokens off; + # Keep the 301 below relative so it works through the Gateway without knowing the public host. + absolute_redirect off; + + # Probes. Kept OUT of the access log so a 15s kubelet probe does not bury the one request an + # operator actually made. + location = /healthz { + access_log off; + default_type text/plain; + return 200 "ok\n"; + } + + # /runbooks and /runbooks/ both land on the index (runbooks/README.md, mounted as the + # extensionless key `README`). + location = /runbooks { + return 301 /runbooks/; + } + location = /runbooks/ { + types { } + default_type "text/plain; charset=utf-8"; + try_files /runbooks/README =404; + } + + # The runbooks themselves. One static file per alert runbook_url. + location /runbooks/ { + types { } + default_type "text/plain; charset=utf-8"; + # A ConfigMap volume is a tree of symlinks into ..data/; nginx must follow them. + disable_symlinks off; + try_files $uri =404; + add_header Cache-Control "public, max-age=300"; + # An unknown name gets a 404 with a POINTER, not a blank page: the likeliest cause is an + # alert whose runbook was never written, and the reader needs to know where to look next. + error_page 404 = @no_such_runbook; + } + location @no_such_runbook { + default_type "text/plain; charset=utf-8"; + return 404 "No runbook by that name.\n\nIndex of all runbooks: /runbooks/\nSource of truth: runbooks/.md in the home-network-k3s-impl repo.\nIf an alert linked you here, its runbook_url has no matching runbooks/.md, and the\nconformance check alert-runbooks-resolvable should have caught it before merge.\n"; + } + + # Nothing else is served from this pod. The apex `/` prefix belongs to homepage. + location / { + default_type "text/plain; charset=utf-8"; + return 404 "This host serves /runbooks/ only.\n"; + } + } diff --git a/runbooks/serving/service.yaml b/runbooks/serving/service.yaml new file mode 100644 index 0000000..771a9be --- /dev/null +++ b/runbooks/serving/service.yaml @@ -0,0 +1,19 @@ +--- +# ClusterIP fronting the runbook server. The shared Gateway routes the apex `/runbooks` prefix here +# (serving/httproute.yaml); Service port 80 -> container port 8080. +apiVersion: v1 +kind: Service +metadata: + name: runbooks + namespace: runbooks + labels: + app.kubernetes.io/name: runbooks +spec: + type: ClusterIP + selector: + app.kubernetes.io/name: runbooks + ports: + - name: http + port: 80 + targetPort: http + protocol: TCP diff --git a/runbooks/targets-down.md b/runbooks/targets-down.md new file mode 100644 index 0000000..4f05924 --- /dev/null +++ b/runbooks/targets-down.md @@ -0,0 +1,134 @@ +# targets-down — More than 10% of Prometheus scrape targets are down + +**Alert:** `PrometheusTargetsMissing` (warning, 15m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.monitoring-self` +**Expression:** `(count(up == 0) / count(up)) > 0.10` + +## What this means here + +A tenth of everything Prometheus is supposed to scrape is unreachable. This is a **blind-spot** alert: +its purpose is to tell you that other alerts may be silent because their data is missing, not because +the thing they watch is healthy. + +Live baseline on prod, measured 2026-09-13: **55 targets across 19 jobs** — +`apiserver`, `cdi-prometheus-metrics`, `coredns`, `keycloak-discovery`, `keycloak-service`, +`keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`, `kube-prometheus-stack-grafana`, +`kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`, `kube-state-metrics`, `kubelet`, +`kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`, `node-exporter`, `oauth2-proxy`, +`semaphore-db/semaphore-db`, `velero`, `velero/node-agent`. So 10% is roughly **6 targets** — one +node's worth of exporters, or one CNPG cluster's three pods. + +**The far more dangerous failure this alert CANNOT see: a target that was never configured.** `up` +only counts targets Prometheus knows about. A missing ServiceMonitor produces no `up == 0` series and +no alert at all. This cluster has been bitten by that repeatedly, and these are the known live gaps as +of 2026-09-13 — every one of them means the corresponding alerts are **inert**: + +| Missing scrape job | Alerts that cannot fire | +|---|---| +| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornVolumeCapacityHigh`, `LonghornNodeStorageHigh` | +| cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` | +| Flux controllers (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` | +| Hubble / Cilium (`hubble_drop_total`) | `HubbleDeniedFlowSpike` | +| etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all | + +Precedent for how costly this is: Velero shipped with three backup alerts and +`metrics.serviceMonitor.enabled: false`, so *"a nightly backup could fail — or silently stop running — +for weeks with nothing to say so."* That ServiceMonitor now exists. The rest of the table does not. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which targets are down, grouped by job: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s -G --data-urlencode "query=up == 0" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "\"job\"|\"instance\"|\"namespace\"" ' +``` + +The scrape-error text, which usually names the cause outright: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s "localhost:19090/api/v1/targets?state=active" | python3 -c " +import json,sys +for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]: + if t[\"health\"]!=\"up\": print(t[\"labels\"].get(\"job\"), t[\"scrapeUrl\"], t[\"lastError\"][:120])"' +``` + +Which monitors exist at all (the gap this alert cannot see): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get servicemonitors,podmonitors -A' +``` + +## Likely causes, most likely first + +### 1. A node is down or NotReady +One node takes out its `node-exporter` and `kubelet` targets together — roughly the 10% threshold on +its own. Go to `/runbooks/node-down`; this alert is the echo. + +### 2. A workload was rolled, scaled to zero, or crashlooped +Targets follow pods. A CNPG cluster restarting takes three targets with it. Transient; if it persists, +the workload is the problem, not the monitoring. + +### 3. A NetworkPolicy blocks the scrape +Default-deny is the baseline everywhere (R-NET-01) and a new component's monitor will be scraped the +moment it exists — `serviceMonitorSelectorNilUsesHelmValues: false` means Prometheus selects every +monitor in the cluster — so a newly added workload can produce a permanently-down target purely +because nothing allows ingress from `monitoring`. The `lastError` will be a timeout. Fix is an +explicit allow in the component's CiliumNetworkPolicy, in git. + +### 4. TLS or auth on the scrape endpoint +`lastError` will say so (certificate or 401/403). Note a related trap from this cluster: after the +Cilium upgrade, an in-cluster call to a `*.home` hostname is re-checked against the destination's real +backend identity, so `toEntities: world` no longer covers it — see `/runbooks/denied-flows`. + +### 5. The exporter is genuinely broken +CDI once had a broken HTTP ServiceMonitor that caused a false `CDIOperatorDown`, and +`KubeVirtControlPlaneDown` once used a non-existent metric with the wrong job label. Both were fixed. +An exporter that is up but publishing the wrong series is the hardest case — see +`/runbooks/cnpg-exporter` for the canonical example (the CNPG exporter can be `up=1` while its whole +`cnpg_pg_stat_*` family is missing). + +## When to escalate or stop + +- **Treat this alert as "trust nothing else for now".** Before acting on the *silence* of any other + alert, confirm its series exists (`count()`), because the table above shows five whole rule + groups are currently evaluating against nothing. +- Escalate if the down targets are `apiserver` or `kubelet` — that is a control-plane problem, not a + monitoring one. +- Adding a missing ServiceMonitor is a git change (a new `ServiceMonitor` next to the component, or a + chart value). Do not "fix" a blind spot by silencing the alert. +- The Longhorn, cert-manager and Flux monitor gaps are known open work. Note them; do not try to + hand-apply monitors into the cluster — every durable change goes through git. + +--- +Source of truth: `runbooks/targets-down.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` / commit `2d5171e` (Velero had no ServiceMonitor, so three critical backup +alerts could never fire); commits `6a7e9d1`, `b57b3e9` (the false CDI/KubeVirt alerts). diff --git a/runbooks/update-rollback.md b/runbooks/update-rollback.md new file mode 100644 index 0000000..78209fc --- /dev/null +++ b/runbooks/update-rollback.md @@ -0,0 +1,146 @@ +# update-rollback — A HelmRelease failed its health gate and is rolling back + +**Alert:** `UpdateHealthGateFailed` (critical, 10m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.automated-updates` +**Expression:** `max by (namespace, name) (gotk_reconcile_condition{kind="HelmRelease", type="Ready", status="False"}) == 1` + +## What this means here + +helm-controller holds a HelmRelease at Ready=False while a failed upgrade remediates and rolls back to +the last-good revision. This is the **R-UPD-03 safety rail firing** — the bad-update auto-rollback +working — so the alert is a report that the platform protected itself, not necessarily that something +is broken right now. + +Read the expression honestly, though: it matches **any** HelmRelease at Ready=False, not only one that +is mid-rollback. The rule's own comment flags this as a `TODO(bring-up)`: tighten it with the `reason` +label (`UpgradeFailed` / `RolledBack`) once confirmed against the deployed helm-controller. So a +HelmRelease that simply cannot install — a bad value, a missing CRD, a crashlooping pod — lights this +`critical` too. Your first job is to tell those two cases apart, because they need opposite responses: +a rollback is self-healing and wants a git revert; a failed *install* is a stall and may be gating an +entire tier (see `/runbooks/flux-reconcile`). + +Real precedent that rollback does work here: during the 2026-09-01 Cilium incident the Helm release +history shows `v8 e83bf695 → v9 eebd6c88 (the bad render) → v10 e83bf695 (identical to v8)`, and the +release secret for v9 reads `failed`. Helm's wait genuinely failed and the release was remediated — a +claim that remediation never engaged was **measured wrong**. That is why the cluster self-healed in +about six minutes. +**This alert is currently INERT.** `count(gotk_reconcile_condition)` returned **0 series** on prod on +2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every `gotk_*` rule +evaluates against nothing. Read the HelmRelease CRs directly with the commands below, and see +`/runbooks/targets-down` for the full list of inert rule groups. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which release, and what happened: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get helmreleases.helm.toolkit.fluxcd.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,REASON:.status.conditions[?\(@.type==\"Ready\"\)].reason,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,LASTSUCCESS:.status.lastAppliedRevision' +``` + +The Helm revision history — this is what distinguishes "rolled back" from "never installed": + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n get secrets -l owner=helm --sort-by=.metadata.creationTimestamp \ + -o custom-columns=NAME:.metadata.name,STATUS:.metadata.labels.status,VERSION:.metadata.labels.version' +``` + +The controller's reasoning: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n flux-system logs deploy/helm-controller --tail=120 2>&1 | grep -iE "|rollback|remediat|upgrade failed" | tail -30' +``` + +What the workload itself is doing: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n get pods -o wide; \ + kubectl --context k3s-prod -n get events --sort-by=.lastTimestamp | tail -20' +``` + +If the release touches backups or databases, verify them with **fully-qualified names** before deciding +anything is fine — `kubectl get backup -A` prints "No resources found" on this cluster even with six +Velero and 27 CNPG backups present: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get backups.velero.io -A; echo; \ + kubectl --context k3s-prod get backups.postgresql.cnpg.io -A; echo; \ + kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A' +``` + +## Likely causes, most likely first + +### 1. A chart upgrade shipped an incompatible image or value +The recorded instances: **kube-prometheus-stack 65 → 88 brought Grafana 13, which OOM-looped** and +became "the hidden blocker that stalled the whole kps upgrade and the platform tier" (fixed by raising +Grafana 256 → 512 → 768Mi and Prometheus 2 → 3Gi, commit `b2b1aa5`); and **headlamp-0.40.1** shipped a +`-session-ttl` chart/image desync that crashlooped. Remediation: pin the previous version in git +(`update.pin` is the documented escape hatch, docs/11) and push. The rollback has already restored +service; the git change stops it happening again next interval. + +### 2. Not a rollback at all — a first install that cannot succeed +`REASON` will be an install/dependency failure and there will be no earlier `deployed` Helm secret. If +the release is inside the gating `platform` tier, everything downstream is frozen behind it — go to +`/runbooks/flux-reconcile`, which covers the "no matches for kind" / CRD-ordering class. + +### 3. A timeout, not a failure +`platform` has `timeout: 5m` while Longhorn's HelmRelease has `timeout: 15m` — **the tier can time out +while a legitimate Longhorn upgrade is still running.** Give it another interval before declaring a +failure, and check whether pods are still progressing. + +### 4. The rollback itself is stuck +Helm cannot roll back cleanly if the upgrade already mutated something immutable (a StatefulSet +selector, a PVC). Then Ready=False is permanent until an operator intervenes. + +### 5. A post-upgrade health check is failing for an unrelated reason +A node under I/O pressure makes pods slow to become Ready and the upgrade's wait expires. Cross-read +`/runbooks/node-io`. + +## When to escalate or stop + +- **Get the operator if the release is `cilium` or anything in the gating `platform` tier.** A CNI + rollback affects every node's datapath. +- **Do NOT enable `driftDetection.mode: enabled` on a CNI.** During the 2026-09-01 incident the Helm + storage manifest *was* the bad render, so drift correction would have actively re-asserted the broken + value. This is a recorded conclusion, not a preference. +- **Do not `helm rollback` by hand.** helm-controller owns the release; a manual rollback creates drift + it will fight. Use git. +- **Do not suspend the HelmRelease to silence the alert** — that trades a visible failure for + `/runbooks/flux-suspended` and silent drift. +- If the rolled-back release is CNPG or Keycloak, expect knock-on SSO effects and read + `/runbooks/keycloak-down`; note also that a CNPG update under `primaryUpdateMethod: switchover` + cannot change `imageName` and a postgresql parameter in the same reconcile — the operator rejects it. + +--- +Source of truth: `runbooks/update-rollback.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` "2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium" (the measured +Helm revision history and the drift-detection conclusion); the Grafana OOM and headlamp desync. diff --git a/runbooks/vpn-down.md b/runbooks/vpn-down.md new file mode 100644 index 0000000..1cfb6d2 --- /dev/null +++ b/runbooks/vpn-down.md @@ -0,0 +1,183 @@ +# vpn-down — The egress VPN tunnel is down + +**Alert:** `VPNTunnelDown` (critical, 5m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.networking` +**Expression:** +``` +absent(node_network_receive_bytes_total{device="tun0"}) +and on() ( + (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0) + or (count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0) +) +``` + +## What this means here + +There is **no in-cluster egress-gateway pod** on this platform (ADR-0015). The tunnel is a **host +systemd service** — `vpn-egress-tunnel`, OpenVPN to PIA — running on one node, and steering is Cilium +eBPF via a `CiliumEgressGatewayPolicy` (`vpn-egress-steering`). OpenVPN creates and destroys `tun0` with +the connection, so the `tun0` byte-counter series exists **if and only if** the tunnel is up. The rule +keys on that presence rather than `node_network_up`, because `tun0` reports `operstate=unknown` and +`node_network_up` is therefore 0 even when the tunnel is fine. + +Live state, measured 2026-09-13: **the tunnel is up** (`node_network_receive_bytes_total{device="tun0"}` += 1 series) and the gateway node is labelled (`kube_node_labels{...egress_gateway="true"}` = 1 series), +so this alert is armed and quiet. The gateway is **`k3s-prod-3`** (192.168.2.12, `eno1`). + +**The two facts that decide your response:** + +1. **Cilium OSS has no egress-gateway failover.** Gateway-node loss is an outage, not a degradation. + The ADR-0016 supersede drafted on 2026-09-06 is still open. Worse, Cilium OSS picks *"the first node + in lexical ordering"* among nodeSelector matches — so labelling `k3s-prod-1` would **silently steal** + the gateway role and the public egress IP from `k3s-prod-3`. `[vpn_egress_gateway_prod]` is a + separate inventory group holding only `k3s-prod-3` for this reason; run the role as + `-e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3`, with `--list-hosts` first, + every time. +2. **The `absent()` was originally evaluated cluster-wide and became a permanent false alarm.** On a + cluster with no gateway node it could never clear, and from 2026-09-05T21:23Z it fired continuously + into `email-critical` — unbounded noise on the one channel that must stay trustworthy. The two + guards OR'd above are the fix: the alert arms itself the moment a node is labelled, and stays armed + through a kube-state-metrics outage. Before that, the very first version keyed on + `absent(gluetun_vpn_up)`, a metric gluetun's control API never exported at all — a pre-existing + perma-false-alarm. **If this alert is firing, verify the tunnel by hand before believing it.** +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +defaults its current-context to k3s-**dev**, so a command without `--context` silently answers +about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) +Do not run two `dev.sh` invocations at once — they share one container and interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Confirm on the host, not from Prometheus: + +``` +ssh k3s-prod-3 'systemctl status vpn-egress-tunnel --no-pager | head -20; echo; ip -br addr show tun0; echo; ip link show tun0 | head -2' +``` + +Which node is the gateway, and is it labelled? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink; echo; \ + kubectl --context k3s-prod get ciliumegressgatewaypolicies.cilium.io -o yaml | grep -A6 "nodeSelector\|egressIP\|interface"' +``` + +**The check that catches the silent failure** (see cause 1 below) — does Cilium's datapath point at the +interface that actually exists? + +``` +ssh k3s-prod-3 'ip -o link | awk -F: "{print \$1, \$2}" | grep tun0' +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg bpf egress list' +``` + +If the `Egress Ifindex` in that map is not the live `tun0` ifindex, egress is broken **even though this +alert is quiet**. + +Is traffic actually leaving through the tunnel? + +``` +ssh k3s-prod-3 'curl -s --interface tun0 --max-time 10 https://ipinfo.io/ip; echo; nft list table inet vpn_killswitch 2>/dev/null | grep -E "counter|killswitch_" | head' +``` + +## Likely causes, most likely first + +### 1. The tunnel is UP but Cilium is steering at a dead interface (the silent one) +This is the richest failure on the platform and it does **not** set off this alert. +**Cilium resolves a `CiliumEgressGatewayPolicy`'s `interface: tun0` exactly once**, caches +(egress IP, ifindex), and never re-resolves. On `k3s-dev-3`, `cilium-dbg bpf egress list` showed +`Egress Ifindex 3` — **there is no ifindex 3 on that node**; the live `tun0` was **1780**. Egress was +dead for about 80 minutes with nothing alerting and the conformance ledger still reading green. +`egress-gateway-reconciliation-trigger-interval: 1s` does **not** cover it (netlink link events are not +Kubernetes object events). + +Measured behaviour, so you can pick a lever that works: +| action | result | +|---|---| +| annotate the CEGP | map → 1780 in 8s, PIA IP returns | +| `systemctl restart vpn-egress-tunnel` | tun0 1780→1985, **map stays 1780**, still broken | +| the exact `vpn-killswitch` stop/30s/start sequence | still broken 20s+ later | +| set **any** unrelated label on the gateway **node** | recovers within 10s | + +The durable fix is in `roles/vpn_egress_gateway`: the OpenVPN up-script publishes a node label +`homelab.keithjackson.dev/egress-uplink=-` on every connect (via +`systemctl start --no-block`, so it can never block the tunnel), plus a 2-minute convergence timer for +boot. A node label, not a CEGP annotation — a node writing to GitOps state is a drift fight. +**Nothing monitors this coupling yet**; that gap is exactly why dev sat dead for 80 minutes. + +### 2. PIA handed two gateways the same tunnel address +Standing rule: **one PIA server per gateway.** The vaulted `.ovpn` pins one server, so when prod came up +green, dev went down: both gateways dialled `158.173.241.112:1198` and PIA gave both the tunnel address +`10.72.192.7`. Each reconnect steals it — a measured ~2-minute mutual-eviction cycle (dev restarting at +22:57:05, 22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC). Fix: the role var +`vpn_egress_gateway_pia_remote` ("` `") rewrites only the `.ovpn` `remote` line; prod is +pinned to **`158.173.241.135`** (pool `158.173.241.{34,55,112,135,239}`). *"A single green probe is not +a stability check."* + +### 3. PIA "unreachable" — suspect a stale server list first +The 2026-08-22/23 chain went port collision → UDP/53 → 8080 → "MTU" (Cilium VXLAN pod MTU 1450 is real +but irrelevant to a **14-byte** handshake) → "the network blackholes PIA" (**wrong**) before the actual +cause: **gluetun's stale bundled server list**, dialling retired `46.246.x`/`212.112.x` addresses when +live SE Stockholm is `158.173.241.x`. Lesson recorded verbatim: when a client "can't reach" the +provider, suspect the stale server list **first**. + +### 4. The gateway node is down +`absent()` also fires when the gateway node or its node-exporter is gone, so this alert covers +gateway-node-down by design. Go to `/runbooks/node-down`. Remember: **there is no failover** — while +`k3s-prod-3` is down, VPN-classed egress is an outage. + +### 5. Nothing is wrong and the alert is lying +Given the two false-alarm post-mortems above, always confirm `ip -br addr show tun0` on the host before +acting. + +## What the kill-switch does and does not do + +Proven with counters on `k3s-prod-3`, because the intuitive reading is wrong: with +**`bpf.masquerade: true`** (which Egress Gateway requires) SNAT and egress happen in eBPF and +`bpf_redirect()` **skips netfilter POSTROUTING**. Measured: `forward` saddr=pod-CIDR **0**, +`postrouting` oif WAN saddr=pod-CIDR **0**. **What actually fails closed is Cilium's own drop** — +tcpdump during tunnel stop and restart showed **0 packets leaving either device**. The nft table is +kept as defence-in-depth for node-originated tunnel-sourced traffic and was proven both ways: +`ping -I 10.72.192.7` with the ip-rule → 3/3 at 144 ms with `killswitch_accept` advancing; rule removed +→ `killswitch_drop` 0→3 packets / 252 bytes, 100% loss. Every rule carries a `counter`, so: +**if `killswitch_drop` ever counts in normal operation, the datapath has changed.** A reported "leak +window" was **measured as no WAN leak** — a stale egress IP follows the cached ifindex, not a routing +lookup. Note the `.nft` file uses `table` + `flush table`, which clears rules but leaves named-set +elements intact. + +## When to escalate or stop + +- **Escalate immediately if the gateway node is down.** No failover exists; this is an outage with no + in-cluster remedy. +- **Never label a second node `egress-gateway=true`.** Lexical ordering means `k3s-prod-1` would + silently take the role and the public egress IP. +- **Do not restart the tunnel expecting Cilium to follow** — measured above, it does not. If the map is + stale, touching a node label is the lever that works (operator action). +- **Do not remove or "simplify" the nft kill-switch** because the counters read zero. Zero on the pod + path is the expected, measured result of eBPF masquerade; the table guards a different path. +- A leak concern is a security matter (docs/07) — escalate rather than experimenting. + +--- +Source of truth: `runbooks/vpn-down.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` "2026-09-12 — VPN egress on PROD (k3s-prod-3), and three things the repo +believed that are not true"; the 2026-08-22/23 gluetun chain; commit `155df8e` (R-NET-06 re-proven on +prod, recovery ≤3s).