diff --git a/clusters/base/kustomization.yaml b/clusters/base/kustomization.yaml index 6f56415..62c609a 100644 --- a/clusters/base/kustomization.yaml +++ b/clusters/base/kustomization.yaml @@ -50,6 +50,13 @@ resources: # reverse-proxy sidecar (Option B). POST-platform tier (dependsOn platform-config, wait:false); same # non-gating leaf-UI rationale as headlamp/homepage/oauth2-proxy (a UI must never gate the platform). - crafty.yaml # Flux Kustomization -> ./platform/crafty (dependsOn platform-config) + # Alert runbooks — POST-platform tier (dependsOn platform-config, wait:false). Serves the 34 + # runbooks the alert rules' `runbook_url` annotations point at, at /runbooks/; before + # this tier existed that path was a 404 and every alert handed the operator a dead link. Same + # non-gating leaf rationale as headlamp/homepage/crafty. Carries NO postBuild substitution on + # purpose — it renders documentation containing `${...}` shell examples that envsubst would empty; + # see clusters/base/runbooks.yaml. + - runbooks.yaml # Flux Kustomization -> ./runbooks (dependsOn platform-config) # Velero backup controller — POST-platform tier (dependsOn platform-config, wait:false). A backup # controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy. # Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name). diff --git a/clusters/base/runbooks.yaml b/clusters/base/runbooks.yaml new file mode 100644 index 0000000..06e33ac --- /dev/null +++ b/clusters/base/runbooks.yaml @@ -0,0 +1,57 @@ +--- +# Flux Kustomization: alert runbooks — reconciles ./runbooks. +# +# WHAT THIS SERVES. Every alert rule in this repo carries +# `runbook_url: "https://${BASE_DOMAIN}/runbooks/"` — 42 annotations over 34 distinct names. +# Until this tier existed that path returned 404 on the live cluster, so every alert that reached a +# human handed them a dead link at the moment they needed help. This Kustomization renders the +# markdown under ./runbooks into a ConfigMap and a two-replica nginx that serves it at that exact +# prefix. See runbooks/README.md and runbooks/serving/httproute.yaml. +# +# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false) — the same placement and +# the same reasoning as homepage/headlamp/oauth2-proxy. It needs the wildcard TLS cert, the shared +# Gateway (including the `https-apex` listener) and split-horizon DNS, all of which arrive with +# `platform` + `platform-config`; and a leaf web server must never be able to stall the platform +# chain, which is the whole lesson of DECISIONS-LOG's "Headlamp gating stall". wait:false guarantees +# its health can never gate anything even if something is later mistakenly pointed dependsOn it. +# +# NO postBuild.substituteFrom — THE ONE DELIBERATE DIVERGENCE FROM EVERY OTHER TIER, AND THE MOST +# IMPORTANT LINE IN THIS FILE. +# This tier renders the runbook PROSE into a ConfigMap, and runbooks are full of shell: `$POD`, +# `$(kubectl ...)`, and — unavoidably, since they document these very annotations — the literal text +# `${BASE_DOMAIN}`. Flux's postBuild substitution runs envsubst over the ENTIRE built stream, and it +# FAILS OPEN: an undefined `${X}` becomes the empty string while the Kustomization still reports +# Ready (DECISIONS-LOG 2026-09-01 — that is how `k8sServiceHost` was emptied and Cilium broke on 2 +# of 3 dev nodes). Enabled here it would silently mutilate the documentation: every braced expansion +# in every command example blanked, in a file nobody re-reads after writing. Worse, the L0 check +# `conformance/tests/l0-static/flux-postbuild-substitution` scans only files reachable through +# kustomization `resources:`, not `configMapGenerator` inputs, so the damage would be invisible to +# the gate as well. +# The fix is to have nothing to substitute. The only knob this tier would have wanted is the apex +# hostname, and runbooks/serving/httproute.yaml avoids needing it by declaring NO `hostnames:` and +# inheriting the `https-apex` listener's own hostname instead. So this tier is +# environment-independent by construction and R-ENV-02 ("authored once, promoted dev -> prod") holds +# without substitution rather than because of it. +# IF A FUTURE KNOB EVER GENUINELY NEEDS SUBSTITUTION HERE: do not add substituteFrom to this +# Kustomization. Split the serving manifests into their own substituted tier and leave the content +# ConfigMap in an unsubstituted one, or the runbooks lose their shell examples. +# +# NO SOPS decryption stanza: this tier carries no encrypted Secret (markdown plus five plain +# manifests), so — like homepage — neither clusters/dev nor clusters/prod patches it. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: runbooks + namespace: flux-system +spec: + interval: 10m + retryInterval: 2m + timeout: 5m + prune: true + wait: false + dependsOn: + - name: platform-config # wildcard TLS + Gateway (incl. the https-apex listener) first + sourceRef: + kind: GitRepository + name: flux-system + path: ./runbooks diff --git a/conformance/tests/l0-static/alert-runbooks-resolvable/chainsaw-test.yaml b/conformance/tests/l0-static/alert-runbooks-resolvable/chainsaw-test.yaml new file mode 100644 index 0000000..b9b006a --- /dev/null +++ b/conformance/tests/l0-static/alert-runbooks-resolvable/chainsaw-test.yaml @@ -0,0 +1,89 @@ +# ============================================================================= +# L0 STATIC — every alert's runbook_url resolves to a runbook that is actually served +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# +# Every platform-health alert rule in this repo carries +# `runbook_url: "https://${BASE_DOMAIN}/runbooks/"` — 42 annotations across four files, over 34 +# distinct names. For the whole of phases 4-6 there was NO runbooks/ directory and NO HTTPRoute serving +# that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. +# So an alert reaching a human handed them a dead link at the one moment documentation is read, while the +# annotation made the problem look solved — the same green-surface-over-absent-mechanism class as the +# import Job that reported success while storing placeholder secrets, and the guard whose glob silently +# stopped matching. +# +# The link is served now (runbooks/, clusters/base/runbooks.yaml). This test is the guard that keeps it +# true, and it asserts BOTH halves of the claim because the regression is silent in both directions: an +# alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its +# configMapGenerator entry is in git, reviewed, merged — and still 404s in the cluster. "The file exists" +# and "the URL resolves" are different claims; only the second one helps an operator. +# +# GIVEN every `runbook_url` annotation on every PrometheusRule in the repo, +# WHEN each is parsed out of git and resolved against runbooks/, +# THEN (1) each URL has EXACTLY the shape the server resolves — https://${BASE_DOMAIN}/runbooks/, +# the ${BASE_DOMAIN} token (never a literal hostname), no `.md`, no trailing slash, no query; +# (2) every referenced has a runbooks/.md in git; +# (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the +# dev-container kubectl invocation with --context k3s-prod — so a stub cannot satisfy (2) and +# make the link look fixed while still helping nobody; +# (4) every runbook is PROJECTED INTO THE SERVER by runbooks/kustomization.yaml's +# configMapGenerator under the extensionless key the URL asks for; +# (5) route, listener and server agree with the annotations (PathPrefix == the annotations' prefix, +# a sectionName that platform/gateway/gateway.yaml actually defines, the namespace opted in to +# the Gateway, and an nginx location for that same prefix with a text/plain default_type); +# (6) the Flux tier is wired into clusters/base AND declares NO postBuild.substituteFrom — +# substitution would run envsubst over the runbook prose and silently empty every `${...}` +# shell example in it (it fails open: DECISIONS-LOG 2026-09-01, the emptied Cilium +# k8sServiceHost), and the flux-postbuild-substitution check cannot see it because that check +# scans files reachable through `resources:`, not configMapGenerator inputs; +# AND (7) the check is LOAD-BEARING: this Test DECLARES +# `conformance.home.k3s/scope: whole-repo`, so ci/l0.sh Stage 3b's +# `--include-scope whole-repo` unions it into EVERY gate invocation rather than only the runs +# whose touched requirements happen to include R-OBS-04; and ci/gate.sh sets the merge block +# when the l0 stage fails. A SKIP of a scope-declaring check is UNWIRED, never green. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library only); +# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test wraps the SAME +# script. +# +# NOT proven (named in check.sh's header): this does not FETCH the URL. That it returns HTTP 200 with +# real content was proven by an in-cluster probe recorded in the commit message; a live probe needs a +# cluster, which L0 by definition does not have. What is asserted here is every static precondition for +# that 200. It also does not judge whether a runbook is correct or current — (3) is a floor, not a +# review — and it does not prove Alertmanager delivers the annotation to a human (that is the L3 +# `alerting` test). R-OBS-04 is tagged for the narrow slice only: the platform's own health alerts carry +# runbook links that resolve to real, served procedure rather than to a 404. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: alert-runbooks-resolvable + labels: + conformance.home.k3s/R-OBS-04: "true" + annotations: + conformance.home.k3s/requirements: "R-OBS-04" + conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE — this is what makes the check unconditional. Stage 3b of ci/l0.sh runs + # `conformance/run.sh --layer l0 --include-scope whole-repo`, which UNIONS every test carrying this + # annotation into the selection regardless of which requirements the change touched. That union is + # load-bearing here: ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" — + # NOT R-OBS-04 — and maps platform/kubevirt/* and runbooks/* to nothing at all, so the single most + # likely breaking change (adding an alert with a runbook_url, or adding a runbook without its + # configMapGenerator entry) attributes a requirement set that would otherwise FILTER THIS CHECK OUT + # of exactly the change that introduces the bug. Declaring the scope replaces the hand-written + # per-test ci/l0.sh stage that this repo used to need (and that nobody remembered to add). + conformance.home.k3s/scope: "whole-repo" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Every runbook_url on every alert rule resolves to a runbook that exists in git, clears a quality + floor, and is actually projected into the server that answers that path — so an alert can never + hand an operator a dead link. +spec: + steps: + - name: audit-every-alert-runbook-link + description: >- + Parse every runbook_url out of every PrometheusRule and assert the runbook exists, is real, is + served at that exact path, and that the serving tier is wired and unsubstituted. + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/alert-runbooks-resolvable/check.sh b/conformance/tests/l0-static/alert-runbooks-resolvable/check.sh new file mode 100755 index 0000000..8777529 --- /dev/null +++ b/conformance/tests/l0-static/alert-runbooks-resolvable/check.sh @@ -0,0 +1,410 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — alert-runbooks-resolvable +# Serves: R-OBS-04 ("The platform alerts on its own health (nodes, storage, GitOps, certs, +# VPN)." — spec docs/10-observability-and-alerting.md) +# +# WHY THIS EXISTS — A DEAD LINK IN AN ALERT IS WORSE THAN NO LINK +# -------------------------------------------------------------- +# Every platform-health alert rule in this repo carries +# runbook_url: "https://${BASE_DOMAIN}/runbooks/" +# 42 such annotations across four files, over 34 distinct names. For the whole of phases 4-6 +# there was NO runbooks/ directory and NO HTTPRoute serving that path: a live in-cluster probe +# of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. So an alert reaching a +# human at 3am handed them a link that fails, at the one moment the documentation is read. The +# annotation LOOKED like the problem was solved, which is the specific failure this repo keeps +# finding (an import Job that reports success while storing placeholder secrets; a guard whose +# glob silently stopped matching) — a green-looking surface over an absent mechanism. +# +# The link is now served (runbooks/, clusters/base/runbooks.yaml). This check is the guard that +# keeps it true, because the regression is SILENT IN BOTH DIRECTIONS and nothing else catches it: +# * add an alert with a new runbook_url and forget the runbook -> that alert links to a 404, +# and no test, lint, render or policy notices; +# * write runbooks/.md but forget to add it to the ConfigMap generator -> the file is in +# git, reviewable, apparently done, and the URL still 404s in the cluster. "The file exists" +# and "the URL resolves" are different claims and only the second one helps an operator. +# Both halves are asserted below, which is why this check reaches past the markdown into the +# kustomization, the HTTPRoute, the nginx config and the Flux tier. +# +# GIVEN the repository as authored, +# WHEN every `runbook_url` annotation on every PrometheusRule is parsed out of git, +# THEN (1) each URL has EXACTLY the shape the server resolves — +# https://${BASE_DOMAIN}/runbooks/, `${BASE_DOMAIN}` as the token (never a literal +# hostname: a literal is how prod was authored to serve dev's hostnames, see the +# ServiceDefinition schema's note), no `.md` suffix, no trailing slash, no query; +# (2) every referenced has a runbooks/.md in git; +# (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the +# dev-container kubectl invocation — so a one-line stub cannot satisfy (2) and make the +# link look fixed while still helping nobody (the failure mode this check is most likely +# to be defeated by is a placeholder file, not a missing one); +# (4) every runbook markdown file is actually PROJECTED INTO THE SERVER by +# runbooks/kustomization.yaml's configMapGenerator, under the EXTENSIONLESS key the URL +# asks for — the "exists in git but 404s in the cluster" half; +# (5) the route and the server agree with the annotations: the HTTPRoute's PathPrefix is the +# prefix the URLs use, it attaches to a Gateway listener that actually exists, its +# namespace is opted in to that Gateway, and the nginx config answers that same prefix; +# (6) the Flux tier that reconciles all of it is wired into clusters/base AND declares NO +# postBuild.substituteFrom — substitution here would run envsubst over the runbook prose +# and silently empty every `${...}` shell example in it (it fails open: DECISIONS-LOG +# 2026-09-01, the emptied Cilium k8sServiceHost); +# AND (7) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and ci/gate.sh sets the +# merge block when the l0 stage fails. +# +# WHY (7) IS AN UNCONDITIONAL ci/l0.sh STAGE and not left to the requirement-scoped run: Stage 3b +# scopes `conformance/run.sh --layer l0` to the TOUCHED requirements (R-TEST-03), and +# ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" — NOT R-OBS-04 — and +# maps platform/kubevirt/* and runbooks/* to nothing at all. So the single most likely breaking +# change (adding an alert with a runbook_url to platform/monitoring/alerts.yaml, or adding a +# runbook and forgetting the generator entry) attributes a requirement set that would FILTER THIS +# CHECK OUT. Declaring `conformance.home.k3s/scope: whole-repo` makes ci/l0.sh Stage 3b union it into +# EVERY gate invocation instead; a SKIP of a scope-declaring check is UNWIRED, never green. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check): +# - It does not fetch the URL. That it returns HTTP 200 with real content was proven by an +# in-cluster probe (recorded in the commit message and DECISIONS-LOG); a live probe needs a +# cluster, which L0 by definition does not have. What is asserted here is every static +# precondition for that 200: the file, its projection, the key name, the route, the prefix. +# - It does not judge whether a runbook is CORRECT or current. (3) is a floor, not a review: it +# can tell a stub from a runbook, not a good runbook from a stale one. Nothing automatable can. +# - It does not prove Alertmanager actually delivers the annotation to a human. That is the L3 +# `alerting` test. +# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06) and is not consulted. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +# Honest SKIP (never green) if there are no alert rules at all — nothing to audit yet. +if ! grep -rl 'kind: PrometheusRule' "${ROOT}/platform" >/dev/null 2>&1; then + echo "check SKIPPED: no PrometheusRule found under platform/ — no alert rules to audit" >&2 + exit 3 +fi + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +"""Static audit: every alert runbook_url resolves to a runbook that is actually served. + +Pure standard library, no PyYAML — the same constraint and the same style as the neighbouring +L0 checks (lib/coverage.py, lib/status.py). Every parse contract is documented at its parser; +all of them are line-constrained shapes in hand-authored manifests. +""" +from __future__ import annotations + +import re +import sys +from pathlib import Path + +ROOT = Path(sys.argv[1]) +sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree + +FAILURES: list[str] = [] +ADVISORIES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def read(p: Path) -> str: + try: + return p.read_text() + except (OSError, UnicodeDecodeError): + return "" + + +# =============================================================== (1) parse the annotations +# Parse contract: `runbook_url:` is an annotation on a Prometheus alerting rule, authored one per +# line as `runbook_url: ""` (single or double quoted, or bare). Only files that declare +# `kind: PrometheusRule` are scanned, so a runbook_url quoted inside this check or inside prose +# elsewhere in the repo is not mistaken for an alert annotation. Commented-out lines are skipped: +# platform/monitoring/alerts.yaml carries a `# runbook_url points at a docs base` header comment. +SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo" +RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""") + +rule_files = sorted( + p for p in ROOT.rglob("*.yaml") + if ".git" not in p.parts and "kind: PrometheusRule" in read(p) +) +check(len(rule_files) > 0, + "no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; " + "this check would silently audit nothing") + +# url -> [ ":", ... ] +refs: dict[str, list[str]] = {} +for f in rule_files: + rel = f.relative_to(ROOT) + for lineno, line in enumerate(read(f).splitlines(), 1): + if line.lstrip().startswith("#"): + continue + m = RUNBOOK_URL_RE.match(line) + if m: + refs.setdefault(m.group(1), []).append(f"{rel}:{lineno}") + +n_annotations = sum(len(v) for v in refs.values()) +check(n_annotations > 0, + "no runbook_url annotation parsed out of any PrometheusRule — the parser contract broke; " + "this check would pass vacuously while every alert link rots") + +# ---- URL SHAPE. The server resolves exactly one shape. Anything else is a link that 404s even +# though a file with a plausible name exists, so shape is a failure, not a style note. +URL_RE = re.compile(r"^https://\$\{BASE_DOMAIN\}/(runbooks)/([a-z0-9][a-z0-9-]*)$") +names: dict[str, list[str]] = {} # runbook name -> annotation sites +prefixes: set[str] = set() +for url, sites in sorted(refs.items()): + m = URL_RE.match(url) + check(m is not None, + f"runbook_url '{url}' ({', '.join(sites)}) is not the shape this platform serves. " + f"Required: https://${{BASE_DOMAIN}}/runbooks/ — the ${{BASE_DOMAIN}} TOKEN (never a " + f"literal hostname: a literal is how prod got authored to serve dev's public names), a " + f"lowercase-hyphen , and NO '.md' suffix, trailing slash or query. The runbook " + f"server maps the URL path straight onto an extensionless ConfigMap key, so any other " + f"shape is a 404 no matter which files exist") + if m: + prefixes.add(m.group(1)) + names.setdefault(m.group(2), []).extend(sites) + +check(len(prefixes) <= 1, + f"alert annotations use more than one runbook path prefix ({sorted(prefixes)}); the single " + f"HTTPRoute + nginx location can only serve one, so the others are dead links") +PREFIX = (sorted(prefixes) or ["runbooks"])[0] + +# =================================================== (2)(3) the runbooks exist and are real +RUNBOOK_DIR = ROOT / PREFIX +check(RUNBOOK_DIR.is_dir(), + f"{PREFIX}/ is not a directory in this repo, so every one of the {n_annotations} " + f"runbook_url annotations is a dead link") + +# The quality floor. (2) alone is satisfiable by `touch runbooks/node-down.md`, which makes the +# link resolve and still abandons the operator — a worse outcome than the 404, because the 404 at +# least told the truth. These four sections are the operator's path through an incident: what the +# alert means HERE, what to type first, what is probably wrong, and when to stop. `dev.sh` + +# `--context k3s-prod` are required because kubectl does not work on the workstation at all +# (the toolchain lives only in the pinned dev container) and because dev.sh defaults its +# current-context to k3s-DEV — a runbook that omits the context teaches the reader to answer +# questions about the wrong cluster during an incident on this one. +REQUIRED_SECTIONS = ( + "## What this means here", + "## First commands", + "## Likely causes, most likely first", + "## When to escalate or stop", +) +MIN_BYTES = 1500 + +for name, sites in sorted(names.items()): + md = RUNBOOK_DIR / f"{name}.md" + if not md.is_file(): + check(False, + f"runbook '{name}' does not exist: {md.relative_to(ROOT)} is missing, but " + f"{len(sites)} alert annotation(s) link to it ({', '.join(sites[:3])}" + f"{', ...' if len(sites) > 3 else ''}). An alert firing now hands the operator a 404. " + f"Write {PREFIX}/{name}.md (see {PREFIX}/README.md for the required sections)") + continue + body = read(md) + check(len(body) >= MIN_BYTES, + f"{md.relative_to(ROOT)} is {len(body)} bytes, under the {MIN_BYTES}-byte floor — a stub " + f"that makes the link resolve while still helping nobody is worse than the 404 it " + f"replaced, because it looks fixed") + for sec in REQUIRED_SECTIONS: + check(sec in body, + f"{md.relative_to(ROOT)} has no '{sec}' section — the four operator sections are what " + f"make this a runbook rather than a restatement of the alert name") + check("dev.sh" in body, + f"{md.relative_to(ROOT)} never mentions dev.sh — kubectl does not exist on the " + f"workstation; every command in a runbook here must be runnable as " + f"`infra/devcontainer/dev.sh run bash -c '...'` and the runbook must say so") + check("--context k3s-prod" in body, + f"{md.relative_to(ROOT)} never passes --context k3s-prod — dev.sh merges both clusters " + f"and picks a current-context PER INVOCATION — a default that has already changed once " + f"(k3s-dev until dev was decommissioned on 2026-09-13, k3s-prod now). A context-less command " + f"silently answers about whichever cluster the default happens to be, in the middle of an " + f"incident") + +# ADVISORY, not a failure: a runbook nothing links to yet. Writing the runbook BEFORE the alert is +# the right order, and the repo's precedent for "authored but unreferenced" is advisory (see the +# dead-knob advisory in flux-postbuild-substitution/check.sh). +served_extra = {"README"} +if RUNBOOK_DIR.is_dir(): + for md in sorted(RUNBOOK_DIR.glob("*.md")): + stem = md.stem + if stem in served_extra or stem in names: + continue + ADVISORIES.append(f"{md.relative_to(ROOT)} is a runbook no alert links to (written ahead of " + f"its alert, or the alert was removed — advisory, not a failure)") + +# ================================= (4) every runbook is PROJECTED INTO THE SERVER, not just in git +# THIS IS THE HALF A FILE-EXISTENCE CHECK WOULD MISS. The pod serves a ConfigMap, not the git tree. +# Parse contract: runbooks/kustomization.yaml declares one configMapGenerator whose `files:` entries +# are `- =.md`, one per line. The key is EXTENSIONLESS on purpose (the URL carries no +# `.md`, and the ConfigMap volume mount turns each key into a file the nginx root resolves directly). +KUST = RUNBOOK_DIR / "kustomization.yaml" +check(KUST.is_file(), f"{PREFIX}/kustomization.yaml is missing — the Flux tier renders nothing and " + f"every runbook 404s no matter what is in git") +GEN_FILE_RE = re.compile(r"^\s*-\s*([A-Za-z0-9._-]+)=([A-Za-z0-9._/-]+\.md)\s*(?:#.*)?$") +generated: dict[str, str] = {} +kust_text = read(KUST) +if kust_text: + check("configMapGenerator:" in kust_text, + f"{PREFIX}/kustomization.yaml has no configMapGenerator — the runbook markdown is not " + f"rendered into the ConfigMap the server mounts") + for line in kust_text.splitlines(): + if line.lstrip().startswith("#"): + continue + m = GEN_FILE_RE.match(line) + if m: + generated[m.group(1)] = m.group(2) + check(len(generated) > 0, + f"{PREFIX}/kustomization.yaml declares no `=.md` generator entries — the " + f"parser contract broke, or the markdown is being added with implicit keys (which would " + f"carry the '.md' suffix the alert URLs do NOT have, so every link would 404)") + +for name in sorted(names): + check(generated.get(name) == f"{name}.md", + f"runbook '{name}' is not projected into the server: {PREFIX}/kustomization.yaml has no " + f"configMapGenerator entry `- {name}={name}.md`. The file can be in git, reviewed and " + f"merged and the URL still returns 404 — 'the file exists' and 'the URL resolves' are " + f"different claims") + +# And the reverse, so a generator entry cannot point at a file that is not there. +for key, fname in sorted(generated.items()): + check((RUNBOOK_DIR / fname).is_file(), + f"{PREFIX}/kustomization.yaml generates key '{key}' from {PREFIX}/{fname}, which does not " + f"exist — `kustomize build` fails, so the WHOLE tier stops reconciling and every runbook " + f"goes stale at once") + +# ============================== (5) the route, the listener and the server agree with the URLs +ROUTE = RUNBOOK_DIR / "serving" / "httproute.yaml" +NGINX = RUNBOOK_DIR / "serving" / "nginx-config.yaml" +NS = RUNBOOK_DIR / "serving" / "namespace.yaml" +GATEWAY = ROOT / "platform" / "gateway" / "gateway.yaml" + +route_text = read(ROUTE) +check(bool(route_text), f"{PREFIX}/serving/httproute.yaml is missing — nothing routes the " + f"/{PREFIX} prefix to the server") +# Parse contract: a single HTTPRoute with one PathPrefix match and one parentRefs entry naming a +# listener via sectionName; hand-authored, one key per line. +m_path = re.search(r"^\s*type:\s*PathPrefix\s*$\n\s*value:\s*(\S+)\s*$", route_text, re.M) +check(m_path is not None and m_path.group(1) == f"/{PREFIX}", + f"{PREFIX}/serving/httproute.yaml does not match PathPrefix /{PREFIX} " + f"(found {m_path.group(1) if m_path else 'no PathPrefix match'}). The alert annotations point " + f"at /{PREFIX}/; if the route's prefix drifts from the annotations the links 404 while " + f"both files look individually correct") +m_sec = re.search(r"^\s*sectionName:\s*(\S+)", route_text, re.M) +check(m_sec is not None, + f"{PREFIX}/serving/httproute.yaml declares no parentRefs sectionName. The apex host needs the " + f"dedicated apex listener: the wildcard `*.${{BASE_DOMAIN}}` listener does NOT match the bare " + f"apex, so a route without sectionName binds nothing and serves nothing") +if m_sec: + listener = m_sec.group(1) + gw = read(GATEWAY) + check(re.search(rf"^\s*-\s*name:\s*{re.escape(listener)}\s*$", gw, re.M) is not None, + f"{PREFIX}/serving/httproute.yaml attaches to Gateway listener '{listener}', which " + f"platform/gateway/gateway.yaml does not define — the route would report NotAllowed and " + f"never serve") +# An HTTPRoute with no `hostnames:` inherits the listener's hostname, which is how this tier avoids +# needing Flux substitution (see (6)). Either form is routable; a LITERAL hostname is not acceptable, +# for the same reason the schema forbids it in a ServiceDefinition. +m_host = re.search(r"^\s*hostnames:\s*$\n\s*-\s*(\S+)", route_text, re.M) +check(m_host is None or "${BASE_DOMAIN}" in m_host.group(1), + f"{PREFIX}/serving/httproute.yaml hardcodes hostname '{m_host.group(1) if m_host else ''}' — " + f"use the ${{BASE_DOMAIN}} token, or omit `hostnames:` entirely and inherit the listener's") + +ns_text = read(NS) +check('gateway-access: "shared"' in ns_text or "gateway-access: shared" in ns_text, + f"{PREFIX}/serving/namespace.yaml does not carry gateway-access=shared — the shared Gateway " + f"selects namespaces by that label, so without it the HTTPRoute is refused attachment") + +nginx_text = read(NGINX) +check(f"location /{PREFIX}/" in nginx_text, + f"{PREFIX}/serving/nginx-config.yaml has no `location /{PREFIX}/` block — the route would " + f"deliver the request and the server would answer 404 for every runbook") +check("default_type" in nginx_text and "text/plain" in nginx_text, + f"{PREFIX}/serving/nginx-config.yaml does not force a text/plain default_type. The ConfigMap " + f"keys are EXTENSIONLESS, so with the stock MIME table nginx serves them as " + f"application/octet-stream and a browser DOWNLOADS the runbook instead of showing it") + +# ====================================== (6) the Flux tier is wired, and is NOT substituted +FLUX = ROOT / "clusters" / "base" / f"{PREFIX}.yaml" +BASE_KUST = ROOT / "clusters" / "base" / "kustomization.yaml" +flux_text = read(FLUX) +check(bool(flux_text), + f"clusters/base/{PREFIX}.yaml is missing — nothing reconciles {PREFIX}/ into the cluster, so " + f"the runbooks exist in git and the URL still 404s") +check(re.search(rf"^\s*path:\s*\./{re.escape(PREFIX)}\s*$", flux_text, re.M) is not None, + f"clusters/base/{PREFIX}.yaml does not set `path: ./{PREFIX}`") +check(re.search(r"^\s*wait:\s*false\s*$", flux_text, re.M) is not None, + f"clusters/base/{PREFIX}.yaml is not wait:false — a leaf web server that can be legitimately " + f"unhealthy must never gate the reconcile chain (DECISIONS-LOG: 'Headlamp gating stall')") +# Strip comments before testing: this file's own comment block EXPLAINS at length why +# substituteFrom must never be added here, and a naive substring test would fire on the warning. +flux_code = "\n".join(l for l in flux_text.splitlines() if not l.lstrip().startswith("#")) +check("substituteFrom" not in flux_code, + f"clusters/base/{PREFIX}.yaml declares postBuild.substituteFrom. It MUST NOT: this tier " + f"renders documentation into a ConfigMap, and Flux's envsubst would run over the prose and " + f"silently EMPTY every `${{...}}` shell example in every runbook — it fails open, so the " + f"Kustomization would still report Ready (DECISIONS-LOG 2026-09-01, the emptied Cilium " + f"k8sServiceHost). The flux-postbuild-substitution check cannot see this: it scans files " + f"reachable through `resources:`, not configMapGenerator inputs") +check(re.search(rf"^\s*-\s*{re.escape(PREFIX)}\.yaml\b", read(BASE_KUST), re.M) is not None, + f"clusters/base/kustomization.yaml does not list {PREFIX}.yaml — the Flux Kustomization is " + f"authored and never applied, which is the 'looks done, is absent' class this repo has been " + f"bitten by repeatedly") + +# ============================================ (7) this check is load-bearing in the merge gate +L0 = read(ROOT / "ci" / "l0.sh") +GATE = read(ROOT / "ci" / "gate.sh") +# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism, in two halves. +# It is not enough that check.sh exists: the merge gate must SELECT it on every run. The repo used to +# get that with a hand-written "UNCONDITIONAL" stage per whole-repo check in ci/l0.sh, hardcoding the +# path; those stages are gone, because the next whole-repo invariant only got one if somebody +# REMEMBERED to write it. The property is now DECLARED by the test and HONOURED by the gate, so both +# halves are asserted — neither alone is sufficient. The mechanism itself (that the union really +# selects, and that a scope-declared SKIP exits 3 rather than passing green) is proven end-to-end and +# falsifiably by conformance/tests/l0-static/whole-repo-invariants-wired. +SELF_YAML = (ROOT / "conformance" / "tests" / "l0-static" / "alert-runbooks-resolvable" + / "chainsaw-test.yaml") +SELF = read(SELF_YAML) +check(re.search(SCOPE_DECL_RE, SELF, re.M) is not None, + "alert-runbooks-resolvable/chainsaw-test.yaml does not declare " + "conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a " + "change happens to touch a requirement it is labelled with. ci/touched-requirements.sh maps " + "platform/monitoring/* to 'R-OBS-01 R-OBS-02' (NOT R-OBS-04) and maps platform/kubevirt/* and " + "runbooks/* to nothing at all — so adding an alert with a new runbook_url, or adding a runbook " + "without its configMapGenerator entry, would be filtered out of exactly the change that " + "introduces the dead link") +check("--include-scope" in L0, + "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a " + "purely requirement-scoped run again and the scope annotation above is inert decoration — this " + "check would then run only by coincidence") +check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE), + "ci/gate.sh does not set the merge block when the l0 stage fails") + +# ======================================================================== report +print(f"Alert runbook inventory: {n_annotations} runbook_url annotation(s) across " + f"{len(rule_files)} PrometheusRule file(s) -> {len(names)} distinct runbook(s), " + f"prefix /{PREFIX}") +missing = sorted(n for n in names if not (RUNBOOK_DIR / f"{n}.md").is_file()) +print(f" runbooks present: {len(names) - len(missing)}/{len(names)} " + f"projected into the ConfigMap: {sum(1 for n in names if generated.get(n) == f'{n}.md')}/{len(names)}") +if missing: + print(f" MISSING: {', '.join(missing)}", file=sys.stderr) + +if ADVISORIES: + print("\nADVISORIES (not failures):") + for a in ADVISORIES: + print(f" - {a}") + +print() +if not FAILURES: + print(f"check OK: alert-runbooks-resolvable — {CHECKS} assertions (R-OBS-04)") + raise SystemExit(0) +print(f"CHECK FAIL (alert-runbooks-resolvable): {len(FAILURES)} failed, " + f"{CHECKS - len(FAILURES)} passed", file=sys.stderr) +for f in FAILURES: + print(f" FAIL {f}", file=sys.stderr) +raise SystemExit(1) +PY diff --git a/runbooks/README.md b/runbooks/README.md new file mode 100644 index 0000000..11fe78c --- /dev/null +++ b/runbooks/README.md @@ -0,0 +1,142 @@ +# Alert runbooks — home-network-k3s prod + +This is the index page served at `https://home.keithjackson.dev/runbooks/`. + +Every alert rule in this platform carries an annotation of the form + + runbook_url: "https://${BASE_DOMAIN}/runbooks/" + +46 of them, over the 35 names below. Until this directory existed, that path returned **HTTP 404** on the +live cluster: every alert that reached a human handed them a dead link at the exact moment they needed +help. These pages are what makes the link resolve. + +## How to run the commands in these runbooks + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang +is not the cluster being down. + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod`. `dev.sh` merges BOTH clusters into one kubeconfig and picks a +current-context per invocation, and **that default has already changed once** — k3s-dev until dev was +decommissioned on 2026-09-13, k3s-prod now. A command that names its context still means the same thing +next month; one that relies on the default does not, and the failure is silent. `CLUSTER=prod +infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner +naming the cluster — read it. Do not run two `dev.sh` invocations at once; they share one container. + +**Use fully-qualified resource names, always.** Three CRDs on this cluster register the kind `Backup` — +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1`, `velero.io/v1` — and the bare short name resolves to +Longhorn's, which holds zero objects. `kubectl get backup -A` therefore prints **"No resources found"** on +a cluster holding 6 Velero and 27 CNPG backups (verified on prod 2026-09-13). Acting on that false +negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`, +`backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, +`volumes.longhorn.io`, `nodes.longhorn.io`. + +`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git. + +## Alerts you should not trust the silence of (measured 2026-09-13) + +Prometheus scrapes 19 jobs on prod. There is **no** Longhorn, cert-manager, Flux-controller or +Hubble/Cilium job, so these rule groups currently evaluate against nothing and **cannot fire**: + +| Missing scrape job | Inert alerts | +|---|---| +| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornVolumeCapacityHigh`, `LonghornNodeStorageHigh` | +| cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` | +| Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` | +| Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` | +| etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all | + +Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`. + +## The runbooks + +### Nodes +- **`node-down`** — a node's node-exporter is unscrapable; the node may be gone, or its kubelet may have + stopped posting status while the host stays up (the 2026-09-07 shape). +- **`node-io`** — disk I/O pressure/starvation. The etcd WAL shares a partition with Longhorn replicas; + this is the alert that would have called the 2026-09-07 outage and did not exist at the time. +- **`node-memory`** — memory pressure. Includes why `SwapTotal == 0` is not evidence swap is off. +- **`node-disk`** — a node filesystem filling up. `/data` is the one that matters. +- **`node-cpu`** — CPU PSI and run-queue depth, and how to tell CPU contention from I/O contention. + +### Storage +- **`longhorn-degraded`** — a volume has lost redundancy. Normal after any reboot, and the rebuild is the + expensive part. Freeze/thaw commands here. +- **`longhorn-faulted`** — no usable replica. Data at risk; stop before anything destructive. +- **`longhorn-capacity`** — a volume or a node's Longhorn disk near capacity. +- **`pvc-capacity`** — PVC filling up / almost full / predicted to fill. These rules work. +- **`monitoring-capacity`** — Prometheus or Loki running out of disk; don't let the monitor blind itself. + +### GitOps and updates +- **`flux-reconcile`** — a Flux resource stuck at Ready=False. Which tier decides the blast radius. +- **`flux-suspended`** — reconciliation turned off and forgotten. Drift becomes silent. +- **`update-rollback`** — a HelmRelease failed its health gate and is rolling back. +- **`image-automation`** — the git write-back engine is failing (feature is disabled here). +- **`image-policy-stalled`** — an ImagePolicy/ImageRepository cannot resolve a tag. + +### Certificates +- **`cert-issuance`** — ACME issuance failing. The DNS-01 self-check is the usual cause. +- **`cert-expiry`** — a certificate is expiring. Almost always a late symptom of `cert-issuance`. + +### Networking +- **`vpn-down`** — the egress VPN tunnel. Includes the silent egress-gateway staleness that this alert + cannot see, and what the kill-switch actually does. +- **`denied-flows`** — policy-denied flow spike. The Cilium hairpin/identity regression class. + +### Backups and DR +- **`backup-failure`** — a Velero backup failed. The R2 object-tagging incompatibility is documented here. +- **`backup-partial`** — a Velero backup completed with partial failures, and which skips are deliberate. +- **`backup-missed`** — no successful Velero backup in 24h. +- **`cnpg-archiving`** — WAL archiving stuck / no recoverability point. **An availability incident, not + just a backup one.** +- **`cnpg-backup-missed`** — no recent CNPG base backup. Velero does not cover these databases. +- **`cnpg-wal-growth`** — `pg_wal` not being recycled. Never delete WAL by hand. +- **`cnpg-exporter`** — the CNPG SQL exporter erroring: a blind spot that looks exactly like health. +- **`cnpg-primary-spread`** — two or more CNPG primaries on one node. Blast radius, not an outage, + and the rebalance costs ~9s of writes per database — one cluster at a time. + +### Identity +- **`keycloak-down`** — the IdP is down. Blast radius, and the break-glass path that still works. +- **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable. +- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that + every green signal missed. + +### Virtualisation +- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running. +- **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps. +- **`kubevirt-crashloop`** — a virt-launcher pod restarting repeatedly. +- **`cdi-import-stuck`** — a DataVolume import not progressing. Check node I/O before restarting it. + +## Editing a runbook + +Each page is `runbooks/.md` in the `home-network-k3s-impl` repo. Edit it, push to `main`, and Flux +re-renders the ConfigMap and rolls the two-replica server; kustomize's content hash on the generated +ConfigMap name is what triggers the roll. + +Adding a **new** alert with a `runbook_url` requires the matching `runbooks/.md` **and** its +`=.md` entry in `runbooks/kustomization.yaml`. Both are enforced by +`conformance/tests/l0-static/alert-runbooks-resolvable`, which runs unconditionally in the merge gate — +"the file exists in git" and "the URL resolves in the cluster" are different claims, and it checks both. + +Every page must carry the four operator sections (*What this means here*, *First commands*, *Likely +causes, most likely first*, *When to escalate or stop*), mention `dev.sh`, and pass `--context k3s-prod`. +The check enforces that too, because a stub that makes the link resolve while helping nobody is worse than +the 404 it replaced — it looks fixed. + +## Why this route has no login + +Deliberately public. The UIs that declare forward-auth (pihole, jackett, qbittorrent, radarr, sonarr) +really are closed: enforcement is per-pod, each one's HTTPRoute targeting its oauth2-proxy sidecar on +:4180 rather than the app port. If +these pages did too, `keycloak-down` would be unreadable exactly when Keycloak is down, +`keycloak-db` unreadable when its database is down, and `keycloak-client-secrets` unreadable during the +precise failure that broke all 12 OIDC clients on 2026-09-06 behind an entirely green cluster. An +operator who cannot log in is this route's primary reader. The content is operational procedure — metric +names, thresholds, kubectl invocations, topology — and no secrets. See +`runbooks/serving/httproute.yaml` for the full reasoning, including how forward-auth is actually +enforced here and the instruction to leave this route **out** of any Gateway-level ext-authz attachment +if one is ever introduced. + +The running narrative of real incidents on this cluster is `DECISIONS-LOG.md`. diff --git a/runbooks/backup-failure.md b/runbooks/backup-failure.md new file mode 100644 index 0000000..b87a65c --- /dev/null +++ b/runbooks/backup-failure.md @@ -0,0 +1,173 @@ +# backup-failure — A Velero backup failed + +**Alert:** `VeleroBackupFailure` (critical, 5m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups` +**Expression:** `increase(velero_backup_failure_total[1h]) > 0` + +## What this means here + +A Velero backup in the last hour ended in `phase=Failed`. Off-cluster DR (R-STATE-03) is at risk. Velero +backs up the **cluster's Kubernetes objects and its Longhorn PVC data**; it does **not** provide the DR +path for the three CloudNativePG databases — CNPG owns those via its own Barman archive, because a +Velero snapshot of a live Postgres is not a backup. So a Velero failure and a CNPG failure are separate +incidents with separate runbooks (`/runbooks/cnpg-archiving`). + +## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE + +`kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**. +Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and +`velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified +on prod 2026-09-13, where the fully-qualified queries returned **6 Velero backups** and **27 CNPG +backups** while the short name returned nothing. + +This has already caused two false "the backups are missing" conclusions in one night. **A destructive +recovery action taken on that false negative is how a monitoring quirk becomes real data loss.** Use: + +``` +backups.velero.io backups.postgresql.cnpg.io backups.longhorn.io +schedules.velero.io scheduledbackups.postgresql.cnpg.io clusters.postgresql.cnpg.io +``` + +## Where the backups go + +Cloudflare **R2**, since the operator's 2026-09-05 decision (commit `32d6766`). Prod cannot reach the +on-Proxmox MinIO at all — measured from a prod pod, *connection timed out after 15s* — and that MinIO +ran on the same physical host as dev's VMs, so dev's "off-cluster" backup shared a failure domain. +Live config: endpoint `https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`, region +**`auto`**, bucket `k3s-prod-backup`, prefix `velero`, schedule `0 3 * * *`, retention `720h`. +`backupstoragelocation.velero.io/default` should read **Available**. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which backup failed, and why — fully-qualified, always: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get backups.velero.io -A \ + -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,START:.status.startTimestamp,ERRORS:.status.errors,WARN:.status.warnings' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero get backups.velero.io -o yaml | sed -n "/status:/,\$p"' +``` + +The storage location and the controller's log (the R2 error text is in the log, not the CR): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero get backupstoragelocations.velero.io,schedules.velero.io; echo; \ + kubectl --context k3s-prod -n velero logs deploy/velero --tail=150 2>&1 | grep -iE "error|NotImplemented|InvalidArgument|tagging|checksum" | tail -25' +``` + +Did the **volume data** upload even though the backup failed? (This distinction is the whole point — +see cause 1.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero get podvolumebackups.velero.io \ + -o custom-columns=NAME:.metadata.name,BACKUP:.metadata.labels.velero\\.io/backup-name,PHASE:.status.phase,BYTES:.status.progress.bytesDone' +``` + +The plugin pin (cause 1 depends entirely on this): + +``` +grep -n "velero-plugin-for-aws\|checksumAlgorithm\|region" platform/velero/helmrelease.yaml +``` + +## Likely causes, most likely first + +### 1. The R2 object-tagging incompatibility (the known, specific, repo-documented one) +`velero-plugin-for-aws` **v1.14.2** sets PutObject's Tagging unconditionally — +`Tagging: aws.String(o.tagging)` with `o.tagging == ""` — so the SDK emits `x-amz-tagging: ` (empty) on +**every** PutObject. **R2 does not implement object tagging at all** and answers: + +``` +NotImplemented: Header 'x-amz-tagging' with value '' not implemented +``` + +Observed live on prod **twice**, on the write of `velero/backups//velero-backup.json`. The shape +of the failure is the dangerous part: **kopia uploads the volume DATA successfully (its own S3 client), +then the Backup goes `phase=Failed` with NO metadata object — an unrestorable backup.** +**No configuration avoids it** (`config.tagging` with a real value sends a tag set R2 also rejects). The +fix is upstream PR #309, which landed on release-1.14 *after* v1.14.2 was cut, so the image is pinned to +**`velero/velero-plugin-for-aws:v1.14.3-rc.2`**. +**EXIT CONDITION: move to v1.14.3 the moment it ships GA. Do NOT "clean this up" back to v1.14.2 — that +silently breaks every prod backup's metadata.** If this alert fires and the pin has drifted, that is +your answer. + +Two more mandatory R2 settings in the same file: `config.checksumAlgorithm: ""` (R2 rejects the AWS +SDK's `x-amz-checksum-*` headers with **HTTP 400 InvalidArgument** — harmless against MinIO, fatal +against R2) and `region` exactly **`auto`** (it was once hardcoded `us-east-1`, correct-by-accident for +MinIO). + +### 2. A placeholder or wrong endpoint +`clusters/prod` deliberately carried the **loud** placeholder +`https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` rather than inheriting the base (dev) value, +because *"an omitted op means prod silently inherits a dev address — far worse than a failed apply"*. +Check `S3_BACKUP_ENDPOINT`, `S3_BACKUP_REGION`, `VELERO_BACKUP_BUCKET`, `VELERO_BACKUP_PREFIX` and +`BACKUP_CREDENTIALS_SUFFIX` in `clusters/base/cluster-vars.yaml` and the prod overlay. A +`backupstoragelocation` that is not `Available` points straight here. + +### 3. Credentials +The R2 token lives in a SOPS-encrypted Secret (`velero-cloud-prod.sops.yaml`), and the `velero` tier +carries its own `decryption` stanza. An expired or mis-scoped token gives 403s in the log. + +### 4. A volume could not be read +Check `podvolumebackups.velero.io` for a specific volume. Note that NFS-sourced volumes are +**deliberately skipped** by the `velero-volume-policies` ConfigMap — see `/runbooks/backup-partial`, +which is where that belongs. + +### 5. The node or storage was unhealthy during the run +Velero's daily runs at 03:00 and reads Longhorn PVCs. If a node was under I/O pressure or a volume was +degraded, the read can fail. Cross-read `/runbooks/node-io` and `/runbooks/longhorn-degraded`. + +## When to escalate or stop + +- **Escalate promptly.** This is a `critical` because it is a DR alarm: the cluster may currently have no + recent restorable backup. +- **Do not delete a Failed backup to "clean up".** A Failed Velero backup can leave **orphaned kopia + blobs** in the bucket — prod has two such backups from 2026-09-12 (`prod-proof-20260912`, + `prod-vp-probe-20260912`) whose orphaned blobs are **pending owner consent to delete**. Deleting + objects in R2 is an operator decision, not a tidy-up. +- **Do not assume a Completed backup is restorable** if the plugin pin has drifted — the failure mode is + precisely "data uploaded, metadata missing". +- **Be honest about restore**: `dr-restore` is **authored and has never been executed** + (`conformance.home.k3s/never-executed`), so the R2 restore path is unrehearsed. R2 egress is free, + which is the whole reason restore rehearsals are affordable — but nobody has done one yet. +- The three CNPG databases are **not** covered by Velero. If the concern is a database, go to + `/runbooks/cnpg-archiving`. + +--- +Source of truth: `runbooks/backup-failure.md`. Index: `/runbooks/` +Records: commit `2e259a8` and `platform/velero/helmrelease.yaml` lines ~68-95 (the R2 tagging +incompatibility, `checksumAlgorithm`, `region: auto`); commit `32d6766` (the move to R2); commit +`2d5171e` (Velero shipped with no ServiceMonitor, so these alerts could never fire). diff --git a/runbooks/backup-missed.md b/runbooks/backup-missed.md new file mode 100644 index 0000000..a5b8958 --- /dev/null +++ b/runbooks/backup-missed.md @@ -0,0 +1,161 @@ +# backup-missed — No successful Velero backup in 24 hours + +**Alert:** `VeleroNoRecentBackup` (critical, 1h) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups` +**Expression:** `(time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)` + +## What this means here + +A Velero **schedule** has not produced a successful backup in over a day. This is the alert that catches +the quiet failure — not "a backup broke" (that is `/runbooks/backup-failure`) but "backups stopped +happening and nobody noticed". The `{schedule!=""}` filter means it only watches scheduled backups, so +ad-hoc ones do not mask a dead schedule. + +Live shape on prod: two schedules, both `0 3 * * *` — +`schedule.velero.io/velero-daily` and `schedule.velero.io/velero-daily-resources-only` — retention +`720h`, target Cloudflare R2 (bucket `k3s-prod-backup`, prefix `velero`, region `auto`). + +**Why this alert is worth its `critical`:** the precedent is that these rules *could not fire at all*. +Velero shipped with three backup alerts and the chart default `metrics.serviceMonitor.enabled: false`, +confirmed live on prod — `kubectl -n velero get servicemonitor` returned no resources. Recorded verbatim: +*"A nightly backup could fail — or silently stop running — for weeks with nothing to say so."* The +ServiceMonitor now exists (`velero` and `velero/node-agent` are both live scrape jobs as of 2026-09-13), +which is the only reason this alert is real. +## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE + +`kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**. +Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and +`velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified +on prod 2026-09-13: the fully-qualified queries returned **6 Velero backups** and **27 CNPG backups** +while the short name returned nothing. This false negative has already produced two wrong "the backups +are missing" conclusions in one night. **A destructive recovery action taken on it is how a CLI quirk +becomes real data loss.** Always: + +``` +backups.velero.io backups.postgresql.cnpg.io backups.longhorn.io +schedules.velero.io scheduledbackups.postgresql.cnpg.io clusters.postgresql.cnpg.io +``` +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Are the schedules enabled, and when did each last run? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero get schedules.velero.io \ + -o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,PAUSED:.spec.paused,LASTBACKUP:.status.lastBackup,PHASE:.status.phase' +``` + +What backups actually exist (fully-qualified — the short name lies): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get backups.velero.io -A \ + -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,SCHEDULE:.metadata.labels.velero\\.io/schedule-name,START:.status.startTimestamp --sort-by=.status.startTimestamp' +``` + +Is the storage location usable at all? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero get backupstoragelocations.velero.io \ + -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,LASTVALIDATED:.status.lastValidationTime,DEFAULT:.spec.default' +``` + +Is Velero's own Flux tier reconciling, or was it suspended? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \ + -o custom-columns=NAME:.metadata.name,SUSPENDED:.spec.suspend,READY:.status.conditions[?\(@.type==\"Ready\"\)].status | grep -Ei "velero|NAME"' +``` + +And the controller: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero get pods; \ + kubectl --context k3s-prod -n velero logs deploy/velero --tail=120 2>&1 | tail -40' +``` + +## Likely causes, most likely first + +### 1. The Velero Flux tier was suspended and nobody resumed it +This has happened: **Velero was found suspended live, with the suspension never recorded in git**, and +was resumed with `flux resume`. A suspended backup controller is a silent DR gap and is precisely the +class this alert exists for. Remediation (operator): `flux resume kustomization velero`, and see +`/runbooks/flux-suspended` for why an undocumented live suspension is itself a problem. + +### 2. The schedule is paused +`schedules.velero.io` has `spec.paused`. Note from the live listing that +`velero-daily-resources-only` shows an empty PAUSED column while `velero-daily` shows `false` — read the +field, not the column width. A paused schedule is a git change to unpause. + +### 3. Every run is failing rather than not running +Then `backups.velero.io` will show recent `Failed` entries and this alert is a downstream symptom. Go to +`/runbooks/backup-failure` — and in particular check the plugin pin +(`velero/velero-plugin-for-aws:v1.14.3-rc.2`), because the R2 tagging bug produces exactly this: volume +data uploads, the backup goes Failed, and no successful backup ever lands. + +### 4. The BackupStorageLocation is not Available +A wrong endpoint, a bad credential, or the placeholder. `clusters/prod` deliberately carried the loud +placeholder `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` rather than inheriting the base (dev) +value, because *"an omitted op means prod silently inherits a dev address — far worse than a failed +apply"*. Check `S3_BACKUP_ENDPOINT`, `S3_BACKUP_REGION` (`auto`), `VELERO_BACKUP_BUCKET`, +`VELERO_BACKUP_PREFIX`, `BACKUP_CREDENTIALS_SUFFIX` (`-prod`). + +### 5. Velero cannot schedule a pod +Node pressure, or the node-agent DaemonSet unhealthy. Cross-read `/runbooks/node-io`. + +### 6. The metric, not the backup, is missing +If `velero_backup_last_successful_timestamp` has no series at all the rule cannot fire — which is the +historical state. Confirm the `velero` scrape job is up (`/runbooks/targets-down`) before trusting either +its silence or its firing. + +## When to escalate or stop + +- **Escalate: this is a DR alarm and 24h is already a real window of exposure.** +- **Do not create an ad-hoc backup and consider the alert answered.** The expression filters + `{schedule!=""}`, so a manual backup will not clear it, and more importantly it does not fix the + schedule. (An operator's ad-hoc run for reassurance is + `velero backup create --from-schedule velero-daily`.) +- **Do not conclude "no backups exist" from `kubectl get backup`.** See the trap above. +- **Remember the databases are separate.** `keycloak-db`, `nextcloud-db` and `semaphore-db` are covered + by CNPG's Barman archive, not by Velero. This alert being clear says nothing about them, and vice + versa — check `/runbooks/cnpg-backup-missed` too. +- **Be honest about restore:** `dr-restore` is authored and has **never been executed**, so the R2 + restore path is unrehearsed. Free R2 egress makes a rehearsal cheap; it is still outstanding. + +--- +Source of truth: `runbooks/backup-missed.md`. Index: `/runbooks/` +Records: commit `2d5171e` (Velero had no ServiceMonitor, so these alerts could never fire); +`DECISIONS-LOG.md` (Velero found suspended with the suspension never in git; `dr-restore` never +executed); commit `32d6766` (R2 decision and endpoint knobs). diff --git a/runbooks/backup-partial.md b/runbooks/backup-partial.md new file mode 100644 index 0000000..af522af --- /dev/null +++ b/runbooks/backup-partial.md @@ -0,0 +1,164 @@ +# backup-partial — A Velero backup completed with partial failures + +**Alert:** `VeleroBackupPartialFailure` (warning, 5m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups` +**Expression:** `increase(velero_backup_partial_failure_total[1h]) > 0` + +## What this means here + +The backup finished, but some resources or volumes were skipped or errored. It is a warning rather than a +critical because *most of it worked* — and that is exactly what makes it dangerous: a +`PartiallyFailed` backup looks like a backup in every listing, and the thing it silently omitted may be +the thing you need. + +**The first question is always: was the skip deliberate?** On this cluster a great deal of skipping is +intentional and correct. +## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE + +`kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**. +Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and +`velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified +on prod 2026-09-13: the fully-qualified queries returned **6 Velero backups** and **27 CNPG backups** +while the short name returned nothing. This false negative has already produced two wrong "the backups +are missing" conclusions in one night. **A destructive recovery action taken on it is how a CLI quirk +becomes real data loss.** Always: + +``` +backups.velero.io backups.postgresql.cnpg.io backups.longhorn.io +schedules.velero.io scheduledbackups.postgresql.cnpg.io clusters.postgresql.cnpg.io +``` + +## The deliberate exclusions (know these before calling anything a failure) + +**NFS-sourced volumes are skipped on purpose, and the reason is a near-miss worth remembering.** The +opt-out file-system-backup schedule (`defaultVolumesToFsBackup: true`, excluding only `monitoring` and +`unifi`) would have read the NFS media library **three times** — `plex-media`, `sonarr-media` and +`radarr-media` are all bound to the same `192.168.2.102:/mnt/Storage/plex` export, **8.32 TiB +measured** — plus `nextcloud-legacy-archive` (270 GiB), pushing roughly **8.6 TiB to R2 nightly with +720h retention**. Namespace exclusion could not express the fix, because those namespaces also hold real +Longhorn PVCs that must be backed up. + +The fix is a **per-volume opt-out by SOURCE TYPE**: the ConfigMap `velero-volume-policies` skips any +volume whose source is NFS, and the schedule references it. The per-pod +`backup-volumes-excludes` annotation was **rejected for failing OPEN** (forget the annotation and you +back up 8 TiB). Proven on prod: a Backup of namespace `nextcloud` produced exactly **one** +PodVolumeBackup (`data`, Longhorn, 875,417,102 bytes, Completed) and the NFS volume produced none. +**Residual gap, stated honestly: the rule matches NFS only.** A future non-NFS bulk source would not be +covered. + +**The three CNPG databases are also out of scope by design.** `keycloak-db`, `nextcloud-db` and +`semaphore-db` have their own DR path (Barman WAL archive + base backups to R2) because a Velero snapshot +of a live Postgres is not a backup. If a `*-db-N` volume is skipped, that is correct — see +`/runbooks/cnpg-archiving` for the path that does cover them. + +**`monitoring` and `unifi` are excluded** from file-system backup as well. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which backup, and what exactly it warned about: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get backups.velero.io -A \ + -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,ERRORS:.status.errors,WARN:.status.warnings,START:.status.startTimestamp' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero get backups.velero.io -o yaml | sed -n "/status:/,\$p"' +``` + +Per-volume results — this tells you whether a skip was the NFS policy or a real failure: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero get podvolumebackups.velero.io \ + -o custom-columns=NAME:.metadata.name,BACKUP:.metadata.labels.velero\\.io/backup-name,POD:.spec.pod.name,VOL:.spec.volume,PHASE:.status.phase,BYTES:.status.progress.bytesDone' +``` + +The volume policy that does the deliberate skipping: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero get cm velero-volume-policies -o yaml' +``` + +The controller's own account of what it skipped: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n velero logs deploy/velero --tail=200 2>&1 | grep -iE "skip|excluded|warn|error" | tail -30' +``` + +## Likely causes, most likely first + +### 1. The deliberate NFS / CNPG / monitoring exclusions +Most common by far, and correct. Confirm against the `velero-volume-policies` ConfigMap and the +schedule's excluded namespaces. **No action.** If you are tempted to "fix" it, re-read the 8.6 TiB +near-miss above. + +### 2. A resource Velero could not serialise +Usually a CRD whose API was unavailable at the moment of the backup, or an admission webhook refusing a +read. The log names the group/kind. Often transient; if it recurs for the same kind, that kind is +genuinely not backed up. + +### 3. A volume was busy, degraded, or on a node under pressure +Velero's daily runs at 03:00 against Longhorn PVCs. A degraded volume or a node with high I/O-PSI can +fail one volume's read while the rest succeed. Cross-read `/runbooks/longhorn-degraded` and +`/runbooks/node-io`. + +### 4. A partial R2 write +Related to the tagging/checksum incompatibilities in `/runbooks/backup-failure`: those normally produce a +hard `Failed`, but a partially-written backup is worth checking the plugin pin for +(`velero/velero-plugin-for-aws:v1.14.3-rc.2` — do **not** revert to v1.14.2). + +### 5. A namespace was created or deleted mid-backup +Benign race. Leaked test namespaces have existed on this cluster and been deleted; a namespace +disappearing under Velero produces exactly this warning. + +## When to escalate or stop + +- **Decide, and record, whether the skipped items matter.** The honest failure mode here is shrugging at + a `PartiallyFailed` and discovering later that something was never covered. If the skip was not one of + the documented exclusions, treat it as an uncovered resource. +- **Do not add namespaces to the exclusion list to quiet the alert.** Those namespaces hold real + Longhorn PVCs; namespace-level exclusion is the mistake the source-type policy exists to avoid. +- **Do not switch to the per-pod `backup-volumes-excludes` annotation** — it was rejected for failing + OPEN, and the consequence of forgetting it is 8 TiB per night. +- **Do not delete a PartiallyFailed backup** — like a Failed one it can leave orphaned kopia blobs, and + deleting R2 objects is an operator decision. +- Escalate if the same resource or volume is skipped on consecutive nights: that is a coverage gap, which + is an R-STATE-03 matter and not a transient. + +--- +Source of truth: `runbooks/backup-partial.md`. Index: `/runbooks/` +Records: commit `2e259a8` (the 8.6 TiB near-miss, the source-type volume policy, and the rejection of the +fail-open annotation); `docs/BULK-NFS-STORAGE.md`. diff --git a/runbooks/cdi-import-stuck.md b/runbooks/cdi-import-stuck.md new file mode 100644 index 0000000..204559e --- /dev/null +++ b/runbooks/cdi-import-stuck.md @@ -0,0 +1,172 @@ +# cdi-import-stuck — A CDI DataVolume import is not progressing + +**Alert:** `CDIDataVolumeImportStuck` (warning, 30m) +**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt` +**Expression:** `sum(kubevirt_cdi_import_pods_high_restart) > 0` + +## What this means here + +A CDI DataVolume import or clone has not completed after 30 minutes, so **the VM disk it backs cannot +start**. `kubevirt_cdi_import_pods_high_restart` counts importer pods stuck restarting; a sustained +positive value means the import is not making progress rather than merely being slow. + +The 30-minute `for` is generous on purpose: on this hardware a genuine large import is legitimately slow. +The reference case is the **UniFi VM's 60 GiB import**, and the `apps-vms` tier is deliberately +**non-gating** (`wait: false`, `dependsOn: platform-config`) precisely because a VM's image import plus +in-guest install is legitimately long-not-Ready. + +**Two things to know before triaging:** + +1. **Imports are the single heaviest write this platform performs**, and the etcd WAL (`/data/k3s`) shares + one partition with every Longhorn replica (`/data/longhorn`). A 60 GiB import is exactly the class of + write load behind the 2026-09-07 `k3s-prod-1` NotReady outage. So "restart the import" is not a free + action — check `/runbooks/node-io` first. +2. **There is a known false-positive precedent in this area.** A broken HTTP CDI ServiceMonitor once caused + a false `CDIOperatorDown` (commit `6a7e9d1`), and the sibling `KubeVirtControlPlaneDown` once used a + non-existent metric with the wrong job label (`b57b3e9`). The metric name in this rule is flagged in + the file as one to confirm against the deployed CDI. `cdi-prometheus-metrics` **is** a live scrape job + on prod (measured 2026-09-13), so the target is up — but verify the series exists before trusting either + this alert's firing or its silence. + +Platform versions: **CDI v1.66.0** with **KubeVirt v1.9.0**, on real nested KVM. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which DataVolume, and how far it got: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get datavolumes.cdi.kubevirt.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PHASE:.status.phase,PROGRESS:.status.progress,RESTARTS:.status.restartCount' +``` + +The importer pod is where the real error is: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get pods -A | grep -E "importer|upload|clone"; echo; \ + kubectl --context k3s-prod -n unifi logs -l app=containerized-data-importer --tail=100 2>&1 | tail -50' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n unifi describe pod -l app=containerized-data-importer 2>&1 | sed -n "/Events:/,\$p"' +``` + +Is there anywhere to put it — and is the scratch class configured? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get pvc -A | grep -Ei "scratch|dv|rootdisk"; echo; \ + kubectl --context k3s-prod get sc; echo; \ + kubectl --context k3s-prod get cdis.cdi.kubevirt.io -o yaml | grep -A5 scratchSpace' +``` + +Is the node being flattened by the import? **Check this before restarting anything:** + +``` +ssh k3s-prod-1 'cat /proc/pressure/io; df -h /data' +``` + +Can the importer reach the image source at all? (Default-deny is the baseline everywhere.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \ + hubble observe --verdict DROPPED --last 100 -o compact | grep -i import' +``` + +Does the metric this alert depends on actually exist? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + for q in "up{job=\"cdi-prometheus-metrics\"}" "count(kubevirt_cdi_import_pods_high_restart)"; do \ + printf "%-50s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \ + | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done' +``` + +## Likely causes, most likely first + +### 1. Not enough schedulable storage for the target or the scratch space +The recorded failure, and the reason a whole StorageClass exists: **the 3-replica default class failed the +UniFi 60Gi import with insufficient storage / `ReplicaSchedulingFailure`.** The fix was a dedicated +single-replica **`longhorn-scratch`** class (`numberOfReplicas: 1`, `dataLocality: best-effort`), +referenced by the CDI CR's `scratchSpaceStorageClass`. If scratch is pointed at the 3-replica default, +this is your cause. Note that a 60 GiB import needing 3x replication plus scratch space is a very +different ask from 60 GiB. + +### 2. An egress or default-deny block on the image source +Import pods are ordinary pods under the cluster's default-deny baseline (R-NET-01). The alert's own +description names this: *"often an egress/default-deny block on the image source, or a slow mirror."* +Remediation is an explicit allow in git; see `/runbooks/denied-flows` — and in particular the +post-Cilium-upgrade finding that `toEntities: world` no longer covers a destination that resolves to an +in-cluster backend. + +### 3. The importer cannot open the raw Block device +`nonroot-devices: true` is set on the k3s servers because the **non-root CDI importer must open a raw +Block device that Kubernetes never chowns for it** (commit `df5979f`). A permission error in the importer +log against a Block-mode PVC is this. `longhorn-vm` is Block-mode only, so VM disks are always in scope. + +### 4. The source is slow, unreachable, or the URL is wrong +A 404/403 or a stalled transfer appears plainly in the importer log. A slow mirror is a genuine cause — the +30m `for` already allows for it, so check throughput rather than assuming a fault. + +### 5. Node I/O pressure is starving the import +Self-reinforcing: the import is heavy, the disk is shared with etcd, and a node under I/O pressure slows +the import which then runs longer. `/runbooks/node-io`. + +### 6. Longhorn is degraded +An import onto a degraded volume competes with the rebuild. `/runbooks/longhorn-degraded`. + +### 7. The alert is wrong +Given the CDI/KubeVirt false-positive precedents, if the DataVolumes all read `Succeeded` this is a +monitoring fault. Fixing the rule is a real change, not a shrug — a permanently-firing alert is a +recorded, expensive failure mode here. + +## When to escalate or stop + +- **Check node I/O pressure BEFORE restarting or recreating an import.** A 60 GiB re-import is hours of + write load on the partition the embedded etcd fsyncs to, and that combination took a node out of the + cluster on 2026-09-07. If I/O-PSI is already elevated, wait or freeze Longhorn rebuilds first + (`/runbooks/longhorn-degraded`). +- **Do not delete the target DataVolume/PVC casually** — that guarantees a full re-import. +- **Do not point `scratchSpaceStorageClass` at the 3-replica default** to "keep things consistent". The + single-replica class exists because the default one demonstrably failed. +- **Do not widen a NetworkPolicy to `world`** to let an import out; author the narrow allow. +- Scope it honestly: the VM tier is non-gating, so a stuck import does not affect the container platform. + The exception is the UniFi VM, which is the home network's controller. +- Escalate for k3s server-flag changes (`nonroot-devices`) or CDI CR changes — Ansible and operator work. + +--- +Source of truth: `runbooks/cdi-import-stuck.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` (the UniFi 60Gi import failing on the 3-replica default; `longhorn-scratch`); +commits `df5979f` (`nonroot-devices`), `6a7e9d1` and `b57b3e9` (false CDI/KubeVirt alerts). diff --git a/runbooks/cert-expiry.md b/runbooks/cert-expiry.md new file mode 100644 index 0000000..eceabc6 --- /dev/null +++ b/runbooks/cert-expiry.md @@ -0,0 +1,146 @@ +# cert-expiry — A certificate is expiring soon, or imminently + +**Alerts:** `CertificateExpiringSoon` (warning, <14d, 1h) and `CertificateExpiryImminent` (critical, <3d, 1h) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.certificates` +**Expression:** `(certmanager_certificate_expiration_timestamp_seconds - time()) < (14 * 24 * 3600)` / `(3 * 24 * 3600)` + +## What this means here + +cert-manager has a Certificate whose expiry is inside 14 days (warning) or 3 days (critical) and it has +not renewed. cert-manager normally renews at two-thirds of lifetime — about 60 days into a 90-day Let's +Encrypt certificate — so **by the time this fires, renewal has already been failing for weeks**. Treat +it as a late symptom of `/runbooks/cert-issuance`, not as a separate problem: the fix is almost always +"why is issuance failing", and expiry is the clock running out on that. + +What is at stake concretely. There is effectively **one certificate that matters**: `wildcard-home` in +the `gateway` namespace, secret `wildcard-home-tls`, requesting **both** `*.${BASE_DOMAIN}` **and** +`${BASE_DOMAIN}` from `${ACME_CLUSTER_ISSUER}` (= `letsencrypt-prod` on prod). Every HTTPS listener on +the shared Gateway terminates with it — the wildcard `https` listener and the `https-apex` listener both +reference `wildcard-home-tls`. If it expires: + +- every public hostname serves an untrusted certificate; +- **SSO breaks in-cluster**, because OIDC clients validate the issuer's TLS. This is not hypothetical: + while the wildcard was still LE-**staging**, in-cluster clients rejected it with + `x509: certificate signed by unknown authority`, which is what forced the move to `letsencrypt-prod`; +- and the page you are reading right now is served over that same certificate. + +**There is no history of an actual expiry on this cluster.** The recorded history is all issuance +failures. Say so rather than inventing precedent — but note the corollary: the renewal path has never +been exercised under stress, so do not assume it will quietly work. + +### Verify the alert can even fire (measured 2026-09-13) + +**This alert is currently INERT.** `count(certmanager_clock_time_seconds)` and +`count(certmanager_certificate_expiration_timestamp_seconds)` both returned **0 series** on prod: +there is no cert-manager scrape job among Prometheus's 19 jobs, so all three certificate rules evaluate +against nothing. **Certificate expiry is unmonitored today.** Check expiry by hand with the commands +below, and see `/runbooks/targets-down` for the full list of inert rule groups. Adding cert-manager's +ServiceMonitor is open work. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Every certificate and its real expiry, straight from the CRs (works with no metrics): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get certificates.cert-manager.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,NOTAFTER:.status.notAfter,RENEWAL:.status.renewalTime,ISSUER:.spec.issuerRef.name' +``` + +What the Gateway is actually serving (the answer that matters), read from outside the cluster with an +explicit SNI — testing HTTPS from **inside** a pod is misleading here, and an IP+SNI probe without the +right SNI returns curl exit 35: + +``` +curl -sv --resolve home.keithjackson.dev:443:192.168.2.20 https://home.keithjackson.dev/runbooks/ 2>&1 | grep -E "expire|subject|issuer|SSL certificate" +``` + +``` +echo | openssl s_client -connect 192.168.2.20:443 -servername grafana.home.keithjackson.dev 2>/dev/null | openssl x509 -noout -subject -issuer -dates +``` + +Is a renewal in flight or stuck? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get certificaterequests.cert-manager.io,orders.acme.cert-manager.io,challenges.acme.cert-manager.io -A; echo; \ + kubectl --context k3s-prod -n cert-manager logs deploy/cert-manager --tail=80 | tail -40' +``` + +## Likely causes, most likely first + +### 1. Issuance has been failing since the last successful renewal +The dominant cause, and the one with real history. Go to `/runbooks/cert-issuance` — it covers the +DNS-01 self-check failure (fixed with `dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"` and +`dns01RecursiveNameserversOnly: true`), the Cloudflare token, and the ACME order/challenge chain. Fixing +issuance fixes expiry; nothing else does. + +### 2. Two clusters racing the same `_acme-challenge` TXT record +A specific, recorded hazard for any future two-cluster window. `infra/PROD-PROVISIONING.md` §10 records +that after the 2026-09-04 incident, `platform-config` **would have** requested a +`*.home.keithjackson.dev` wildcard from letsencrypt-prod for a certificate dev already held; both +clusters would then have raced each other's `_acme-challenge` TXT records on one Cloudflare zone and +**neither would reliably solve DNS-01**. It never fired only because the tier was suspended. If a second +cluster exists, confirm which one owns the zone before renewing. + +### 3. Let's Encrypt rate limits +One wildcard is well inside the limits, so this is unlikely — unless something has been retrying in a +loop. `cert-manager` logs will say `too many certificates already issued`. Remediation: stop the loop, +wait out the window; do **not** switch to letsencrypt-staging as a workaround, because in-cluster OIDC +clients reject the staging chain (that is the bug that forced LE-prod). + +### 4. The issuer or its credential changed +`${ACME_CLUSTER_ISSUER}` is a cluster-vars knob; the Cloudflare API token is a SOPS-encrypted Secret. +Note the standing rule: **non-secret, environment-specific values must not live inside an encrypted +Secret** — `${BASE_DOMAIN}` substitution can never reach inside one, which is exactly how prod's OIDC +clients once ended up pointed at dev's Keycloak. + +### 5. The Certificate is Ready but the Secret is stale +Check the Secret's actual notAfter (the `openssl` command above) rather than the CR's status. The Gateway +reads the Secret. + +## When to escalate or stop + +- **`CertificateExpiryImminent` on `wildcard-home` is a cluster-wide outage in under 3 days**, including + SSO and including this runbook server. Get the operator immediately. +- **Do not switch to `letsencrypt-staging`** to make issuance succeed. It will issue, and it will break + every in-cluster OIDC client with `x509: unknown authority`. +- **Do not delete the `wildcard-home-tls` Secret** to force a reissue while issuance is broken — you + would replace a certificate that still has days left with none at all. +- **Do not hand-edit the Secret.** cert-manager owns it; the durable change is in git. +- Verify DNS at the **authority** (`dig @betty.ns.cloudflare.com _acme-challenge.home.keithjackson.dev TXT`), + never at a resolver: 1.1.1.1 has served a deleted TXT record for minutes here and produced a false + failure diagnosis. + +--- +Source of truth: `runbooks/cert-expiry.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` "Fix 5 — cert-manager DNS-01 self-check via public resolvers" and the +LE-staging → LE-prod switch (Session 7); `infra/PROD-PROVISIONING.md` §10 (the queued ACME race). diff --git a/runbooks/cert-issuance.md b/runbooks/cert-issuance.md new file mode 100644 index 0000000..b278f30 --- /dev/null +++ b/runbooks/cert-issuance.md @@ -0,0 +1,165 @@ +# cert-issuance — A Certificate is not Ready (ACME issuance is failing) + +**Alert:** `CertificateNotReady` (warning, 1h) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.certificates` +**Expression:** `certmanager_certificate_ready_status{condition="False"} == 1` + +## What this means here + +cert-manager cannot issue or renew a certificate. The one that matters is `wildcard-home` in the +`gateway` namespace — secret `wildcard-home-tls`, requesting **both** `*.${BASE_DOMAIN}` **and** +`${BASE_DOMAIN}` from `${ACME_CLUSTER_ISSUER}` (`letsencrypt-prod` on prod). Every HTTPS listener on the +shared Gateway terminates with it, so issuance failing is a slow-burning cluster-wide outage: it does +nothing today and takes down every hostname plus in-cluster SSO when the existing certificate expires. +The 1h `for` reflects that — a failing order is worth investigating today, not paging at 3am. + +The solver is **DNS-01 against Cloudflare**, which matters because DNS-01 has a failure mode this +cluster has already hit: + +**The self-check, not the challenge, is the usual failure.** On 2026-08-16 the wildcard stalled with the +challenge at `Presented=true` — the Cloudflare TXT record *was* created, so the API token worked — while +cert-manager's propagation self-check looped **"DNS record not yet propagated"**. The record was +publicly resolvable at 1.1.1.1 and 8.8.8.8; the NAT'd cluster's own DNS could not see it, because +split-horizon CoreDNS answers authoritatively for the zone and never forwards. The fix is live in +`platform/cert-manager/helmrelease.yaml`: + +```yaml +dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53" +dns01RecursiveNameserversOnly: true +``` + +The challenge went pending → valid and the certificate Issued. If you find those two lines missing or +changed, that is your answer. + +### Verify the alert can even fire (measured 2026-09-13) + +**This alert is currently INERT.** `count(certmanager_clock_time_seconds)` returned **0 series** on +prod — there is no cert-manager scrape job among Prometheus's 19 jobs, so all three certificate rules +evaluate against nothing. Certificate health must be read from the CRs, as below. See +`/runbooks/targets-down` for the full list of inert rule groups. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Walk the chain top-down — Certificate → CertificateRequest → Order → Challenge. The failure is almost +always visible at one specific level: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get certificates.cert-manager.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,NOTAFTER:.status.notAfter' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get certificaterequests.cert-manager.io -A; echo ==; \ + kubectl --context k3s-prod get orders.acme.cert-manager.io -A; echo ==; \ + kubectl --context k3s-prod get challenges.acme.cert-manager.io -A -o wide' +``` + +The controller log, which carries the actual ACME error text: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n cert-manager logs deploy/cert-manager --tail=150 2>&1 | grep -iE "error|propagat|challenge|order|rate" | tail -30' +``` + +Is the TXT record really there? **Ask the authority, never a resolver** — 1.1.1.1 has served a deleted +TXT record here for minutes and produced a false failure diagnosis: + +``` +dig +short @betty.ns.cloudflare.com _acme-challenge.home.keithjackson.dev TXT +dig +short @1.1.1.1 _acme-challenge.home.keithjackson.dev TXT +``` + +Is the resolver override still configured? + +``` +grep -n "dns01Recursive" platform/cert-manager/helmrelease.yaml +``` + +The issuer and its credential: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusterissuers.cert-manager.io \ + -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message' +``` + +## Likely causes, most likely first + +### 1. The DNS-01 propagation self-check cannot resolve the TXT record +The recorded failure. Signature: Challenge `Presented=true`, status looping on *"DNS record not yet +propagated"*, and `dig @betty.ns.cloudflare.com` **does** return the TXT. Cause: cert-manager is asking +the cluster's own resolver, which is split-horizon CoreDNS answering authoritatively for the zone and +never forwarding. Remediation: the two `dns01Recursive*` values above, in git. + +### 2. The Cloudflare API token is wrong, expired, or scoped too narrowly +Signature: Challenge never reaches `Presented=true`, and the log carries a Cloudflare 403/6003. The +token lives in a SOPS-encrypted Secret. Note the standing rule from the OIDC incident: **non-secret +environment-specific values must never live inside an encrypted Secret** — `${BASE_DOMAIN}` +substitution cannot reach inside one, and that is precisely how prod's OIDC clients once ended up +pointed at dev's Keycloak. Keep hostnames out; keep only the token in. + +### 3. `EXTERNALDNS_DOMAIN_FILTER` / zone confusion +Recorded as finding F-DNS1: the filter **must be the ZONE** (`keithjackson.dev`), not the subdomain. +A zone mismatch means the TXT record is created in the wrong place or not at all. + +### 4. Two clusters racing the same `_acme-challenge` record +`infra/PROD-PROVISIONING.md` §10: after the 2026-09-04 incident `platform-config` would have requested a +`*.home.keithjackson.dev` wildcard from letsencrypt-prod for a certificate dev already held; both +clusters would then have raced each other's `_acme-challenge` TXT records on one Cloudflare zone and +**neither would reliably solve DNS-01**. Only the tier being suspended prevented it. If any second +cluster exists, establish which one owns the zone first. + +### 5. Let's Encrypt rate limit +`too many certificates already issued` in the log. One wildcard is well inside the limits, so this means +something has been retrying in a loop. Stop the loop and wait out the window. + +### 6. A new Certificate for a hostname that does not resolve +A newly added service whose DNS is not yet published cannot solve DNS-01. Check ExternalDNS first. + +## When to escalate or stop + +- **Escalate if `wildcard-home` has been not-Ready for more than a day**, and check + `/runbooks/cert-expiry` for how much runway the existing certificate has. Under 3 days of runway is an + operator page. +- **Never switch `${ACME_CLUSTER_ISSUER}` to letsencrypt-staging** to get a green certificate. Staging + issues fine and then breaks every in-cluster OIDC client with `x509: certificate signed by unknown + authority` — that is the bug that forced LE-prod in the first place. +- **Do not delete the existing `wildcard-home-tls` Secret** while issuance is broken. A stale but valid + certificate is far better than none. +- **Do not create certificates by hand.** cert-manager owns them; the durable change is in git. +- Rotating the Cloudflare token is an operator action (SOPS, DR path). + +--- +Source of truth: `runbooks/cert-issuance.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` "Fix 5 — cert-manager DNS-01 self-check via public resolvers" +(commit ~`59dd3d9`) and the LE-staging → LE-prod switch; `infra/PROD-PROVISIONING.md` §10. diff --git a/runbooks/cnpg-archiving.md b/runbooks/cnpg-archiving.md new file mode 100644 index 0000000..4505c6d --- /dev/null +++ b/runbooks/cnpg-archiving.md @@ -0,0 +1,198 @@ +# cnpg-archiving — WAL archiving is stuck, or a Postgres cluster has no recoverability point + +**Alerts:** `CNPGWALArchiveBacklog` (warning, 1h) and `CNPGNoRecoverabilityPoint` (warning, 6h) +**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg` +**Expressions:** +``` +max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 0 +max by (namespace, job) (cnpg_collector_first_recoverability_point) == 0 +``` + +## What this means here — and why it is an AVAILABILITY incident, not just a backup one + +`pg_wal/archive_status` holds one marker per WAL segment: `.ready` = written but **not yet archived**, +`.done` = archived and safe to recycle. A healthy archiver drains `.ready` within seconds, so a +persistently non-zero ready count means `barman-cloud-wal-archive` is failing. + +**Two consequences, and the second one is the emergency:** + +1. There is no continuous archive, so **PITR is broken** (R-IDP-05 / R-STATE-03). +2. **`archive_mode=on` means Postgres will NOT recycle an unarchived WAL segment.** `pg_wal` grows + without bound until the data PVC fills and the postmaster **PANICs**. `pg_wal` shares the **data** + PVC on every Cluster here — no `walStorage:` is configured on any of them — so there is no separate + volume to absorb it. + +This is measured, not theoretical. Prod ran **from bootstrap to 2026-09-07** with +`endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`: all three Clusters at `ContinuousArchiving=False` with +`barman-cloud-wal-archive: exit status 4` and `firstRecoverabilityPoint: null`, **for 38+ hours, with no +alert of any kind** — found by hand. On `keycloak-db` that was 17 unarchived segments / ~268 MiB on a +5 GiB PVC, growing **~6.8 MiB/h ≈ 29 days to disk-full** (about 579 MiB already burned by 2026-09-07). +The rule recorded verbatim: *"Never leave a placeholder here, and treat `ContinuousArchiving=False` on +this Cluster as an availability incident, not a backup one."* + +`CNPGWALArchiveBacklog` is *"the one that would have caught the 2026-09-06 incident within the hour +instead of 38 hours later."* `> 0` with `for: 1h` — not a count threshold — because `for` requires the +condition at **every** evaluation for the full hour, so a write burst resets the timer while a genuinely +stuck archiver never returns to zero. A `> 5` threshold would have delayed `semaphore-db` indefinitely: +its backlog sits at 3 because it writes almost nothing. + +## The two alerts are NOT the same statement + +`CNPGWALArchiveBacklog` is about the **mechanism** and is per-**pod** (each instance has its own `pg_wal` +on its own PVC; a `> 0` shape is safe per-pod, and `pod` names the instance to go look at). + +`CNPGNoRecoverabilityPoint` is about **DR coverage** and is per-**CLUSTER**, grouped +`by (namespace, job)`. That grouping is a fix, dated 2026-09-12: only the **primary** reads the backup +catalogue, so a replica reports a hard `0` forever. Grouped by `pod`, this `== 0` fired **permanently on +all six replicas** while all three Clusters were healthy. Verified live: +old `max by (namespace, pod) (...) == 0` → 6 series (`keycloak-db-1/-3`, `nextcloud-db-1/-3`, +`semaphore-db-1/-3`); new `max by (namespace, job) (...) == 0` → **empty** — and not vacuously so: +forcing the metric to 0 for one primary yields exactly one series. `job` is the grouping label because +the metric carries **no `cnpg.io/cluster` label**; the operator creates one PodMonitor per Cluster, so +prometheus-operator derives `job="/"`: `keycloak/keycloak-db`, +`nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`. `namespace` alone happens to work today but +would let two Clusters in one namespace mask each other. + +**Practical note:** if you see this alert firing on *pods* rather than per cluster, the per-cluster +aggregation is not deployed on this cluster yet — check the deployed PrometheusRule before treating it +as a real loss of DR. + +**Severity is `warning` on purpose.** A database heading for disk-full deserves weeks of lead time, and +this repo has been burned by a permanently-firing `critical` (the `VPNTunnelDown` post-mortem). The +`critical` tier for this failure mode is `PersistentVolumeAlmostFull`, which fires only when the outage +is genuinely imminent. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +The three Clusters' archiving and recoverability status — the single most informative command: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint' +``` + +The archiver's actual error text — `exit status 4` and its cause live here: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -i archive | tail -30' +``` + +The backlog on disk, per instance: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c \ + "ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | grep -c ready; du -sh /var/lib/postgresql/data/pgdata/pg_wal; df -h /var/lib/postgresql/data"' +``` + +The backups that exist — **fully-qualified, or you will be told there are none**: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A; echo; \ + kubectl --context k3s-prod get scheduledbackups.postgresql.cnpg.io -A' +``` + +`kubectl get backup -A` prints "No resources found" on this cluster even with 27 CNPG backups present: +three CRDs register the kind `Backup` (`longhorn.io`, `postgresql.cnpg.io`, `velero.io`) and the short +name resolves to Longhorn's, which holds zero. Verified on prod 2026-09-13. **Never decide a recovery +action from the short name.** + +The object-store configuration, which is where the fault usually is: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak keycloak-db -o yaml | sed -n "/backup:/,/^ [a-z]/p"' +``` + +``` +grep -nE "CNPG_BACKUP_PATH|S3_BACKUP_ENDPOINT|S3_BACKUP_REGION|BACKUP_CREDENTIALS_SUFFIX" clusters/base/cluster-vars.yaml clusters/prod/kustomization.yaml +``` + +## Likely causes, most likely first + +### 1. A wrong or placeholder object-store endpoint / path / credential +The recorded cause of the 38-hour incident: prod ran with `endpointURL: +TBD-PROD-S3-BACKUP-ENDPOINT`. The placeholder was **deliberately loud** because *"base defaults are the +DEV values, so an OMITTED op means prod silently inherits a dev address — far worse than a failed +apply"*. Live prod values to expect: `CNPG_BACKUP_PATH = s3://k3s-prod-backup/barman`, +`S3_BACKUP_ENDPOINT = https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`, +`S3_BACKUP_REGION = auto`, `BACKUP_CREDENTIALS_SUFFIX = -prod`, retention `30d`, WAL compression gzip, +`maxParallel: 2`. Remediation: fix it in git and push; Flux applies and the archiver drains on its own. + +### 2. R2-specific S3 incompatibilities +Cloudflare R2 rejects things MinIO tolerates. The two that are mandatory elsewhere in this platform and +worth checking here: `region` must be exactly `auto`, and AWS-SDK checksum headers are rejected with +**HTTP 400 InvalidArgument**. See `/runbooks/backup-failure` for the full R2 story, including the Velero +plugin's `x-amz-tagging` `NotImplemented` failure. + +### 3. Credentials expired or mis-scoped +A SOPS-encrypted Secret selected by `BACKUP_CREDENTIALS_SUFFIX`. Standing rule from the OIDC incident: +**non-secret, environment-specific values must NOT live inside an encrypted Secret** — `${BASE_DOMAIN}` +substitution can never reach inside one. Keep endpoints and paths in cluster-vars, tokens in the Secret. + +### 4. Egress is blocked +Default-deny is the baseline. The database namespaces need outbound HTTPS to R2. Cross-read +`/runbooks/denied-flows`. + +### 5. A fresh Cluster that has not made its first base backup +`firstRecoverabilityPoint` reads null until the first base backup lands; the `for: 6h` exists exactly to +keep a legitimate cold start quiet. Base backups are staggered: keycloak 01:00, nextcloud 02:00, +semaphore 04:00 (Velero at 03:00). If a Cluster is hours old, wait. + +### 6. Scaling out made it worse +Noted honestly: scaling a Cluster while `archive_mode=on` with a failing `archive_command` +**multiplies the disk-full clock**. Do not add instances as a remedy here. + +## When to escalate or stop + +- **Treat `ContinuousArchiving=False` as an availability incident with a countdown.** Compute the runway: + free bytes on the data PVC ÷ WAL growth rate. At ~6.8 MiB/h on a 5 GiB PVC that is weeks; on a busier + database it is days. +- **NEVER delete WAL segments by hand.** It destroys the database. This is the one irreversible mistake + available in this runbook. +- **Do not delete the `.ready` marker files.** Same outcome. +- **Do not expand the PVC and stop there.** It buys time and hides the fault. +- **Do not scale the Cluster** while archiving is failing. +- **Verify before any restore** that `firstRecoverabilityPoint` is non-null and + `backups.postgresql.cnpg.io` shows completed backups. Prod's healthy state on 2026-09-13 was: + all three Clusters `ContinuousArchiving=True`, `LastBackupSucceeded=True`, `firstRecoverabilityPoint` + 2026-09-07, 27 backups (9 per cluster) completing daily, with one historical failure + (`semaphore-db-daily-20260905130918`). That is what "good" looks like. +- Escalate to the operator for credential or endpoint changes, and for any restore. + +--- +Source of truth: `runbooks/cnpg-archiving.md`. Index: `/runbooks/` +Records: `platform/cnpg/alerts.yaml` header (the 2026-09-06 incident, the measured fill rate, and the +2026-09-12 aggregation-scope fix with its live promtool proof); +`platform/keycloak/postgres-cluster.yaml`; commit `32d6766` (the R2 decision). diff --git a/runbooks/cnpg-backup-missed.md b/runbooks/cnpg-backup-missed.md new file mode 100644 index 0000000..c274167 --- /dev/null +++ b/runbooks/cnpg-backup-missed.md @@ -0,0 +1,163 @@ +# cnpg-backup-missed — No recent CNPG base backup for a Postgres cluster + +**Alert:** `CNPGNoRecentBaseBackup` (warning, 2h) +**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg` +**Expression:** +``` +(time() - max by (namespace, job) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600) +and max by (namespace, job) (cnpg_collector_last_available_backup_timestamp) > 0 +``` + +## What this means here + +The newest completed **base backup** for one Postgres cluster is over 36 hours old — one missed daily run +plus slack. The daily `ScheduledBackup` has silently stopped producing backups, so the PITR window is +ageing out against the `30d` retention policy. + +**Nothing else covers this.** The Velero rules in `platform/monitoring/alerts.yaml` watch Velero's own +schedules, and **Velero does not back up these databases** — CNPG owns their DR path, because a Velero +PVC snapshot of a live Postgres is not a backup. So `/runbooks/backup-missed` being clear says nothing +about the databases, and vice versa. + +The three Clusters and their staggered base-backup times: **keycloak 01:00**, **nextcloud 02:00**, +**semaphore 04:00** (Velero's own daily runs at 03:00, deliberately between them). + +## Two expression details that matter when you read the alert + +**The `> 0` guard is load-bearing.** A cluster that has *never* had a backup reports `0`, and +`time() - 0` is about **57 years** — always true. Without the guard this rule would double-notify with +`CNPGNoRecoverabilityPoint`, which is the correct alert for "never had a backup". So this rule means +specifically: *this cluster has had at least one backup, and the newest one is stale.* + +**It is aggregated `by (namespace, job)`, not `by (namespace, pod)`** — fixed 2026-09-12 alongside +`CNPGNoRecoverabilityPoint`. `cnpg_collector_last_available_backup_timestamp` has the same primary-only +defect: measured on prod the same day, `keycloak-db-2` = 3.1h old, `nextcloud-db-2` = 2.1h, +`semaphore-db-2` = 0.1h, and **all six replicas = 0**, i.e. `time() - 0` ≈ **56.7 years**. Unlike its +sibling this rule was *not* firing — because the `> 0` guard was the single thing standing between it and +six more permanent false positives, which is far too thin a margin for a DR alarm. Aggregating `pod` away +removes the dependence: `max` picks the primary's timestamp as the cluster's answer (a demoted primary +drops to 0 and can never win the `max`; a fresher backup always beats a staler one). +`job` is the grouping label because the metric carries **no `cnpg.io/cluster` label** — the operator +creates one PodMonitor per Cluster, so prometheus-operator derives `job="/"`: +`keycloak/keycloak-db`, `nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`. + +**Severity is `warning` on purpose** — this repo has been burned by a permanently-firing `critical` +(the `VPNTunnelDown` post-mortem), and *"noise on a DR alarm is worse than no alarm at all"*. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +**Fully-qualified names, always.** `kubectl get backup -A` prints **"No resources found"** on this +cluster even with **27 CNPG backups** present: three CRDs register the kind `Backup` +(`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1`, `velero.io/v1`) and the bare short name resolves to +**Longhorn's**, which holds zero objects. Verified on prod 2026-09-13. Concluding "the backups are gone" +from that and acting on it is how a CLI quirk becomes data loss. + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CLUSTER:.spec.cluster.name,PHASE:.status.phase,START:.status.startedAt,STOP:.status.stoppedAt --sort-by=.metadata.creationTimestamp | tail -20' +``` + +Are the schedules there and enabled? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get scheduledbackups.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SCHEDULE:.spec.schedule,SUSPEND:.spec.suspend,CLUSTER:.spec.cluster.name,LAST:.status.lastScheduleTime' +``` + +The Cluster's own verdict (and the condition that tells you whether *archiving* is also broken): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIMARY:.status.currentPrimary,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint' +``` + +Why the last attempt failed — the barman error is in the primary's logs: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -iE "barman|backup|archive" | tail -30; echo ====; \ + kubectl --context k3s-prod -n cnpg-system logs deploy/cnpg-controller-manager --tail=120 2>&1 | grep -iE "backup|error" | tail -20' +``` + +## Likely causes, most likely first + +### 1. The ScheduledBackup is suspended, missing, or unapplied +Check `spec.suspend`, and whether the object exists at all for that Cluster. A `ScheduledBackup` that was +never applied is the "looks done, is absent" failure class this repo keeps finding; verify the Flux tier +that owns it is Ready (`/runbooks/flux-reconcile`) and not suspended (`/runbooks/flux-suspended`). + +### 2. The object store is rejecting the write +Same root cause family as `/runbooks/cnpg-archiving`: endpoint, path, region or credentials. Live prod +values to expect are `CNPG_BACKUP_PATH = s3://k3s-prod-backup/barman`, +`S3_BACKUP_ENDPOINT = https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`, +`S3_BACKUP_REGION = auto`, `BACKUP_CREDENTIALS_SUFFIX = -prod`. **If `ContinuousArchiving` is also +False, fix that first** — a cluster whose archiver is broken has a disk-full countdown running, which is +more urgent than a stale base backup. + +### 3. A failed run that nobody noticed +There is precedent: one historical failure, `semaphore-db-daily-20260905130918`. A single failed run does +not clear itself; the next night's run should. If two consecutive nights fail, the cause is +configuration, not luck. + +### 4. The primary changed and the new primary cannot write +The backup catalogue is the **primary's** job. After a switchover, a credential or path problem scoped to +one instance shows up here. Check which pod is `currentPrimary` and read that pod's logs, not another's. + +### 5. Node or storage trouble during the backup window +Base backups are I/O-heavy and run at 01:00-04:00 alongside Velero. Cross-read `/runbooks/node-io`. + +### 6. The metric is missing, not the backup +If `cnpg_collector_last_available_backup_timestamp` has no series, the rule cannot fire. Note that a +**related** blind spot is real and separate: the CNPG SQL exporter can fail while `up` stays 1 — see +`/runbooks/cnpg-exporter`. + +## When to escalate or stop + +- **A stale base backup narrows, but does not eliminate, recoverability** — provided WAL archiving is + healthy, PITR still works from the older base. That is why this is a warning. If **both** this and + `CNPGNoRecoverabilityPoint` / `CNPGWALArchiveBacklog` are lit, DR for that database is effectively + gone: escalate. +- **Never delete WAL, `.ready` markers, or anything under a Postgres data directory.** +- **Do not delete old CNPG Backup objects to tidy up.** Retention is `30d` and the operator manages it; + deleting a Backup object can orphan objects in R2. +- **Do not trigger a base backup as a reflex** on a cluster whose archiver is failing — it adds write + load to a volume that may already be filling. +- **Be honest about restore:** the CNPG restore path from R2 has not been rehearsed on this cluster + (`dr-restore` is authored and never executed). Free R2 egress makes a rehearsal cheap; it is + outstanding. A restore is an operator operation. + +--- +Source of truth: `runbooks/cnpg-backup-missed.md`. Index: `/runbooks/` +Records: `platform/cnpg/alerts.yaml` (the 2026-09-12 aggregation-scope fix with its live measurements, +the `> 0` guard rationale, and the staggered schedule). diff --git a/runbooks/cnpg-exporter.md b/runbooks/cnpg-exporter.md new file mode 100644 index 0000000..7714561 --- /dev/null +++ b/runbooks/cnpg-exporter.md @@ -0,0 +1,168 @@ +# cnpg-exporter — The CNPG metrics collector is erroring (a database is invisible) + +**Alert:** `CNPGMetricsCollectorErroring` (warning, 6h) +**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg` +**Expression:** `max by (namespace, pod) (cnpg_last_error) == 1` + +## What this means here + +**This is the monitor's own blind spot, for databases — a failure that looks exactly like health.** + +CloudNativePG publishes **two independent metric families** and only one of them depends on SQL: + +- `cnpg_collector_*` comes from the instance manager and is **always** present. +- `cnpg_pg_stat_*` (archiver, database, replication) comes from the exporter logging in to the + application database as the `cnpg_metrics_exporter` role. + +When that login fails, `cnpg_last_error` goes to 1 and **the entire `cnpg_pg_stat_*` family blanks for +that instance — while `up` stays 1 and `cnpg_collector_up` stays 1.** Nothing looks broken. The scrape +target is green. The dashboards have gaps that read as "no data" rather than "broken". + +**This is not hypothetical, and it shaped the whole alert file.** Measured live on 2026-09-06 on both +clusters, `cnpg_errors_total` carried: + +``` +pg_stat_archiver on db nextcloud: failed to connect to `user=cnpg_metrics_exporter database=nextcloud`: +FATAL: permission denied for database "nextcloud" (SQLSTATE 42501) +``` + +`nextcloud-db` had been publishing **no** `pg_stat_archiver` / `pg_stat_database` metrics at all. The +consequence recorded verbatim: *an archiving alert built on `pg_stat_archiver` — the obvious choice — +would have been silently blind to one of three prod databases.* That is why every DR rule in +`platform/cnpg/alerts.yaml` keys on `cnpg_collector_*` instead, and why this rule exists to alert on the +blind spot itself. + +**Scope: `pod` is kept here, deliberately and unchanged.** `cnpg_last_error` is genuinely per-instance — +each instance's own exporter reports its own SQL-collection health — and prod proved the metric +discriminates rather than tracking role: all **three** `nextcloud-db` pods (including primary `-2`) +reported 1 while all six `keycloak`/`semaphore` pods reported 0. There is no primary-only defect to fix, +and per-pod is the right granularity because the remedy is a **GRANT applied per database** and you want +to see it clear on every instance. + +`for: 6h` keeps a pod restart or a brief connection blip quiet. Severity stays `warning` so it never +becomes the permanently-firing `critical` this repo has been burned by. + +**Live state, measured 2026-09-13:** `max by (namespace,pod) (cnpg_last_error) == 1` returned **empty** +across the 9 instances that publish `cnpg_last_error` — so the `nextcloud` grant appears to have been +addressed. Re-measure rather than assuming either way. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which instances are erroring: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s -G --data-urlencode "query=max by (namespace,pod) (cnpg_last_error)" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "pod|\"1\""' +``` + +**The message itself** — `cnpg_errors_total` carries it in an `errorUserQueries` label. This is the one +command that tells you the cause outright: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s -G --data-urlencode "query=cnpg_errors_total" localhost:19090/api/v1/query | python3 -m json.tool | head -60' +``` + +Or straight from the instance's metrics endpoint: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n nextcloud-db exec nextcloud-db-2 -c postgres -- \ + curl -s localhost:9187/metrics | grep -E "^cnpg_last_error|^cnpg_errors_total|^cnpg_collector_up" | head -20' +``` + +Confirm which family is actually missing — this is what makes the blind spot visible: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + for q in "count(cnpg_collector_up)" "count(cnpg_last_error)" "count(cnpg_pg_stat_archiver_archived_count)" "count(cnpg_pg_stat_database_xact_commit)"; do \ + printf "%-50s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \ + | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done' +``` + +The exporter role's actual grants: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n nextcloud-db exec nextcloud-db-2 -c postgres -- \ + psql -tAc "select rolname, rolcanlogin from pg_roles where rolname = \"cnpg_metrics_exporter\";" \ + -c "select datname, has_database_privilege(\"cnpg_metrics_exporter\", datname, \"CONNECT\") from pg_database where datistemplate = false;"' +``` + +## Likely causes, most likely first + +### 1. The exporter role lacks CONNECT on the application database +The recorded cause. Signature: `SQLSTATE 42501`, `permission denied for database ""` in +`cnpg_errors_total`. Remedy: **grant the `cnpg_metrics_exporter` role CONNECT on the application +database.** Do it declaratively — CNPG can manage roles and the `managed.roles` / bootstrap surface in the +Cluster spec is where this belongs, so the grant survives a re-seed. A `psql` GRANT by hand fixes the +symptom on one instance and vanishes on the next bootstrap. + +### 2. Custom monitoring queries reference something that does not exist +CNPG lets a Cluster attach extra collector queries via a ConfigMap. A query against a missing table, +extension or column fails the whole collection pass for that instance. `cnpg_errors_total`'s label names +the failing query. + +### 3. The database is not accepting connections +A read-only/recovery state, `max_connections` exhausted, or the instance mid-restart. Cross-read +`/runbooks/keycloak-db` if the instance is `keycloak-db-*`. Note the deliberate setting +`dataDurability: preferred` (never `required`) exists so a degraded standby cannot block writes here. + +### 4. An upgrade changed the exporter's expectations +CNPG **1.30.0** is the verified operator version on both clusters. A major operator or Postgres bump can +change the collector's queries; if this started at an upgrade, read `/runbooks/update-rollback`. + +### 5. The instance is fine and the whole scrape is gone +Then `up` for that `job` would be 0 and this is the wrong runbook — `/runbooks/targets-down`. The precise +value of *this* alert is the case where `up` is **1** and the data is still missing. + +## When to escalate or stop + +- **Understand what you are currently blind to.** While this fires, `cnpg_pg_stat_archiver`, + `cnpg_pg_stat_database` and `cnpg_pg_stat_replication` are missing for that instance. The DR alerts in + `platform/cnpg/alerts.yaml` are deliberately built on `cnpg_collector_*` so they still work — **but do + not build any new alert or dashboard on `pg_stat_*` without checking this first.** That is the trap + this alert exists to mark. +- **Do not silence it.** It is a warning with a 6h `for` precisely so it can be left visible without + becoming noise, and it was a **true positive** for months. +- **Do not grant the exporter role broad privileges** to make it go away. CONNECT on the application + database is the whole requirement; anything more widens the blast radius of a metrics credential. +- **Do not restart the database** to clear it — the error is a grant, not a state. +- Escalate to the operator if the fix requires changing the Cluster's managed roles or bootstrap, since + that touches the database's declarative definition and its DR path. + +--- +Source of truth: `runbooks/cnpg-exporter.md`. Index: `/runbooks/` +Records: `platform/cnpg/alerts.yaml` (the 2026-09-06 `SQLSTATE 42501` measurement on `nextcloud-db`, the +two-metric-families finding, and the 2026-09-12 scope audit that deliberately kept `pod`). diff --git a/runbooks/cnpg-primary-spread.md b/runbooks/cnpg-primary-spread.md new file mode 100644 index 0000000..7287b68 --- /dev/null +++ b/runbooks/cnpg-primary-spread.md @@ -0,0 +1,187 @@ +# cnpg-primary-spread — Two or more CNPG primaries share one node + +**Alert:** `CNPGPrimariesCoLocated` (warning, 30m) +**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg` +**Expression:** `count by (node) () > 1` +**and** gated on every node being Ready: +`count(kube_node_status_condition{condition="Ready",status="true"} == 1) == count(kube_node_info)` + +## What this means here + +More than one CloudNativePG **primary** has been running on the same node for 30 minutes, while all +three nodes are Ready. There are exactly three CNPG Clusters on this platform — `keycloak/keycloak-db`, +`nextcloud-db/nextcloud-db` and `semaphore-db/semaphore-db` — so on a 3-node cluster the healthy shape is +one primary per node, and this alert says that shape has drifted. + +**This is BLAST RADIUS, not an outage.** Each Cluster has `instances: 3` with one pod per node +(`podAntiAffinityType: required`) and one synchronous standby (`method: any`, `number: 1`, +`dataDurability: preferred`), so every one of them survives losing the shared node — you just get two or +three simultaneous PostgreSQL failovers instead of one. Nothing is broken right now, which is why it is a +`warning` with a long `for`. + +**Why it matters anyway: this exact drift has already caused an outage.** All three Clusters used to run +`instances: 1` and all three primaries had drifted onto one node, and **on 2026-09-07 that node took SSO, +Nextcloud and Semaphore down together.** `instances: 3` means the same drift no longer causes an outage — +but if `keycloak-db`'s primary is co-located and that node dies, the SSO failover happens at the same +moment as two others, on nodes that are also absorbing Longhorn replica rebuilds. + +**Why the drift happens and cannot be declared away: CNPG has no declarative primary-placement field.** +The primary lands wherever the last promotion put it — a node reboot, a switchover, an operator upgrade. +So this is avoidable drift that must be *corrected*, not prevented, and the alert re-arms itself so it +will be reported again rather than silently persisting. + +**Why the node-readiness gate is in the expression.** During a real node outage, primaries legitimately +pile onto the survivors and there is nothing to do about it; alerting then would be noise at the worst +moment. The `and on ()` clause suppresses the rule unless every node is Ready, and re-arms when the node +returns. + +This rule was proven in both directions on live prod data (2026-09-13): the expression returned +`{node="k3s-prod-1"} => 3` before the rebalance and a **throwaway twin with `for: 1m`** was observed going +`pending` at +70s and `firing` at +130s; after two switchovers the same expression returned an **empty +vector while the inner join still resolved all three primaries** — quiet because the condition is false, +not because the query went blind. It is also unit-tested in `alerts.promtool-test.yaml` +(run `./run-alert-tests.sh`) with four cases, mutation-checked so that deleting the `cnpg_collector_up` +join, the readiness gate, or shortening `for:` makes a test FAIL. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Where the primaries are — the one command this alert is about: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get pods -A -l cnpg.io/instanceRole=primary \ + -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName' +``` + +Are all three Clusters actually healthy enough to move one? (Fully-qualified: +`clusters.postgresql.cnpg.io` — `cluster` is an ambiguous short name.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase' +``` + +Is every node Ready (the gate the rule itself applies)? + +``` +infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes -o wide' +``` + +Is there storage or I/O trouble that makes a switchover a bad idea right now? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ + -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy' +``` + +``` +ssh k3s-prod-1 'cat /proc/pressure/io' +``` + +And confirm DR is intact before you deliberately move a primary — the primary owns the backup +catalogue, and **fully-qualified names matter**: `kubectl get backup -A` prints "No resources found" on +this cluster even with 27 CNPG backups present, because three CRDs register the kind `Backup` and the +short name resolves to Longhorn's (zero objects): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A | tail -10; echo; \ + kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint' +``` + +## Likely causes, most likely first + +### 1. A node rebooted and the primaries never moved back +The dominant cause. Promotions during the reboot put two primaries on a survivor, and CNPG has no reason +to move them back afterwards. Remediation: a deliberate switchover, below. + +### 2. A rolling operator or Postgres upgrade +`primaryUpdateMethod: switchover` is set deliberately (the CRD default `restart` is a +delete-and-recreate of the *primary*, i.e. tens of seconds with no Postgres — an SSO outage on +`keycloak-db`). Switchovers during an upgrade are correct behaviour and leave the primaries wherever +they landed. + +### 3. One node was cordoned or unschedulable for a while +Same mechanism. Note that `podAntiAffinityType: required` means an instance stays **Pending** until a +node returns rather than co-locating — that is the accepted cost of the 2026-09-07 fix, and it is why +the *instances* stay spread even when the *primaries* do not. + +### 4. A Longhorn or I/O event caused a health-check failover +A primary whose storage stalled gets demoted. Cross-read `/runbooks/node-io` and +`/runbooks/longhorn-degraded` — if that is why, fix the node before rebalancing. + +### 5. The alert is firing during a genuine node outage +It should not be able to: the readiness gate suppresses it. If it fires anyway with a node NotReady, the +gate is broken — that is a monitoring bug worth fixing, since a rule that cannot stop is fatigue. + +## Remediation — move one primary, deliberately + +The cnpg plugin is **NOT in the devcontainer image**; fetch the `kubectl-cnpg` release matching the +operator version (CNPG **1.30.0** here) before you start. + +``` +kubectl cnpg promote -n +``` + +**Do ONE CLUSTER AT A TIME and wait for `readyInstances: 3` before the next.** A switchover interrupts +writes to that database for about **9 seconds** — measured on both clusters moved on 2026-09-13. Nine +seconds of `keycloak-db` unavailability is a brief SSO interruption; doing two at once is an avoidable +compound one. + +Pick the target by reading the primaries table above and choosing an instance that already sits on a node +holding no other primary. Then re-run the table to confirm the new distribution, and let the alert clear +on its own. + +## When to escalate or stop + +- **Do not rebalance during any other incident.** A switchover is a small, deliberate write interruption, + and it is the wrong thing to add to a node that is already under I/O pressure or to a cluster with + degraded Longhorn volumes. This alert has a 30-minute `for` precisely because it is never urgent. +- **Never switch over a Cluster whose `readyInstances` is below 3.** You would be promoting into a + cluster that has no standby left to fall back on. +- **Never change `dataDurability` to `required`** while thinking about durability here: it is a strict + quorum, and one degraded standby would then **block writes** entirely on the SSO database. +- **Do not `kubectl delete pod` the primary to force a failover.** That is `restart` semantics — the + thing `primaryUpdateMethod: switchover` exists to avoid. +- **Check `ContinuousArchiving` and `firstRecoverabilityPoint` first** (`/runbooks/cnpg-archiving`). If + archiving is broken, that is a real countdown to a disk-full PANIC and it outranks blast-radius + tidying completely. +- Escalate to the operator for the switchover itself if the affected Cluster is `keycloak-db`: the brief + write interruption is user-visible across SSO, kubectl OIDC, Grafana, Headlamp and Gitea. + +--- +Source of truth: `runbooks/cnpg-primary-spread.md`. Index: `/runbooks/` +Records: `platform/cnpg/alerts.yaml` (the rule's own both-directions proof on live prod data and its +promtool tests); commits `f338e91` (instances 3), `e8d8601` (required anti-affinity + switchover), +`cac413e` (synchronous replication, `dataDurability: preferred`); the 2026-09-07 single-node SSO outage. diff --git a/runbooks/cnpg-wal-growth.md b/runbooks/cnpg-wal-growth.md new file mode 100644 index 0000000..4d11b76 --- /dev/null +++ b/runbooks/cnpg-wal-growth.md @@ -0,0 +1,161 @@ +# cnpg-wal-growth — pg_wal is not being recycled + +**Alert:** `CNPGWALVolumeNotRecycling` (warning, 2h) +**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg` +**Expression:** +``` +max by (namespace, pod) (cnpg_collector_pg_wal{value="count"}) + > max by (namespace, pod) (cnpg_collector_pg_wal{value="max"}) +``` + +## What this means here + +`pg_wal` is holding more WAL segments than `max_wal_size` allows for, which means **checkpoints are no +longer reclaiming space**. On CNPG the units are **segments of 16 MiB**: `max` is `max_wal_size` +(**64 = 1 GiB** on this platform) and `count` is what is currently sitting in `pg_wal`. + +This is the **storage escalation** of `/runbooks/cnpg-archiving`, not an independent fault. +`CNPGWALArchiveBacklog` is the day-one signal — it fires within an hour of the archiver getting stuck. +This one fires when the resulting backlog has grown large enough to be a *disk* problem. With +`archive_mode=on` a failing `archive_command` is exactly what this looks like after a while, and because +**`pg_wal` shares the data PVC on every Cluster here** (no `walStorage:` is configured on any of them) +it ends in the volume filling and the postmaster **PANICking**. + +It does **not** fire today and is not meant to: measured, prod `keycloak-db` holds 17 segments against +max 64 (dev held 36 against 64), and across all nine prod instances counts ran 16-37 against 64. At +prod's observed ~6.8 MiB/h, `keycloak-db` would reach 64 segments roughly **40 days before** its 5 GiB +PVC actually ran out — so this alert leads the PVC-fill alerts by weeks. + +**Why `pod` is kept here, and is required.** Every instance reports its own `count` and `max`, `pg_wal` +is per-instance on a per-instance PVC, and the comparison only means anything pod-against-itself — +aggregating `pod` away would compare one instance's `count` with another's `max`. A `>` shape is also +safe per-pod: no instance can sit permanently true. (Contrast the `== 0` and staleness rules in the same +file, which **must** aggregate `pod` away — that is the 2026-09-12 fix.) + +**One metric that looks right and is not:** the CNPG-native +`cnpg_collector_pg_wal{value="volume_size"}` is **NaN on both clusters**, precisely because `pg_wal` +shares the data PVC rather than having its own. Do not build anything on it. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which instance, and how far past `max_wal_size`: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s -G --data-urlencode "query=max by (namespace,pod) (cnpg_collector_pg_wal{value=\"count\"})" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "pod|\"1\""' +``` + +The same thing from inside the instance, which also shows you the archive backlog and the free space in +one view: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c \ + "du -sh /var/lib/postgresql/data/pgdata/pg_wal; \ + ls /var/lib/postgresql/data/pgdata/pg_wal | grep -c \"^0\"; \ + ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | grep -c ready; \ + df -h /var/lib/postgresql/data; \ + psql -tAc \"show max_wal_size; show archive_mode; select pg_walfile_name(pg_current_wal_lsn());\""' +``` + +Is archiving the cause? (It almost always is.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint; echo; \ + kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -i archive | tail -20' +``` + +Is a replication slot or a stale standby holding WAL instead? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- \ + psql -x -c "select slot_name, active, wal_status, safe_wal_size, restart_lsn from pg_replication_slots;" \ + -c "select application_name, state, sync_state, replay_lag from pg_stat_replication;"' +``` + +Compute the runway — free bytes on the PVC divided by the observed growth rate: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s -G --data-urlencode "query=predict_linear(kubelet_volume_stats_available_bytes{job=\"kubelet\",namespace=\"keycloak\"}[24h], 7*24*3600)" localhost:19090/api/v1/query' +``` + +## Likely causes, most likely first + +### 1. WAL archiving is failing (dominant cause) +Go to `/runbooks/cnpg-archiving` — that is the actual fix. Fix the `archive_command` and Postgres recycles +the backlog on its own, without any intervention in `pg_wal`. Historical precedent: prod ran 38+ hours +from bootstrap with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, all three Clusters at +`ContinuousArchiving=False` with `barman-cloud-wal-archive: exit status 4`. + +### 2. An inactive replication slot is pinning WAL +A standby that has been gone long enough leaves a slot holding `restart_lsn`, and Postgres keeps every +segment after it. `pg_replication_slots.wal_status` will read `extended` or `lost`. The healthy fix is to +bring the standby back (CNPG manages the slots), not to drop the slot by hand. Note +`podAntiAffinityType: required` means a lost instance stays **Pending** until a node returns — expected, +and accepted. + +### 3. Genuinely heavy write load +Then `count` exceeding `max_wal_size` is checkpointing lagging, not a leak, and it self-corrects. Confirm +that `archive_status/*.ready` is near zero: a large `pg_wal` with an empty ready queue is a busy +database, not a stuck archiver. + +### 4. `max_wal_size` was lowered +64 segments = 1 GiB is the configured value on both clusters. A reduction makes this rule fire without +anything actually being wrong. Check the Cluster's postgresql parameters — and note that under +`primaryUpdateMethod: switchover` the operator **rejects an update that changes `imageName` and a +postgresql parameter in the same reconcile**. + +### 5. The instance is a replica that fell behind +Replay lag holds WAL on the primary. `pg_stat_replication.replay_lag` shows it. + +## When to escalate or stop + +- **NEVER delete WAL segments or `archive_status` marker files by hand.** It destroys the database. This + is the single irreversible mistake available here, and the alert's own description says so: *"Fix + archiving — do NOT delete WAL by hand."* +- **Do not drop a replication slot** to free space without the operator: you may make a lagging standby + unrecoverable and force a full re-seed. +- **Do not expand the PVC and stop there.** It buys days and hides the fault. +- **Do not scale the Cluster out** while archiving is failing — that **multiplies** the disk-full clock. +- Escalate when the computed runway is under a week, or when `PersistentVolumeAlmostFull` + (`/runbooks/pvc-capacity`) is also firing for the same PVC — at that point an outage is imminent and a + Postgres data volume under 5% free will PANIC the postmaster. + +--- +Source of truth: `runbooks/cnpg-wal-growth.md`. Index: `/runbooks/` +Records: `platform/cnpg/alerts.yaml` (the measured segment counts, the NaN `volume_size` finding, and the +2026-09-12 scope audit that deliberately KEPT `pod` here). diff --git a/runbooks/denied-flows.md b/runbooks/denied-flows.md new file mode 100644 index 0000000..a44af99 --- /dev/null +++ b/runbooks/denied-flows.md @@ -0,0 +1,154 @@ +# denied-flows — A spike in policy-denied network flows + +**Alert:** `HubbleDeniedFlowSpike` (warning, 10m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.networking` +**Expression:** `sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5` + +## What this means here + +Cilium is dropping more than 5 flows/second on policy. East-west traffic is **default-deny** everywhere +(R-NET-01): a bare workload reaches DNS and nothing else, and every allow is declared per +source → destination → port (R-NET-02). So a denied-flow spike is almost always **a NetworkPolicy +regression** — something legitimate that is no longer allowed — and only rarely a probe. + +### Verify the alert can even fire (measured 2026-09-13) + +**This alert is currently INERT.** `count(hubble_drop_total)` returned **0 series** on prod: there is no +Hubble/Cilium scrape job among Prometheus's 19 jobs, so Hubble metrics are not being collected and this +rule evaluates against nothing. **Policy drops are unmonitored today** — you will hear about them as +"service X cannot reach service Y", and you diagnose them with `hubble observe`, below. See +`/runbooks/targets-down` for the full list of inert rule groups. + +### The regression class that has actually bitten this cluster + +**Cilium's CVE-2025-30162 hairpin change (the expensive one).** After the 1.16 → 1.20 upgrade, the +shared Gateway's L7 proxy re-checks a pod's egress policy against the destination's **real backend +identity**. So `toEntities: world` **no longer covers an in-cluster call to a `*.home.keithjackson.dev` +hostname** — the destination resolves to an in-cluster backend, not to `world`. Every app's SSO call +started returning **403 with `server: envoy`**. The fix is an explicit `toEndpoints` allow to the +Keycloak backend, which `catalog/render.py` now emits for every service with +`auth.mode ∈ {oidc, forward-auth}`. + +**The durable rule that follows: any new app → app-by-hostname dependency needs its own explicit allow.** +If a newly added or newly SSO-enabled service is 403ing or timing out, this is the first thing to check. + +A second recorded instance of the same class: Flux's `source-controller` was denied by +`gitea-allow-gateway-ingress`, whose ingress rule was `fromEntities: [ingress]` only — so an in-cluster +`wget` to Gitea simply timed out. And a matching diagnostic trap: **testing HTTPS from inside a pod is +misleading here**; test from outside with `curl --resolve`. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +See the actual drops — this is the tool, since the metric is not collected: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \ + hubble observe --verdict DROPPED --last 200 -o compact' +``` + +Narrow to a suspected pair: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \ + hubble observe --verdict DROPPED --from-namespace nextcloud --last 100 -o compact' +``` + +What policies apply to the source workload, and is the policy actually enforced? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get ciliumnetworkpolicies.cilium.io,networkpolicies.networking.k8s.io -A; echo; \ + kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg endpoint list | head -30' +``` + +Cilium's own drop counters, as a cross-check: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg metrics list | grep -i drop' +``` + +Is the 403 an Envoy/L7 verdict rather than an L3/L4 drop? (The hairpin class above shows up here, not in +`hubble observe`.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n logs --tail=50 | grep -iE "403|envoy|oidc|well-known"' +``` + +## Likely causes, most likely first + +### 1. A newly added or newly SSO-enabled service has no explicit allow to Keycloak +The CVE-2025-30162 hairpin class described above. Signature: HTTP **403 with `server: envoy`** on a call +to `keycloak.${BASE_DOMAIN}` or another `*.home` hostname from inside the cluster, while the same URL +works from outside. Remediation: the allow is **rendered**, not hand-written — set `auth.mode` in +`catalog/.yaml` and re-render so `render.py` emits the `toEndpoints` allow to the keycloak +backend. Do not patch a CiliumNetworkPolicy by hand; it will be overwritten. + +### 2. A platform component was moved or a namespace label changed +`fromEntities: [ingress]`-only ingress rules (the Gitea/source-controller case) break any in-cluster +client. Remediation: add the specific allow in the component's `networkpolicy.yaml`, in git. + +### 3. A missing dependency declaration in the catalog +`dependencies:` in a ServiceDefinition is what generates app→app allows (R-NET-02). A service talking to +something it never declared is denied by design. The declaration is the fix. + +### 4. A scrape target that nothing allows +Prometheus selects **every** ServiceMonitor in the cluster +(`serviceMonitorSelectorNilUsesHelmValues: false`), so a new workload's monitor is scraped immediately +and will be denied until the component allows ingress from `monitoring`. Cross-read +`/runbooks/targets-down`. + +### 5. An actual probe or scan +Possible, and it is why the alert is worded as a security signal (docs/07). Distinguish it by the +**source**: a spike from outside the cluster or from an unexpected identity, rather than one named +workload repeatedly failing one destination. Escalate rather than tuning policy. + +## When to escalate or stop + +- **Do not widen a policy to make drops stop.** Default-deny is a safety-critical requirement here + (R-NET-01 is flagged `safety_critical: true` in the requirement registry). The correct fix is the + narrowest explicit allow, authored in git, usually via the catalog so it is rendered consistently. +- **Do not hand-edit a rendered CiliumNetworkPolicy** — it comes from `catalog/render.py` and will be + reverted on the next reconcile. +- **Escalate if the source is unexpected.** An unexplained spike from outside the cluster is a security + event, not a configuration bug. +- If drops coincide with a Cilium upgrade, assume the hairpin/identity class above before assuming a + policy authoring mistake — the upgrade changed the *evaluation*, not the policy. +- Note the diagnostic trap when verifying a fix: probe from **outside** with `curl --resolve`, because + in-pod HTTPS tests to the platform's own hostnames are misleading. + +--- +Source of truth: `runbooks/denied-flows.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` (the Cilium CVE-2025-30162 world-hairpin / envoy 403 finding and its +`render.py` fix; source-controller denied by `gitea-allow-gateway-ingress`). diff --git a/runbooks/flux-reconcile.md b/runbooks/flux-reconcile.md new file mode 100644 index 0000000..70d12bb --- /dev/null +++ b/runbooks/flux-reconcile.md @@ -0,0 +1,164 @@ +# flux-reconcile — A Flux resource has failed to reconcile for 15 minutes + +**Alert:** `FluxReconciliationFailure` (warning, 15m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux` +**Expression:** `max by (namespace, name, kind) (gotk_reconcile_condition{type="Ready", status="False"}) == 1` + +## What this means here + +Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than +it sounds, because **`git push origin main` IS the deploy** — there is no other apply step. A stuck +Kustomization means git and the cluster have diverged and nothing will close the gap. + +The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` → +`platform-config-cdi` → `apps-wave-0/1/2`, plus the non-gating leaves `apps-vms`, `oauth2-proxy`, +`headlamp`, `homepage`, `crafty`, `runbooks`, `backup`. Which tier is failing decides the blast radius: + +- **`flux-system`, `cilium`, `platform`, `platform-config` are gating (`wait: true`).** One Failed + HelmRelease in the `platform` tier stalls **everything downstream** on "dependency not ready". That + is not theoretical: a crashlooping Headlamp HelmRelease once stalled `platform-config`, + `oauth2-proxy` and every apps wave (the "Headlamp gating stall"), and oauth2-proxy twice deadlocked + by **blocking its own fix** — it sat in the gating tier while needing Keycloak + trusted TLS + + split-horizon DNS, so `platform-config`, which applies the fix, could never reconcile. Both were + resolved by moving the leaf into its own `wait: false` tier. +- **The leaf tiers are `wait: false`** precisely so their unhealth can never gate anything. + +Prod's source is `http://gitea.gitea.svc.cluster.local:3000` — the **in-cluster Service**, on purpose. +It used to be a public hostname, and the apex flip taught the expensive version of that lesson: the +HTTPRoute serving the source's own hostname lives in a downstream tier that cannot reconcile while the +source is down, so `srcReady=False` / *"unable to list remote"* became a **deadlock**. If you are +tempted to point the GitRepository at a public name, don't. +**This alert is currently INERT.** `count(gotk_reconcile_condition)` and `count(gotk_suspend_status)` +both returned **0 series** on prod on 2026-09-13: there is no scrape job for the Flux controllers in +Prometheus's 19 jobs, so every `gotk_*`-based rule (`FluxReconciliationFailure`, +`FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, +`UpdateHealthGateFailed`) evaluates against nothing. **Flux health is therefore unmonitored today** and +must be read from the CRs directly, with the commands below. See `/runbooks/targets-down` for the full +list of inert rule groups. Adding the Flux controllers' ServiceMonitor is open work. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +The whole chain at a glance — this is the command to run first, every time: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \ + -o custom-columns=NAME:.metadata.name,PATH:.spec.path,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,REV:.status.lastAppliedRevision' +``` + +Is the SOURCE healthy? If not, nothing downstream can be: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n flux-system get gitrepositories.source.toolkit.fluxcd.io \ + -o custom-columns=NAME:.metadata.name,URL:.spec.url,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message' +``` + +HelmReleases (the usual culprit inside `platform`): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get helmreleases.helm.toolkit.fluxcd.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message' +``` + +The controller's own view, with the real error text: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n flux-system logs deploy/kustomize-controller --tail=80; echo ====; \ + kubectl --context k3s-prod -n flux-system logs deploy/helm-controller --tail=80' +``` + +## Likely causes, most likely first + +### 1. "no matches for kind X" — a CR applied before its CRD +The recurring structural failure here. Flux server-side-apply fails the **whole** apply, so a tier that +ships both a CRD-installing HelmRelease and CRs of that CRD can never converge — the CRD never gets +created because the apply that would create it also contains the CR. Symptoms: +`no matches for kind "ClusterIssuer"` / `"PrometheusRule"`. The standing rule: **any component shipping +CRs of a CRD it installs must split the CRs into `platform-config`** (`dependsOn: platform`), and +second-order CRs whose CRD only appears after a controller reconciles go in `platform-config-cdi`. +Remediation is a git change moving the resource; `bootstrap/standup.sh crds` is the belt-and-braces +repair for a cluster already wedged. + +### 2. A HelmRelease is genuinely failing its install/upgrade +Read the message. Known instances: the `headlamp-0.40.1` `-session-ttl` chart/image desync; Grafana 13 +OOM-looping after kps 65→88. Note `platform`'s `timeout: 5m` versus Longhorn's HelmRelease +`timeout: 15m` — **the tier can time out while a legitimate Longhorn upgrade is still running**. Give +it another interval before concluding it is broken. + +### 3. A `${TOKEN}` problem in post-build substitution +Substitution **fails open**: an undefined variable becomes the empty string and the Kustomization still +reports Ready — which is how `k8sServiceHost: ""` broke Cilium on 2 of 3 dev nodes on 2026-09-01. A +*parse* error is the loud version and does fail the whole Kustomization +(`unable to parse variable name`). If the message mentions envsubst or a variable, run the L0 check: +`conformance/tests/l0-static/flux-postbuild-substitution/check.sh`. Note **strict substitution became +default-on in kustomize-controller v1.9.0**, so a Flux upgrade flips this behaviour whether or not you +opt in. + +### 4. A remote kustomize base timed out +The kustomize-controller's git fetch has a 27s timeout and remote GitHub bases flake against it. +`external-snapshotter` was vendored into the repo for this reason; **`platform/kubevirt` still uses +remote bases**, so a `BuildFail` there is probably this. Remediation: vendor it the same way. + +### 5. SOPS decryption +A tier that carries an encrypted Secret needs its own `decryption` stanza patched in by the overlay, by +name. A newly split-out tier that inherited a secret but not the patch fails here. + +## Remediation notes + +- Everything durable is a **git change**. `flux reconcile` only re-triggers; it does not fix a bad + manifest, and it is blocked for agents here. +- Flux is **level-triggered**: an interrupted reconcile is retried, so a restart of a controller is + safe and often the fastest way past a wedged in-flight apply (operator action). +- `dependsOn` is **revision-gated**, not merely Ready-gated: a dependent cannot reconcile revision N + until its dependency has applied N. That is what makes the chain safe and also what makes one stuck + tier so expensive. + +## When to escalate or stop + +- **If `flux-system`, `cilium`, `platform` or `platform-config` is the failing tier, treat it as a + platform outage in progress** — everything downstream is frozen, including the fix. Get the operator. +- **Do not `flux suspend` a tier to make the alert stop.** That trades a visible failure for + `/runbooks/flux-suspended` and silent drift, which is how prod's whole `platform` tier ended up + suspended and left that way after the 2026-09-04 external-dns incident. +- **Do not enable `driftDetection.mode: enabled` on a CNI.** During the 2026-09-01 Cilium incident the + Helm storage manifest *was* the bad render, so drift correction would have actively re-asserted the + broken value. +- If the GitRepository itself is not Ready, check Gitea before Flux — and remember prod deliberately + reads from the in-cluster Service so that public DNS can never be in this path. + +--- +Source of truth: `runbooks/flux-reconcile.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` "2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium" (substitution +fails open), the Headlamp gating stall, the oauth2-proxy cold-start deadlock, and the 2026-09-13 +GitOps source deadlock. diff --git a/runbooks/flux-suspended.md b/runbooks/flux-suspended.md new file mode 100644 index 0000000..3a766bc --- /dev/null +++ b/runbooks/flux-suspended.md @@ -0,0 +1,134 @@ +# flux-suspended — A Flux resource has been suspended for an hour + +**Alert:** `FluxResourceSuspended` (warning, 1h) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux` +**Expression:** `max by (namespace, name, kind) (gotk_suspend_status) == 1` + +## What this means here + +Someone turned reconciliation off for a resource and it is still off an hour later. While suspended, +Flux neither applies new commits nor corrects drift — **git stops being the truth** for that subtree, +silently, and everything downstream of a suspended gating tier is frozen with it. + +Suspension is a legitimate emergency valve here and is used deliberately: `cilium` was split into its +own Kustomization partly so that `flux suspend kustomization cilium` becomes an option that does not +freeze the other eleven platform components. The problem is never the suspend; it is the **forgotten** +suspend. Two real cases: + +- After the 2026-09-04 external-dns incident, prod's whole `platform` tier and everything downstream + were suspended **and left suspended** (`infra/PROD-PROVISIONING.md` §10). That suspension is also + the only reason a second bug never fired: `platform-config` would otherwise have requested a + `*.home.keithjackson.dev` wildcard from letsencrypt-prod for a certificate dev already held, and the + two clusters would have raced each other's `_acme-challenge` TXT records on one zone — neither would + reliably solve DNS-01. +- **Velero was found suspended live, with the suspension never recorded in git**, and resumed with + `flux resume`. A backup controller that is not reconciling is the definition of a silent DR gap. + +The 1h `for` is the point: short suspensions during maintenance are expected and should not notify. +**This alert is currently INERT.** `count(gotk_reconcile_condition)` and `count(gotk_suspend_status)` +both returned **0 series** on prod on 2026-09-13: there is no scrape job for the Flux controllers in +Prometheus's 19 jobs, so every `gotk_*`-based rule (`FluxReconciliationFailure`, +`FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, +`UpdateHealthGateFailed`) evaluates against nothing. **Flux health is therefore unmonitored today** and +must be read from the CRs directly, with the commands below. See `/runbooks/targets-down` for the full +list of inert rule groups. Adding the Flux controllers' ServiceMonitor is open work. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +What is suspended, across every Flux kind: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'for k in kustomizations.kustomize.toolkit.fluxcd.io helmreleases.helm.toolkit.fluxcd.io \ + gitrepositories.source.toolkit.fluxcd.io helmrepositories.source.toolkit.fluxcd.io \ + imagerepositories.image.toolkit.fluxcd.io imagepolicies.image.toolkit.fluxcd.io \ + imageupdateautomations.image.toolkit.fluxcd.io; do \ + echo "== $k"; kubectl --context k3s-prod get "$k" -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SUSPENDED:.spec.suspend 2>/dev/null | grep -v ""; done' +``` + +Is the suspension declared in git, or is it live-only drift? (This is the important question.) + +``` +grep -rn "suspend" clusters/ platform/ | grep -v "^Binary" +``` + +If it is not in git, it was applied by hand and nothing will restore it after a controller restart — +which means the resource may un-suspend itself at an unpredictable moment, or stay off forever. + +How far behind is it? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \ + -o custom-columns=NAME:.metadata.name,SUSPENDED:.spec.suspend,APPLIED:.status.lastAppliedRevision; echo; \ + kubectl --context k3s-prod -n flux-system get gitrepositories.source.toolkit.fluxcd.io \ + -o jsonpath="{.items[0].status.artifact.revision}{\"\n\"}"' +``` + +## Likely causes, most likely first + +### 1. A maintenance suspension nobody resumed +By far the most common, and the Velero case above is the archetype. Remediation (operator): +`flux resume kustomization ` — but **only after** reading the next paragraph. + +### 2. It is suspended for a reason that still applies +Before resuming, find out what would be applied. A tier suspended during an incident may be holding +back a change that caused the incident. The 2026-09-04 case is the cautionary one: resuming +`platform-config` while a second cluster still held the same wildcard certificate would have started an +ACME DNS-01 race on a shared Cloudflare zone. **Diff first**: compare +`status.lastAppliedRevision` against the GitRepository's current revision and read the commits in +between. + +### 3. A crashlooping leaf was suspended instead of being moved +The correct fix for "a leaf UI keeps failing and stalls the tier" is a `wait: false` tier of its own — +that is what happened to `oauth2-proxy`, `headlamp`, `homepage`, `crafty`, `velero` and `runbooks`. +Suspension as a permanent substitute leaves drift correction off for that component forever. + +### 4. Image automation +`ImageUpdateAutomation`/`ImagePolicy`/`ImageRepository` are **authored and deliberately disabled** here +(pending a Gitea write credential and `flux install --components-extra`), so if those show up, this is +expected state, not an incident. See `/runbooks/image-automation`. + +## When to escalate or stop + +- **Do not resume a gating tier (`cilium`, `platform`, `platform-config`) without the operator.** + Resuming applies every commit since the suspension in one pass, to the tier everything depends on. +- **Do resume `velero` promptly** if it is the suspended resource — a suspended backup controller is a + DR gap, and it has happened before. +- If the suspension is not in git, record it in git (or resume it). A live-only suspension is + undocumented state, which is the failure class this repo keeps finding. +- If you cannot determine *why* something was suspended, stop and ask. That is a safer outcome than + either resuming blind or leaving it off. + +--- +Source of truth: `runbooks/flux-suspended.md`. Index: `/runbooks/` +Records: `infra/PROD-PROVISIONING.md` §10 (the 2026-09-04 suspension and the queued ACME race); +`DECISIONS-LOG.md` (Velero found suspended with the suspension never in git). diff --git a/runbooks/image-automation.md b/runbooks/image-automation.md new file mode 100644 index 0000000..596f1e5 --- /dev/null +++ b/runbooks/image-automation.md @@ -0,0 +1,129 @@ +# image-automation — Flux's image-update-automation cannot write back to git + +**Alert:** `ImageUpdateAutomationFailing` (warning, 30m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.automated-updates` +**Expression:** `max by (namespace, name) (gotk_reconcile_condition{kind="ImageUpdateAutomation", type="Ready", status="False"}) == 1` + +## What this means here + +`ImageUpdateAutomation` is the write-back half of hands-off updates (ADR-0009): it commits new image +tags into git so Flux then deploys them. When it is Ready=False, **stable image advances silently +stop** — nothing breaks, nothing degrades, the platform just quietly stops receiving updates. That is +the R-UPD-02 failure mode this alert exists for. + +**Read this first: on this cluster, image automation is authored and DELIBERATELY DISABLED.** It has +been that way since 2026-08-24, pending (a) a Gitea write credential for the automation to push with +and (b) `flux install --components-extra=image-reflector-controller,image-automation-controller`. The +line is still commented out in `platform/kustomization.yaml`. So: + +- If this alert fires, the most likely explanation is that **the feature was partially enabled** — + someone uncommented the component or a rendered app started emitting `ImageRepository`/`ImagePolicy` + CRs whose controllers are not installed. +- Rendered apps emit those CRs by default; `catalog/render.py --no-image-automation` is the documented + workaround for exactly this. +- There is consequently **no real incident history** for this alert. Everything below is mechanism, not + post-mortem, and it says so honestly rather than inventing precedent. +**This alert is currently INERT.** `count(gotk_reconcile_condition)` returned **0 series** on prod on +2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every `gotk_*` rule +evaluates against nothing. Read the CRs directly with the commands below, and see +`/runbooks/targets-down` for the full list of inert rule groups. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Are the controllers even installed? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n flux-system get deploy | sort; echo; \ + kubectl --context k3s-prod api-resources --api-group=image.toolkit.fluxcd.io' +``` + +The automation objects and their messages: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get imageupdateautomations.image.toolkit.fluxcd.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,LASTRUN:.status.lastAutomationRunTime' +``` + +The controller log — the git error is always in here: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n flux-system logs deploy/image-automation-controller --tail=100 2>&1 | tail -40' +``` + +Is the repo configured with a write path at all? + +``` +grep -rn "image-automation\|ImageUpdateAutomation\|components-extra" platform/kustomization.yaml bootstrap/ clusters/ 2>/dev/null | head -20 +``` + +## Likely causes, most likely first + +### 1. The controllers are not installed (expected state here) +`api-resources --api-group=image.toolkit.fluxcd.io` returns nothing, or the deployments are absent. +Then the CRs are inert objects and the alert reflects a half-enabled feature. Remediation: either +finish enabling it (an operator runs `flux install --components-extra=...` — note that Flux's own +controllers were installed **imperatively** and carry no Flux ownership labels, so **nothing under +`clusters/**` can install a controller or change a controller flag**), or stop rendering the CRs with +`catalog/render.py --no-image-automation`. + +### 2. No git write credential +The automation must push. Prod's source is `http://gitea.gitea.svc.cluster.local:3000`, and the +read-only source credential is not a write credential. The controller log will say authentication +failed or permission denied. Remediation: a Gitea token in a SOPS-encrypted Secret referenced by the +`GitRepository`/automation, added in git — and note the standing rule from the OIDC incident: +**non-secret environment-specific values must not live inside an encrypted Secret**, because +`${BASE_DOMAIN}` substitution can never reach inside one. + +### 3. It cannot push because the branch moved +Automation commits to a branch; a force-push or a protected branch rejects it. The log names the ref. + +### 4. The commit is being made but the Kustomization is not picking it up +Then this is not the right runbook — go to `/runbooks/flux-reconcile`. + +## When to escalate or stop + +- **This is a warning with a 30m `for` for a reason: nothing is broken.** Do not treat it as an + outage. The consequence is "no new image versions", which is measured in days. +- **Do not create a Gitea write credential ad hoc.** It is a new secret in the DR path and belongs to + the operator, SOPS-encrypted, in git. +- **Do not enable the extra Flux controllers to make the alert stop.** Enabling automation on a + cluster that has never run it will start proposing image bumps across every workload at once; that is + a planned change, with `/runbooks/update-rollback` as its companion. +- If the alert is firing because the feature is half-enabled, the honest fix is to finish or to revert + — both are git changes and both want the operator's decision. + +--- +Source of truth: `runbooks/image-automation.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` (image automation authored-and-disabled since 2026-08-24; the Flux +controllers are outside Flux's own inventory). diff --git a/runbooks/image-policy-stalled.md b/runbooks/image-policy-stalled.md new file mode 100644 index 0000000..2edcf7d --- /dev/null +++ b/runbooks/image-policy-stalled.md @@ -0,0 +1,137 @@ +# image-policy-stalled — An ImagePolicy or ImageRepository cannot resolve its tag + +**Alert:** `ImagePolicyStalled` (warning, 1h) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.automated-updates` +**Expression:** `max by (namespace, name, kind) (gotk_reconcile_condition{kind=~"ImagePolicy|ImageRepository", type="Ready", status="False"}) == 1` + +## What this means here + +An `ImageRepository` cannot scan a registry, or an `ImagePolicy` cannot pick a tag out of what was +scanned. The effect is narrower than `/runbooks/image-automation`: automation itself is healthy, but +**this one service stops receiving updates** (R-UPD-01), while everything else continues. It is +therefore easy to miss, which is what the 1h `for` and the per-object labels are for. + +The same framing applies as for image-automation: **this feature is authored and deliberately disabled +on this cluster** and has been since 2026-08-24 (no Gitea write credential, and the +`image-reflector-controller` / `image-automation-controller` components are not installed — the line is +still commented out in `platform/kustomization.yaml`). Rendered apps emit `ImageRepository` and +`ImagePolicy` CRs anyway, so the objects can exist with no controller to reconcile them; +`catalog/render.py --no-image-automation` is the documented workaround. **There is no incident history +for this alert** — the section below is mechanism, honestly labelled. + +One live constraint worth knowing while reading registry errors: the images in this platform come from +`docker.io`, `ghcr.io`, `lscr.io`, `quay.io`, `registry.k8s.io` and `registry.gitlab.com`. Docker +Hub's anonymous rate limit is the most likely real-world cause of a scan failure once automation is +switched on. +**This alert is currently INERT.** `count(gotk_reconcile_condition)` returned **0 series** on prod on +2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every `gotk_*` rule +evaluates against nothing. Read the CRs directly with the commands below, and see +`/runbooks/targets-down` for the full list of inert rule groups. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which object, and the message (the message is nearly always the whole answer): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get imagerepositories.image.toolkit.fluxcd.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,IMAGE:.spec.image,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,TAGS:.status.lastScanResult.tagCount; echo; \ + kubectl --context k3s-prod get imagepolicies.image.toolkit.fluxcd.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,LATEST:.status.latestImage,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message' +``` + +Are the controllers installed at all? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n flux-system get deploy | grep -i image || echo "image controllers NOT installed (expected on this cluster)"' +``` + +The reflector's log: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n flux-system logs deploy/image-reflector-controller --tail=80 2>&1 | tail -30' +``` + +Can the cluster reach the registry at all? (Default-deny is the baseline; a new namespace may simply +not be allowed out.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get pods -A --field-selector status.phase!=Running | head; echo; \ + kubectl --context k3s-prod get events -A --field-selector reason=Failed --sort-by=.lastTimestamp | grep -i "pull\|registry" | tail -10' +``` + +## Likely causes, most likely first + +### 1. The image controllers are not installed (expected state here) +The CRs exist because `catalog/render.py` emits them; nothing reconciles them. Remediation: render with +`--no-image-automation`, or have the operator finish enabling automation. Note that the Flux +controllers were installed **imperatively** and are outside Flux's own inventory, so no git change can +install them. + +### 2. Registry rate limit +Docker Hub anonymous pulls/scans are limited per IP, and all three nodes share one WAN address. The +message says `toomanyrequests`. Remediation: a registry credential (SOPS-encrypted, in git), or scan +less often via `spec.interval`. + +### 3. Registry auth +A private repository with no `secretRef`, or an expired token. The message names it. + +### 4. The policy's range matches nothing +`ImagePolicy` semver ranges come from each ServiceDefinition's `update.range` (for example +`">=2024.0.0"` for pihole, `">=24.0.0"` for keycloak). A range that no published tag satisfies — or a +repository that publishes only `latest` (this platform has `lscr.io/linuxserver/jackett:latest` and +`plexinc/pms-docker:latest`) — leaves the policy with nothing to select. Remediation: fix the range in +`catalog/.yaml` and re-render; a `latest`-only image cannot participate in automation at all +and should not carry a policy. + +### 5. Egress is blocked for that namespace +Default-deny everywhere (R-NET-01) means outbound registry access is an explicit allow. Cross-read +`/runbooks/denied-flows`, and in particular the post-Cilium-upgrade finding that `toEntities: world` no +longer covers a call whose destination resolves to an in-cluster backend. + +## When to escalate or stop + +- **Warning, 1h, single service: this does not page.** The consequence is one workload pinned at its + current version. +- **Do not widen an `update.range` to make the alert clear.** The range is the service's update policy + (R-UPD-01); loosening it to satisfy a scanner is how an unintended major version arrives. +- **Do not add registry credentials ad hoc** — new secret, DR path, operator's call. +- If the same object has been stalled for days, the honest outcome is to remove its automation CRs + (`--no-image-automation`) rather than leave a permanently-failing object training everyone to ignore + the alert. Alert fatigue is a recorded, expensive failure mode on this cluster. + +--- +Source of truth: `runbooks/image-policy-stalled.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` (image automation authored-and-disabled; `--no-image-automation`); +`catalog/*.yaml` (`update.channel` / `update.range` per service). diff --git a/runbooks/keycloak-client-secrets.md b/runbooks/keycloak-client-secrets.md new file mode 100644 index 0000000..9c72463 --- /dev/null +++ b/runbooks/keycloak-client-secrets.md @@ -0,0 +1,201 @@ +# keycloak-client-secrets — The OIDC client reconciler is failing or has gone silent + +**Alerts:** `KeycloakClientSecretReconcilerFailing` (warning, 15m) and +`KeycloakClientSecretReconcilerStale` (warning, 2h) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity` +**Expressions:** +``` +max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"}) > 0 +absent(kube_job_status_succeeded{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"} == 1) +``` + +## What this means here + +These two rules watch the CronJob (every 15 minutes) that repairs OIDC client **secrets** and +environment-derived client **URLs** in the Keycloak realm. They exist because of the single worst +green-looking outage this cluster has had, and they are best understood through it. + +### The 2026-09-06 lockout — why "everything is green" means nothing here + +The Keycloak Operator resolves each OIDC client's secret from an `(env:NAME)` placeholder in the realm +import. **When it cannot resolve one, it stores the LITERAL placeholder text as the client secret and the +import Job still reports SUCCESS.** On prod, **all 12 clients** ended up holding the literal placeholder +and **every OIDC login on the cluster failed** with *"Invalid client or Invalid client credentials"*. +Meanwhile: the import Job succeeded, the CR rested at `Done=True / Started=False`, +`KeycloakDown` and `KeycloakDatabaseDown` were quiet, and **the conformance test that asserts +`Done=True` passed throughout the outage.** The only observable was a human failing to log in. Dev had +exactly one broken client (oauth2-proxy — its key was added *after* dev's import completed, and +**a completed import is never re-run**). + +The realm import is a **ONE-SHOT that fails open**: the operator logs *"Job already executed - not +recreating"*, so a git change to `keycloak-realm-import.yaml` does **not** re-import. + +### The second half: redirect URIs, same one-shot, one field over + +The realm import carries **26 `${BASE_DOMAIN}` references** (39 files under `platform/` and `apps/` +reference the knob). Substitution happened **once**, at the old hostname, and then froze. After the apex +flip, *"every OIDC login fails with a `redirect_uri` mismatch while the CR still reads `Done=True` and +nothing alerts."* The reconciler was extended with a second pass over `redirectUris`, `webOrigins`, +`rootUrl`, `baseUrl` and `adminUrl`, with desired state **generated** by +`ci/render-keycloak-client-urls.py` (gated by an L0 `--check`). + +Semantics worth knowing before you read a Job log: +- URLs are copied **verbatim, including `${BASE_DOMAIN}`**, and rendered by the *same* Flux + Kustomization as the import. +- **List fields compare order-insensitively (`sorted`, not `set`)** — measured, not defensive: Keycloak + genuinely reorders lists, and an order-sensitive compare would have rewritten `kubernetes` (8 URIs) and + `oauth2-proxy` (2) **every 15 minutes forever while reporting success**. +- A field the desired state omits is **UNMANAGED, never forced empty**. +- **Public clients' URLs are managed even though their secrets are not** (`kubernetes` is public, with + PKCE, so no kubeconfig holds a secret). +- The two passes **fail closed independently** — demonstrated live: pass 1 failed on an empty secrets + directory while pass 2 completed. + +### Why there are exactly two rules, and what each catches + +`...Failing` sees a run that *ran and failed* — unrepairable drift, or the admin API unreachable. It +fires alongside `KeycloakDown` during an IdP outage, deliberately: the reconciler genuinely is not +running, so treat `KeycloakDown` as the cause when both are lit. Self-clears when failed Jobs age out +(TTL 1h). + +`...Stale` sees the **healer being quietly absent** — the CronJob suspended, never applied by Flux, or +unable to schedule a pod at all. That is the "looks applied and isn't" failure this whole change exists +to end. It is also the rule that would catch the healer missing during a domain migration. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Does the CronJob exist, is it suspended, and when did it last succeed? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak get cronjobs.batch \ + -o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,SUSPEND:.spec.suspend,LAST:.status.lastScheduleTime,LASTSUCCESS:.status.lastSuccessfulTime' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak get jobs.batch --sort-by=.metadata.creationTimestamp \ + -o custom-columns=NAME:.metadata.name,SUCCEEDED:.status.succeeded,FAILED:.status.failed,START:.status.startTime | tail -15' +``` + +**Read the Job log — it names the pass, the client, and for a URL the exact before/after, and it never +prints a secret** (that is a design property, asserted by the L0 check): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak logs job/$(kubectl --context k3s-prod -n keycloak get jobs.batch \ + --sort-by=.metadata.creationTimestamp -o jsonpath="{.items[-1:].metadata.name}") --tail=200' +``` + +Is the IdP itself up? (If not, that is the cause; come back here after.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak get pods; echo; \ + kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak' +``` + +Confirm the declaration is self-consistent **offline**, before touching the cluster — this is the same +check the merge gate runs unconditionally (Stage 3d): + +``` +bash conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh +python3 ci/render-keycloak-client-urls.py --check +``` + +Confirm what a user actually experiences: + +``` +curl -s -o /dev/null -w '%{http_code}\n' --resolve keycloak.home.keithjackson.dev:443:192.168.2.20 \ + https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration +``` + +``` +infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod auth whoami' +``` + +(Expected on a working OIDC path: `oidc:` with `[oidc:platform-admins]`. From this workstation the +OIDC context hangs on a browser prompt — that is not evidence of breakage.) + +## Likely causes, most likely first + +### 1. Keycloak or its database is down, so the reconciler cannot reach the admin API +`...Failing` fires alongside `KeycloakDown` by design. Fix the IdP first: `/runbooks/keycloak-down`, +`/runbooks/keycloak-db`. Nothing here is actionable until the admin API answers. + +### 2. Unresolvable drift on one client +The log names the client and the pass. Most often a secret key missing from the SOPS Secret, or a client +in the realm that git does not describe. The contract: keys are **`-client-secret`**, and the +public `kubernetes` client declares **no** secret. Remediation is a git change to the declaration, then +let the CronJob heal it — the reconciler is compare-then-write and idempotent. + +### 3. A hostname migration left the realm behind +The `redirect_uri` mismatch case. Regenerate desired state with `ci/render-keycloak-client-urls.py` and +push; the reconciler's second pass applies it. Historical scale: re-rendering at the post-flip apex +produced **24 drifted fields across 12 clients**, with `kubernetes` correctly `ok` because its localhost +callbacks are not domain-derived. + +### 4. The CronJob is suspended or was never applied (`...Stale`) +Check `spec.suspend` and the Flux tier. The ordering requirement is a **real Flux `dependsOn` edge with +`wait: true`**, not a comment — the L0 check asserts that, because a comment is not an ordering guarantee. + +### 5. The pod cannot read its own secrets +A specific, recorded trap: the reconciler needs **`fsGroup: 65532`** with the client-secrets volume at +**`defaultMode: 0440`**. With `0400` and no fsGroup the container dies with `PermissionError` on **every** +run. If the log shows that, this is it. + +### 6. Everything is green and logins still fail +Then trust the human, not the dashboard — that is the entire lesson of 2026-09-06. Audit the live realm +directly with `kcadm.sh` inside `keycloak-0`. + +## When to escalate or stop + +- **Do not "fix" the realm by re-applying the import.** It is a one-shot: the operator logs *"Job already + executed - not recreating"* and nothing happens. The documented path is `kcadm.sh` inside `keycloak-0` + (admin credentials in Secret `keycloak-admin-credentials`; `kcadm` is present, `curl` and `python3` are + **not**) **and** fixing git so the change is reproducible. Doing only the live half creates drift; doing + only the git half changes nothing. +- **Never print or paste a client secret.** The reconciler deliberately logs URL values in full (they are + public hostnames) and never logs secrets. Keep that property. +- **Do not widen the reconciler's endpoint allow-list.** It is hard-scoped so that `/users`, `/groups`, + `/roles` and `/authentication` are unreachable — a blast-radius bound, not an oversight. +- **Do not disable or suspend the CronJob to stop the alert.** That is the healer, and `...Stale` exists + precisely because a suspended healer is invisible. +- Escalate for: rotating client secrets, editing the live realm, or anything touching MFA (a + conditional-OTP misconfiguration once 403'd every platform-admin login with + `"Invalid role name: null"`, fixed to `condUserRole: platform-admin`). + +--- +Source of truth: `runbooks/keycloak-client-secrets.md`. Index: `/runbooks/` +Records: `ci/l0.sh` Stage 3d and +`conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh` (the 2026-09-06 incident and the +reconciler's contract — 212 L0 assertions, an 84-assertion selftest, 12/12 injected regressions killed); +`DECISIONS-LOG.md` "Keycloak client REDIRECT URIs reconcile too — the same one-shot, one field over". diff --git a/runbooks/keycloak-db.md b/runbooks/keycloak-db.md new file mode 100644 index 0000000..0fd0d7a --- /dev/null +++ b/runbooks/keycloak-db.md @@ -0,0 +1,169 @@ +# keycloak-db — Keycloak's PostgreSQL is unavailable + +**Alert:** `KeycloakDatabaseDown` (critical, 5m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity` +**Expression:** +``` +max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*", condition="true"}) == 0 +or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*"}) +``` + +## What this means here + +No ready `keycloak-db-*` pod for five minutes. `keycloak-db` is a **CloudNativePG Cluster** with +**3 instances** on Longhorn (3 replicas each), and it is the database the IdP cannot live without. The +`pod=~"keycloak-db-.*"` scoping matters for the same reason as in `/runbooks/keycloak-down`: the +namespace also holds the Keycloak server and operator pods, and an unscoped `max()` would be masked by +those. + +**Blast radius is the whole platform's identity:** Keycloak fails → kubectl OIDC, Grafana, Headlamp, +Gitea and forward-auth all fail. The certificate-based kubeconfig is an independent break-glass path and +still works. This page is deliberately served **without** forward-auth so it stays readable during +exactly this outage. + +## The HA properties you are relying on, and why they are set the way they are + +- **`instances: 3`.** All three CNPG Clusters used to run `instances: 1`, and all three primaries had + drifted onto one node — **on 2026-09-07 that node took SSO, Nextcloud and Semaphore down together.** + Scale-out to 3 is what gives you a standby to fail over to. +- **`podAntiAffinityType: required`**, not CNPG's default `preferred`. `preferred` is a scheduler + *hint* and on the path that matters — recreating an instance after node loss — it co-locates. + Recorded conclusion: *"`preferred` is how a 3-instance cluster decays back into the single-node + failure domain."* The accepted cost is that an instance stays **Pending** until a node returns. +- **`primaryUpdateMethod: switchover`.** The CRD default is `restart`, which is a delete-and-recreate of + the **primary** — tens of seconds with no Postgres, i.e. an SSO outage. Constraint to remember: under + `switchover` the operator **rejects an update that changes `imageName` and a postgresql parameter in + the same reconcile**. +- **Synchronous replication `method: any`, `number: 1`, `dataDurability: preferred` — never `required`.** + `required` is a strict quorum: with fewer than `number` healthy standbys, **writes BLOCK**. On this + database one degraded standby (a node reboot, a Longhorn rebuild, a Cilium restart) would become a + total write outage. `dataDurability` is only honoured while `standbyNamesPre/Post` are unset. + +**And the thing that is easy to get wrong:** *"Standbys replicate corruption and deletions faithfully; +only the Barman archive is a backup."* Three ready replicas are not a recovery plan. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +The Cluster's own verdict first — fully-qualified, because `cluster` is an ambiguous short name: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase' +``` + +Where the instances are, and whether they are all on one node (the 2026-09-07 defect): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak get pods -l cnpg.io/cluster=keycloak-db -o wide' +``` + +Why a pod is not ready: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak describe pod keycloak-db-2 | sed -n "/Events:/,\$p"; echo ====; \ + kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=80 2>&1 | tail -40' +``` + +The conditions that matter for recoverability — check these **before** any recovery action: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak keycloak-db \ + -o custom-columns=ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get backups.postgresql.cnpg.io -n keycloak' +``` + +`kubectl get backup -A` prints "No resources found" here even with 27 CNPG backups present — three CRDs +register the kind `Backup` and the short name resolves to Longhorn's, which has none. Never decide +anything from the short name. + +Storage underneath: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak get pvc; \ + kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy' +``` + +## Likely causes, most likely first + +### 1. A node is down or was rebooted +The instances live on Longhorn with hard anti-affinity, so a node loss takes one instance. With +`instances: 3` the cluster should survive it and promote; with `podAntiAffinityType: required` the lost +instance stays **Pending** until a node returns, which is expected and accepted. Go to +`/runbooks/node-down`. + +### 2. The data PVC filled up +The most dangerous cause, because the mechanism is a *backup* fault presenting as a *storage* failure. +With `archive_mode=on` and a failing `archive_command`, Postgres **refuses to recycle** unarchived WAL; +`pg_wal` shares the data PVC on every Cluster here (no `walStorage:` configured) and grows until the +volume is full and the postmaster **PANICs**. Measured on this very database: 17 unarchived segments / +~268 MiB on a 5 GiB PVC, growing ~6.8 MiB/h — about 29 days from 93% free to dead. Go to +`/runbooks/cnpg-archiving` and `/runbooks/pvc-capacity`. **Never delete WAL by hand.** + +### 3. A Longhorn volume is degraded or faulted +`/runbooks/longhorn-degraded`, `/runbooks/longhorn-faulted`. + +### 4. An upgrade is stuck mid-switchover +Signature: the operator refusing an update that changes `imageName` and a postgresql parameter together. +Split the change in git. See `/runbooks/update-rollback`. + +### 5. The database is up and Keycloak still fails +Then read `/runbooks/keycloak-down` and, more likely, `/runbooks/keycloak-client-secrets` — the +2026-09-06 lockout had all 12 OIDC clients holding literal placeholder secrets behind an entirely green +cluster. + +## When to escalate or stop + +- **Escalate immediately.** This is SSO's foundation; a 5m `for` on a `critical` is already generous. +- **Never change `dataDurability` to `required`** while trying to "make replication safer" — it converts + one degraded standby into a total write outage on the SSO database. +- **Do not delete a CNPG pod or PVC to force a rebuild.** The operator manages instance lifecycle; + deleting a PVC destroys that instance's data and, if it is the primary, may lose transactions that the + archive does not have. +- **Do not attempt a restore before checking `firstRecoverabilityPoint` and `ContinuousArchiving`.** Prod + ran 38+ hours from bootstrap with `ContinuousArchiving=False` and `firstRecoverabilityPoint: null` on + all three Clusters and nothing said so. Know what you actually have before you destroy what you have. +- **Standbys are not backups.** Recovery is from the Barman archive in R2 (`CNPG_BACKUP_PATH = + s3://k3s-prod-backup/barman`, 30d retention), which is an operator operation. + +--- +Source of truth: `runbooks/keycloak-db.md`. Index: `/runbooks/` +Records: commits `f338e91` (instances 3), `e8d8601` (required anti-affinity + switchover), `cac413e` +(sync replication, `dataDurability: preferred`); `platform/cnpg/alerts.yaml` header (the 2026-09-06 +archiving incident and the measured WAL fill rate). diff --git a/runbooks/keycloak-down.md b/runbooks/keycloak-down.md new file mode 100644 index 0000000..d14948d --- /dev/null +++ b/runbooks/keycloak-down.md @@ -0,0 +1,166 @@ +# keycloak-down — The Keycloak IdP is down (SSO is unavailable) + +**Alert:** `KeycloakDown` (critical, 5m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity` +**Expression:** +``` +max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+", condition="true"}) == 0 +or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+"}) +``` + +## What this means here + +No ready Keycloak **server** pod for five minutes. The regex is deliberately scoped to +`keycloak-[0-9]+` (the StatefulSet pods `keycloak-0`, `keycloak-1`) because the `keycloak` namespace also +holds the operator (`keycloak-operator-*`) and the database (`keycloak-db-*`); an unscoped `max()` would +be satisfied by those being ready and would never fire for the IdP itself. That scoping was a fix — the +original matched `.*postgres.*`. + +## Blast radius — read this before triaging anything else + +Keycloak is the root of trust. With SSO down, all of these are down or degraded **at once**: + +- **kubectl OIDC login** (the public `kubernetes` client with PKCE, groups → RBAC) +- **Grafana** (OIDC), **Headlamp** (forwards the user's id_token to the apiserver), **Gitea** +- **forward-auth** via oauth2-proxy, and the five per-app oauth2-proxy sidecars (pihole, jackett, + qbittorrent, radarr, sonarr) — these **fail closed**, 302ing to a Keycloak that is not there + +**You can still get in.** The certificate-based kubeconfig is untouched by the OIDC work and remains a +fully independent break-glass path — that is a deliberate design property, not an accident. From this +workstation the working non-interactive path is the dev-container kubeconfig; the host `k3s-oidc` +context will hang on a browser prompt, which is not the cluster being down. + +**And you can still read this page.** The runbook route is deliberately **public, with no +forward-auth**, precisely so that `keycloak-down` is readable by an operator who cannot log in. If that +ever changes, this runbook becomes unreachable exactly when it is needed. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Everything in the namespace, in one view — server, operator and database together: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak get pods -o wide; echo; \ + kubectl --context k3s-prod -n keycloak get statefulsets,deployments' +``` + +Is the database the real cause? (It usually is. Fully-qualified: `clusters.postgresql.cnpg.io`.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak \ + -o custom-columns=NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase' +``` + +The server's own complaint: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak logs keycloak-0 --tail=120 2>&1 | tail -50; echo ====; \ + kubectl --context k3s-prod -n keycloak describe pod keycloak-0 | sed -n "/Events:/,\$p"' +``` + +The operator's view of the CR: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak get keycloaks.k8s.keycloak.org -o yaml | grep -A20 "status:"' +``` + +Is it reachable at all, from outside, with an explicit SNI (in-pod HTTPS tests to these hostnames are +misleading here): + +``` +curl -s -o /dev/null -w '%{http_code}\n' --resolve keycloak.home.keithjackson.dev:443:192.168.2.20 \ + https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration +``` + +## Likely causes, most likely first + +### 1. The CNPG database is not available +Keycloak cannot start without Postgres. This is the highest-probability cause and it has its own runbook: +`/runbooks/keycloak-db`. Note that `keycloak-db` is a 3-instance CNPG Cluster with +`primaryUpdateMethod: switchover` and synchronous replication at `dataDurability: preferred` — +`preferred` specifically so that one degraded standby does **not** block writes, which on this database +would be a total SSO write outage. + +### 2. A node that held the Keycloak pods went away +Both Keycloak pods and all three `keycloak-db` instances previously ran on **one node** — all three CNPG +primaries had drifted onto `k3s-prod-2` — and **on 2026-09-07 that node took SSO, Nextcloud and +Semaphore down together**. The fixes were `instances: 3` on every Cluster and +`podAntiAffinityType: required` (CNPG's default `preferred` is only a scheduler hint and co-locates on +the path that matters — recreating an instance after node loss). Check `-o wide` above: if everything is +on one node, that is a live re-occurrence of the same defect. Go to `/runbooks/node-down`. + +### 3. A HelmRelease / operator upgrade went bad +Read `/runbooks/update-rollback`. Under `primaryUpdateMethod: switchover` the CNPG operator **rejects an +update that changes `imageName` and a postgresql parameter in the same reconcile** — a stuck reconcile +with that message is this. + +### 4. TLS to the issuer is broken +Keycloak's clients validate the issuer's certificate. While the wildcard was LE-**staging**, in-cluster +clients rejected it with `x509: certificate signed by unknown authority`. If `/runbooks/cert-issuance` or +`/runbooks/cert-expiry` is also firing, fix that first. + +### 5. Logins fail but Keycloak is UP +Then this is the **wrong alert and the wrong runbook**, and the distinction has cost this cluster dearly. +On 2026-09-06 every OIDC login on prod failed — *"Invalid client or Invalid client credentials"* — with +**all 12 clients holding the literal `(env:...)` placeholder as their secret**, while +`KeycloakDown`/`KeycloakDatabaseDown` were quiet, the import Job reported success, the CR read +`Done=True`, and the conformance test that asserts `Done=True` **passed throughout**. The only observable +was a human failing to log in. Go to `/runbooks/keycloak-client-secrets`. +Similarly, a `redirect_uri` mismatch after a hostname change is not an outage — same runbook. +And once, MFA broke every platform-admin login on its own: conditional-OTP used a non-existent config +key, giving `"Invalid role name: null"` → 403, fixed to `condUserRole: platform-admin`. + +## When to escalate or stop + +- **Escalate immediately.** SSO down is a platform-wide outage; treat it as such. +- **Use the certificate-based kubeconfig** (the dev-container path) rather than trying to fix OIDC login + first. Do not burn the outage on getting `kubectl` to authenticate through the thing that is down. +- **Do not edit the realm to "reset" anything.** The realm import is a **ONE-SHOT that fails open**: the + operator logs *"Job already executed - not recreating"*, so a git change to + `keycloak-realm-import.yaml` does **not** re-import. The documented path is `kcadm.sh` inside + `keycloak-0` (admin credentials in the Secret `keycloak-admin-credentials`; `kcadm` is present, + `curl`/`python3` are **not**) **and** fixing git for reproducibility. Doing only one of the two is how + drift is created. +- **Do not delete the Keycloak CR or the StatefulSet.** Recreating the IdP is a restore operation, not a + restart. +- **Do not touch `keycloak-db` volumes.** If the database is the cause, read `/runbooks/keycloak-db` and + `/runbooks/cnpg-archiving`; only the Barman archive is a backup, standbys replicate deletions + faithfully. +- Break-glass local admins exist for gitea, nextcloud and semaphore. Nextcloud has **no native + group→admin mapping**, so platform-admins there are promoted by hand. + +--- +Source of truth: `runbooks/keycloak-down.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` (the 2026-09-06 client-secret lockout; the 2026-09-07 single-node SSO +outage); commits `f338e91`, `e8d8601`, `cac413e` (CNPG instances/anti-affinity/switchover/sync). diff --git a/runbooks/kubevirt-control-plane.md b/runbooks/kubevirt-control-plane.md new file mode 100644 index 0000000..c6c3cfb --- /dev/null +++ b/runbooks/kubevirt-control-plane.md @@ -0,0 +1,156 @@ +# kubevirt-control-plane — The KubeVirt control plane is degraded + +**Alert:** `KubeVirtControlPlaneDown` (critical, 10m) +**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt` +**Expression:** +``` +(sum(kubevirt_virt_controller_ready_status) == 0) +or (sum(kubevirt_virt_api_ready_status) == 0) +or (absent(up{job="kubevirt-prometheus-metrics"} == 1)) +``` + +## What this means here + +No `virt-controller` or no `virt-api` replica reports ready, so **VM lifecycle operations — create, start, +stop, migrate — will fail** (docs/06). Note carefully what this does *not* mean: **already-running VMs +keep running.** `virt-launcher` pods and their qemu processes are independent of the control plane. So +this is an "I cannot manage VMs" alert, not "the VMs are down". + +The `absent()` clause also fires if the whole KubeVirt metrics target disappears. The scrape job label is +**`kubevirt-prometheus-metrics`** (the Service name), **not** `kubevirt` — that precision is a fix: this +very alert once used a non-existent metric with the wrong job label and fired falsely (commit `b57b3e9`). +There is a sibling precedent too: a broken HTTP CDI ServiceMonitor caused a false `CDIOperatorDown` +(commit `6a7e9d1`). **So the first question with this alert is always whether it is telling the truth.** + +Live state, measured 2026-09-13: `kubevirt-prometheus-metrics` **is** among Prometheus's 19 scrape jobs +and `kubevirt_virt_api_ready_status` returns **2 series**, so the first two clauses are evaluable and +quiet. (`kubevirt_vmi_phase_count` was absent at the same moment — that metric only exists when VMIs do, +which affects `/runbooks/kubevirt-launch-failed`, not this rule.) + +Platform context: **KubeVirt v1.9.0 + CDI v1.66.0**, on real nested KVM (`/dev/kvm` present, vmx/svm +verified; the Proxmox nodes use `cpu_type=host`). The `kubevirt` and `cdi` namespaces deliberately carry +**no `app-namespace` label**, so Kyverno does not generate the default-deny baseline into them — it would +sever the virt control plane. Do not "fix" that by adding the label. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Is the control plane actually unhealthy, or is this a metrics artefact? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n kubevirt get pods -o wide; echo; \ + kubectl --context k3s-prod get kubevirts.kubevirt.io -A \ + -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,READY:.status.conditions[?\(@.type==\"Available\"\)].status' +``` + +Are the VMs themselves still running? (This is the reassuring half.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get vmi -A -o wide; echo; \ + kubectl --context k3s-prod get vm -A' +``` + +Is the scrape target up, and do the two metrics exist? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + for q in "up{job=\"kubevirt-prometheus-metrics\"}" "sum(kubevirt_virt_controller_ready_status)" "sum(kubevirt_virt_api_ready_status)"; do \ + printf "%-55s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \ + | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done' +``` + +Why a component is unready: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n kubevirt logs deploy/virt-controller --tail=80 2>&1 | tail -30; echo ====; \ + kubectl --context k3s-prod -n kubevirt logs deploy/virt-api --tail=80 2>&1 | tail -30; echo ====; \ + kubectl --context k3s-prod -n kubevirt get events --sort-by=.lastTimestamp | tail -20' +``` + +Is the Flux tier that owns KubeVirt healthy? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \ + -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message' +``` + +## Likely causes, most likely first + +### 1. The metrics target, not the control plane +Check the pods first. If `virt-api` and `virt-controller` are Running and Ready, this is a monitoring +fault — and there is direct precedent for false positives on this exact alert (`b57b3e9`) and its CDI +sibling (`6a7e9d1`). Remediation: fix the ServiceMonitor/metric in git. Cross-read +`/runbooks/targets-down`. + +### 2. A node that hosted the control-plane pods went away +`virt-controller` and `virt-api` are ordinary Deployments. Go to `/runbooks/node-down`. + +### 3. The KubeVirt Flux tier failed to reconcile +Specific known hazard: **`platform/kubevirt` still uses remote kustomize bases**, and the +kustomize-controller's git fetch has a **27s timeout** that remote GitHub bases flake against — a +`BuildFail` there is probably this. `external-snapshotter` was vendored into the repo for exactly this +reason; vendoring KubeVirt the same way is the durable fix. See `/runbooks/flux-reconcile`. + +### 4. Node pressure +Heavy sustained node load — TCG-emulated VMs are the recorded example — can degrade the Cilium datapath +into intermittent LB-VIP reachability problems, which looks like control-plane flapping. A restart +cleared it. Cross-read `/runbooks/node-io` and `/runbooks/node-cpu`. + +### 5. A bad operator upgrade +`/runbooks/update-rollback`. + +### 6. A NetworkPolicy severed the control plane +Only if someone added `homelab.keithjackson.dev/app-namespace=true` to `kubevirt` or `cdi`. Those +namespaces are deliberately unlabelled so Kyverno does not inject a default-deny baseline that would cut +virt-handler off. Reverting that label is the fix. + +## When to escalate or stop + +- **Prioritise honestly.** VM *lifecycle* is broken; running VMs are not. The UniFi VM keeps serving. + Treat this as urgent-but-not-catastrophic unless a VM also needs to start. +- **Do not drain a node to move virt pods.** This cluster does not drain, and a VM live-migrates on drain + **only** with per-VM `evictionStrategy: LiveMigrate` — a blanket setting would wedge drains for + RWO-disk VMs. +- **Do not delete a VMI or `virt-launcher` pod** while the control plane is down: nothing will bring the + VM back until `virt-controller` is ready again. +- **Do not add the `app-namespace` label to `kubevirt` or `cdi`** to "harden" them. +- Escalate for vendoring the remote bases or for any operator upgrade. If the alert is a false positive, + fixing the rule is itself worth a change — a permanently-firing critical is a recorded, expensive + failure mode here. + +--- +Source of truth: `runbooks/kubevirt-control-plane.md`. Index: `/runbooks/` +Records: commits `b57b3e9` (this alert's wrong metric/job label) and `6a7e9d1` (false `CDIOperatorDown`); +`DECISIONS-LOG.md` (KubeVirt v1.9.0 + CDI v1.66.0 on nested KVM; the remote-base 27s fetch timeout). diff --git a/runbooks/kubevirt-crashloop.md b/runbooks/kubevirt-crashloop.md new file mode 100644 index 0000000..9d612cb --- /dev/null +++ b/runbooks/kubevirt-crashloop.md @@ -0,0 +1,163 @@ +# kubevirt-crashloop — A VM's virt-launcher pod is restarting repeatedly + +**Alert:** `KubeVirtVMCrashLooping` (warning, 10m) +**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt` +**Expression:** `increase(kube_pod_container_status_restarts_total{pod=~"virt-launcher-.*"}[15m]) > 3` + +## What this means here + +More than three restarts of a `virt-launcher` pod in 15 minutes. The threshold is chosen to distinguish a +crash-**loop** from a one-off reboot: a guest that restarts once is not interesting, a guest that restarts +four times in a quarter of an hour is either panicking, being OOM-killed, or crashing qemu. + +Note what layer this watches: the **pod**, not the guest. `virt-launcher` wraps qemu, so a guest kernel +panic, an OOM kill of the launcher, and a qemu crash all present the same way here. The distinction comes +from the logs and the events, not from the alert. + +This is a `warning`, correctly: the VM tier is deliberately non-gating (`apps-vms` is `wait: false`) and a +crash-looping VM does not affect the container platform. The one caveat is that on this network the UniFi +VM is the home network's controller, so *that* VM crash-looping is user-visible. + +Two platform facts that shape the likely causes: + +- These are **HP EliteDesk 800 G5 Minis with ~22 GiB usable each**, running the whole platform. A VM is + the largest discretionary memory consumer on a node, so OOM is a live possibility rather than a + theoretical one. +- **`useEmulation: false` is set deliberately** — on KubeVirt v1.9, `true` silently forces TCG. A VM + running under TCG burns a whole core per vCPU, and heavy sustained TCG load has been observed to degrade + the Cilium datapath into intermittent LB-VIP unreachability (cleared by a restart). So "the VM is + unstable **and** the network is flaky" is usually one cause, not two. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which launcher, how many restarts, on which node: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get pods -A -l kubevirt.io=virt-launcher -o wide; echo; \ + kubectl --context k3s-prod get vmi -A -o wide' +``` + +Why it died — the previous container's log is the one that matters: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n unifi logs -l kubevirt.io=virt-launcher --previous --tail=120 2>&1 | tail -60' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n unifi describe pod -l kubevirt.io=virt-launcher 2>&1 | sed -n "/Last State/,\$p" | head -40' +``` + +Was it OOM-killed? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get events -A --field-selector reason=OOMKilling --sort-by=.lastTimestamp | tail -10; echo; \ + kubectl --context k3s-prod top pods -A --sort-by=memory | head -10' +``` + +``` +ssh k3s-prod-1 'dmesg -T | grep -iE "oom|killed process" | tail -10; free -h; swapon --show' +``` + +(An empty `swapon --show` is what you want. On 2026-09-07 all three prod nodes came back from a reboot with +977 MiB of swap **active** while the role reported success — `SwapTotal == 0` is not evidence swap is off. +See `/runbooks/node-memory`.) + +Is it really TCG rather than KVM? + +``` +ssh k3s-prod-1 'ls -l /dev/kvm; grep -c -E "vmx|svm" /proc/cpuinfo; ps -eo pid,pcpu,comm | grep -i qemu' +``` + +Is the disk healthy underneath? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ + -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state | grep -v healthy' +``` + +## Likely causes, most likely first + +### 1. The guest is panicking or the guest OS is broken +Read the serial console via the previous container's log. A guest that panics at boot after a change is +usually a bad cloud-init or a bad image. Two precise traps here: **KubeVirt caps inline `userData` at 2048 +bytes** (the shipped payload is a deliberately minimal ~1.1 KB inline blob; `userDataSecretRef` was tried +and reverted), and **`apps-vms` must NOT carry `postBuild.substituteFrom`** — Flux's envsubst rewrote +`${...}` inside the cloud-init and corrupted it (commit `da1f833`). Do not restore substitution there for +consistency with the app waves. + +### 2. The launcher is being OOM-killed +On ~22 GiB nodes running the whole platform, a VM with a generous memory request plus Prometheus (3Gi) and +Grafana (768Mi) is tight. Remediation: lower the VM's memory in git, or address node memory +(`/runbooks/node-memory`). Check swap is genuinely off first — active swap makes the kubelet's accounting +lie and eviction misbehave. + +### 3. qemu is crashing +The launcher log will carry the qemu error. Most often a device or firmware mismatch: a UEFI image needs +`firmware.efi`, and `containerDisk` does not work here at all (it requires the ALPHA `ImageVolume` gate, +off on k3s — use a CDI DataVolume). See `/runbooks/kubevirt-launch-failed`. + +### 4. The disk is unavailable or degraded mid-flight +A `longhorn-vm` volume that went degraded or faulted takes the guest with it. Cross-read +`/runbooks/longhorn-degraded` and `/runbooks/longhorn-faulted`. Remember `longhorn-vm` is **Block mode +only** and `migratable: true`; a VM on plain `longhorn` cannot live-migrate. + +### 5. Node I/O pressure is starving the guest +A guest whose virtual disk cannot complete writes will wedge and be restarted. `/runbooks/node-io` — and +note the guest's writes land on the same partition as the embedded etcd WAL, so a thrashing VM is a +control-plane risk, not just its own problem. + +### 6. It is running under TCG +`useEmulation: true` forces it on v1.9. Confirm with `/dev/kvm` presence and qemu CPU usage. Fix in git. + +## When to escalate or stop + +- **Bound the damage first.** If the VM is thrashing the disk, its restarts are loading the node that + etcd writes to. If `/runbooks/node-io` is also firing, stopping the VM (an operator action on the `VM` + object, `spec.running: false` in git) is the right move — protecting the control plane beats keeping one + VM trying. +- **Do not delete the DataVolume or the PVC.** A re-import is hours of I/O on the shared partition. +- **Do not `kubectl drain`** the node to relocate it. This cluster does not drain, and a VM live-migrates + on drain **only** with per-VM `evictionStrategy: LiveMigrate`; a blanket setting would wedge drains for + RWO-disk VMs. +- Escalate if the guest needs in-guest repair — that is console work, and for the UniFi VM it is also + home-network-visible. +- If the alert is firing for a `virt-launcher` in a leaked test namespace, deleting that namespace is the + fix; leaked TCG test namespaces have caused API blips here before. + +--- +Source of truth: `runbooks/kubevirt-crashloop.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` Session 7 (`useEmulation`, `firmware.efi`, `containerDisk`/ImageVolume); +commit `da1f833` (cloud-init corrupted by envsubst); the 2026-09-07 swap-persistence finding. diff --git a/runbooks/kubevirt-launch-failed.md b/runbooks/kubevirt-launch-failed.md new file mode 100644 index 0000000..6308be6 --- /dev/null +++ b/runbooks/kubevirt-launch-failed.md @@ -0,0 +1,173 @@ +# kubevirt-launch-failed — One or more VirtualMachineInstances are in the Failed phase + +**Alert:** `KubeVirtVMLaunchFailing` (critical, 5m) +**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt` +**Expression:** `sum by (namespace) (kubevirt_vmi_phase_count{phase="Failed"}) > 0` + +## What this means here + +A VMI reached the `Failed` phase — a scheduling, launch or qemu failure. The IaaS path is broken for that +workload, which is why it is page-worthy even though the VM tier is deliberately **non-gating** +(`apps-vms` is `wait: false`, `dependsOn: platform-config`, because a VM's image import and in-guest +install are legitimately long-not-Ready). + +What actually runs here: the `unifi` namespace holds the UniFi VM (rootdisk 40 GiB, Block mode on +`longhorn-vm`). **UniFi OS Server is not yet installed in the guest** — that is known, documented state, +not a launch failure. + +**Live note, measured 2026-09-13:** `kubevirt_vmi_phase_count` returned **0 series** on prod. That metric +only exists while VMIs do, so this alert is currently evaluating against nothing. Confirm the real state +from `kubectl get vmi -A`, not from this alert's silence. (`kubevirt-prometheus-metrics` *is* a live scrape +job and `kubevirt_virt_api_ready_status` does publish, so the target is up — see +`/runbooks/kubevirt-control-plane`.) + +### The three k3s-specific launch failures this platform already hit and fixed + +Recorded from bring-up, and worth checking against any new VM definition because all three were +non-obvious: + +1. **`containerDisk` requires the ALPHA `ImageVolume` feature gate**, which is off on k3s. The fix was to + switch to a **CDI DataVolume on Longhorn** instead. A VM authored with `containerDisk` will not launch + here. +2. **A UEFI image needs `firmware.efi`** set on the VM. Without it, boot fails. +3. **`useEmulation: true` silently forces TCG on KubeVirt v1.9** — it is set to `false` deliberately. A VM + that appears to launch but burns a whole core per vCPU is this; it also degrades the Cilium datapath + under sustained load. + +Also recorded: a `virt-handler` certificate error seen during that work was **unrelated and cosmetic** — +do not chase it. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which VMI, in which namespace, and what phase: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get vmi -A -o wide; echo; \ + kubectl --context k3s-prod get vm -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,STATUS:.status.printableStatus,READY:.status.ready' +``` + +The reason — the VMI status conditions and the launcher pod's events are where it is stated: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n unifi describe vmi unifi 2>&1 | sed -n "/Status:/,\$p" | head -60' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n unifi get pods -l kubevirt.io=virt-launcher -o wide; echo; \ + kubectl --context k3s-prod -n unifi describe pod -l kubevirt.io=virt-launcher 2>&1 | sed -n "/Events:/,\$p"' +``` + +Is the disk there? (A missing or still-importing DataVolume is the most common cause.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get datavolumes.cdi.kubevirt.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PHASE:.status.phase,PROGRESS:.status.progress; echo; \ + kubectl --context k3s-prod get pvc -A | grep -Ei "rootdisk|dv|unifi"' +``` + +Is nested virtualisation actually available on the node it was scheduled to? + +``` +ssh k3s-prod-1 'ls -l /dev/kvm; grep -c -E "vmx|svm" /proc/cpuinfo' +``` + +Is the storage class right for a live-migratable VM? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get sc; echo; \ + kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy' +``` + +And whether the KubeVirt control plane can act at all: + +``` +infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kubevirt get pods' +``` + +## Likely causes, most likely first + +### 1. The disk is not ready — a CDI import that has not completed or has stalled +The VM cannot start without its DataVolume. Go to `/runbooks/cdi-import-stuck`. The specific recorded +failure: the **3-replica default class failed the UniFi 60Gi import with insufficient storage / +`ReplicaSchedulingFailure`**, which is why a dedicated single-replica **`longhorn-scratch`** class exists +and is referenced by the CDI CR's `scratchSpaceStorageClass`. + +### 2. The VM is authored in a way k3s does not support +The three items above: `containerDisk` without the ALPHA `ImageVolume` gate (use a CDI DataVolume), +missing `firmware.efi` for a UEFI image, or `useEmulation: true` forcing TCG. All are git changes to the +VM definition. + +### 3. The cloud-init payload is too large or corrupted +Two precise traps. **KubeVirt caps inline `userData` at 2048 bytes** — the original rich cloud-init +exceeded it; `userDataSecretRef` was tried and reverted, and the shipped state is a minimal ~1.1 KB inline +payload. And: **`apps-vms` must NOT carry `postBuild.substituteFrom`** — Flux's envsubst rewrote `${...}` +inside the cloud-init and corrupted it (commit `da1f833`). *Do not restore substitution there for +consistency with the app waves.* If a VM stopped booting right after someone "tidied up" the Flux +Kustomizations, this is it. + +### 4. It cannot be scheduled +No node with room, or a storage constraint. A live-migratable VM needs **`longhorn-vm`**: RWX, +**Block mode only** (migratable RWX is a raw block device, not the share-manager/NFS path), annotated +`storageclass.kubevirt.io/is-default-virt-class: "true"` — which is *not* the cluster default class. +A VM pointed at plain `longhorn` will not live-migrate. + +### 5. The CDI importer cannot open the raw Block device +`nonroot-devices: true` is set on the k3s servers because the non-root CDI importer must open a raw Block +device that Kubernetes never chowns for it (commit `df5979f`). A permission error in the importer on a +Block PVC is this. + +### 6. The KubeVirt control plane is down +Then nothing can launch. `/runbooks/kubevirt-control-plane`. + +## When to escalate or stop + +- **Scope it before escalating.** The VM tier is non-gating and does not affect the container platform. + One failed VM is not a platform outage — but the UniFi VM is the home network's controller, so treat + that one as user-visible. +- **Do not delete the VM or its DataVolume to "start clean".** A 60 GiB re-import is hours of I/O on disks + that share a partition with the embedded etcd WAL, and that is the mechanism behind the 2026-09-07 node + outage. Fix the definition, not the data. +- **Do not set `evictionStrategy: LiveMigrate` cluster-wide.** A blanket setting would **wedge drains** + for RWO-disk VMs; it is per-VM on purpose. +- **Do not add `postBuild.substituteFrom` to `apps-vms`.** It corrupts cloud-init. +- Escalate for anything that needs a fresh image import during a storage incident, and for changes to the + k3s server flags (`nonroot-devices`) — those are Ansible/operator work. + +--- +Source of truth: `runbooks/kubevirt-launch-failed.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` Session 7 (the three k3s-specific launch failures); commits `da1f833` +(cloud-init corrupted by envsubst) and `df5979f` (`nonroot-devices`); the `longhorn-scratch` / +UniFi 60Gi import finding. diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml new file mode 100644 index 0000000..2682e79 --- /dev/null +++ b/runbooks/kustomization.yaml @@ -0,0 +1,86 @@ +--- +# Runbooks tier — reconciled by its OWN non-gating post-platform Flux Kustomization +# (clusters/base/runbooks.yaml, dependsOn platform-config, wait:false). Same placement and rationale as +# platform/homepage: a leaf web server must never sit in the gating `platform` tier, where a UI's +# unhealth once stalled the whole chain (DECISIONS-LOG: "Headlamp gating stall"). +# +# WHY THE MANIFESTS LIVE HERE, ALONGSIDE THE DOCS, RATHER THAN IN platform/runbooks/. +# The markdown is the source of truth and the task is to serve it verbatim, which means the ConfigMap must +# be GENERATED from these files — there must not be a second hand-maintained copy of 34 documents that can +# drift from the originals. `configMapGenerator` does that, and its `files:` paths are resolved relative to +# THIS directory. A `platform/runbooks/kustomization.yaml` would therefore need `../../runbooks/.md`, +# which depends on kustomize load restrictions being relaxed — true for Flux's kustomize-controller, +# refused by a plain `kustomize build`, and so a divergence between what the gate can verify and what the +# cluster does. Making this directory the Flux path removes the question entirely: no `../`, identical +# behaviour locally and in-cluster, one copy of every document. The serving manifests are tucked under +# `serving/` so `runbooks/*.md` stays exactly the flat set of pages the URLs name. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: runbooks +resources: + - serving/namespace.yaml + - serving/nginx-config.yaml + - serving/deployment.yaml + - serving/service.yaml + # default-deny baseline + Cilium gateway-ingress allow (DNS-only egress from the baseline). + - serving/networkpolicy.yaml + - serving/httproute.yaml # apex `/runbooks` prefix (binds the Gateway `https-apex` listener) + +configMapGenerator: + # ONE ConfigMap holding every runbook, keyed by the EXACT name the alert annotations use. + # + # THE KEYS ARE EXTENSIONLESS ON PURPOSE. The annotations are `https://${BASE_DOMAIN}/runbooks/` + # with NO `.md`, and this repo must not change them (they are spread across four alerts.yaml files that + # other work is editing). `=` maps `node-down.md` on disk to the key `node-down`, the volume + # mount turns each key into a file under the nginx root, and `/runbooks/node-down` becomes a plain static + # lookup — no rewrite rule, nothing to get subtly wrong at 3am. Drop the `=` and every key would + # carry `.md`, so every alert link would 404 while this file still looked correct. + # + # The generated name carries kustomize's content hash, so editing any runbook rolls the server + # automatically and a stale projection is impossible. + # + # ADDING A RUNBOOK: add the file AND its line here. The L0 check + # conformance/tests/l0-static/alert-runbooks-resolvable asserts both halves, because a file in git with + # no generator entry is reviewed, merged, apparently done — and still a 404 in the cluster. + - name: runbooks-content + options: + labels: + app.kubernetes.io/name: runbooks + files: + # The index, served at /runbooks/ (nginx `try_files /runbooks/README`). + - README=README.md + - backup-failure=backup-failure.md + - backup-missed=backup-missed.md + - backup-partial=backup-partial.md + - cdi-import-stuck=cdi-import-stuck.md + - cert-expiry=cert-expiry.md + - cert-issuance=cert-issuance.md + - cnpg-archiving=cnpg-archiving.md + - cnpg-backup-missed=cnpg-backup-missed.md + - cnpg-exporter=cnpg-exporter.md + - cnpg-primary-spread=cnpg-primary-spread.md + - cnpg-wal-growth=cnpg-wal-growth.md + - denied-flows=denied-flows.md + - flux-reconcile=flux-reconcile.md + - flux-suspended=flux-suspended.md + - image-automation=image-automation.md + - image-policy-stalled=image-policy-stalled.md + - keycloak-client-secrets=keycloak-client-secrets.md + - keycloak-db=keycloak-db.md + - keycloak-down=keycloak-down.md + - kubevirt-control-plane=kubevirt-control-plane.md + - kubevirt-crashloop=kubevirt-crashloop.md + - kubevirt-launch-failed=kubevirt-launch-failed.md + - longhorn-capacity=longhorn-capacity.md + - longhorn-degraded=longhorn-degraded.md + - longhorn-faulted=longhorn-faulted.md + - monitoring-capacity=monitoring-capacity.md + - node-cpu=node-cpu.md + - node-disk=node-disk.md + - node-down=node-down.md + - node-io=node-io.md + - node-memory=node-memory.md + - pvc-capacity=pvc-capacity.md + - targets-down=targets-down.md + - update-rollback=update-rollback.md + - vpn-down=vpn-down.md diff --git a/runbooks/longhorn-capacity.md b/runbooks/longhorn-capacity.md new file mode 100644 index 0000000..9258f12 --- /dev/null +++ b/runbooks/longhorn-capacity.md @@ -0,0 +1,158 @@ +# longhorn-capacity — A Longhorn volume or node is near its storage capacity + +**Alerts:** `LonghornVolumeCapacityHigh` (warning, 30m) and `LonghornNodeStorageHigh` (warning, 30m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn` +**Expressions:** +`(longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85` +`(longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85` + +## What this means here + +Either one volume is 85% full, or one node's Longhorn disk is 85% allocated. They are different +problems with different fixes, so read the labels: `volume` means a workload is filling its own PVC; +`node` means the cluster is running out of places to put replicas. + +The node case is bounded by settings that are already conservative: +`storageReservedPercentageForDefaultDisk: 20`, `storageMinimalAvailablePercentage: 25`, +`storageOverProvisioningPercentage: 200`, `defaultDataLocality: disabled`, hard node anti-affinity. +Measured headroom on prod is roughly **269 GiB scheduled against an 829 GiB ceiling**, so the node +alert firing means something has changed materially, not that the platform has slowly grown. + +The severe consequence of running out of node storage is that a degraded volume **cannot finish +rebuilding** — hard anti-affinity needs three distinct nodes with room, so one full node pins a +volume at 2 replicas indefinitely. Cross-read `/runbooks/longhorn-degraded`. +### Verify the alert can even fire (measured 2026-09-13) + +Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`, +`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`, +`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`, +`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`, +`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`. +There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this +rule depends on before you trust its silence: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \ + sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query' +``` + +**These alerts are currently INERT.** `count(longhorn_volume_capacity_bytes)` and +`count(longhorn_node_storage_capacity_bytes)` both returned **0 series** on prod on 2026-09-13 — +Longhorn ships no ServiceMonitor here. Volume-fill coverage actually comes from +`kubelet_volume_stats_*` instead, i.e. `/runbooks/pvc-capacity`, which is where the working alerts +are. Use this runbook for the Longhorn-side (node allocation) half, and expect to reach it by hand. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Node-level allocation (the half that has no substitute alert): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o yaml | grep -E "name:|storageAvailable|storageMaximum|storageScheduled|allowScheduling"' +``` + +Volume sizes versus actual usage: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ + -o custom-columns=NAME:.metadata.name,SPEC:.spec.size,ACTUAL:.status.actualSize,ROBUST:.status.robustness' +``` + +Which PVC each one is, so you know who is filling it: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SIZE:.spec.capacity.storage,SC:.spec.storageClassName' +``` + +The filesystem view, which is the alert that actually works: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s -G --data-urlencode "query=topk(10, 1 - kubelet_volume_stats_available_bytes{job=\"kubelet\"} / kubelet_volume_stats_capacity_bytes{job=\"kubelet\"})" localhost:19090/api/v1/query' +``` + +On the node itself: + +``` +ssh k3s-prod-1 'df -h /data; du -xh --max-depth=1 /data/longhorn/replicas 2>/dev/null | sort -h | tail' +``` + +## Likely causes, most likely first + +### 1. A CNPG database is not recycling WAL (check this FIRST) +The only cause here that grows without bound. `archive_mode=on` plus a failing `archive_command` means +Postgres will not recycle unarchived WAL; measured on prod `keycloak-db` at ~6.8 MiB/h with `pg_wal` +sharing the data PVC (no `walStorage:` is configured on any Cluster) — about 29 days from 93% free to +disk-full, ending in a postmaster PANIC. Go to `/runbooks/cnpg-archiving`. **Never delete WAL by +hand.** + +### 2. Snapshots and orphaned replicas accumulating +Longhorn snapshots are not free, and cleanup here is **manual**: `orphanAutoDeletion: true` in +`platform/longhorn/helmrelease.yaml` has been **silently inert since Longhorn 1.12** — it was renamed +`orphanResourceAutoDeletion` (now a list) and the chart drops unknown `defaultSettings` keys with no +error. Verify the key's current state in that file before citing it; then prune orphans and stale +snapshots from the Longhorn UI. + +### 3. A workload genuinely grew +Plex/Sonarr/Radarr **media** does not live on Longhorn — it is static NFS from +`192.168.2.102:/mnt/Storage/plex`, via the `bulk` tier. If a media volume is on `longhorn`, that is a +misconfiguration worth fixing, not a capacity problem: the `bulk` StorageClass is a stub +(`provisioner: nfs.csi.k8s.io`, `server:/share: REPLACE-ME`) and provisions nothing, so real bulk +storage is static NFS PVs. Otherwise expand the PVC — all Longhorn classes have +`allowVolumeExpansion: true`; change the size in git and push. + +### 4. Replicas piled onto one node +`replicaAutoBalance: best-effort` should prevent this, and hard anti-affinity bounds it to one replica +per volume per node. If one node is much fuller than the others, check whether a rebuild placed +replicas there while another node was down. + +### 5. The cluster has outgrown its disks +Prod's ceiling is ~829 GiB across three nodes. `docs/plans/longhorn-ssd-tiering.md` is the planned +answer (a ~51 GB hot tier: Prometheus, CNPG, Loki, Gitea). Operator planning work. + +## When to escalate or stop + +- **A node over 85% while any volume is degraded is urgent**, because the rebuild cannot complete. + Free space or restore the other node before anything else. +- **Never delete files under `/data/longhorn`** or inside a Postgres `pg_wal`. Both are irreversible. +- **Do not expand a PVC as a reflex** if the cause is unrecycled WAL — you are buying days and hiding + the real fault. +- Escalate for adding disks, the SSD tiering plan, or anything that needs a node reboot (each reboot + degrades all ~18 volumes and triggers a rebuild storm). + +--- +Source of truth: `runbooks/longhorn-capacity.md`. Index: `/runbooks/` +Records: `platform/longhorn/helmrelease.yaml` (the settings above and the inert `orphanAutoDeletion`); +`platform/cnpg/alerts.yaml` header (the WAL disk-full clock); `docs/BULK-NFS-STORAGE.md`. diff --git a/runbooks/longhorn-degraded.md b/runbooks/longhorn-degraded.md new file mode 100644 index 0000000..803ddb1 --- /dev/null +++ b/runbooks/longhorn-degraded.md @@ -0,0 +1,170 @@ +# longhorn-degraded — A Longhorn volume has been degraded for 10 minutes + +**Alert:** `LonghornVolumeDegraded` (warning, 10m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn` +**Expression:** `longhorn_volume_robustness == 2` (0=unknown 1=healthy 2=degraded 3=faulted) + +## What this means here + +A volume has fewer healthy replicas than it should. It is **still serving I/O** — that is the +difference between degraded and faulted (`/runbooks/longhorn-faulted`). What you have lost is +redundancy, and on a 3-node cluster with hard node anti-affinity +(`replicaSoftAntiAffinity: false`, `numberOfReplicas: 3`) one degraded volume means one more node +loss away from data loss. + +**Degraded is the NORMAL state after any node reboot on this cluster, and it is expensive.** +Recorded plainly in the stabilization notes: *"Longhorn on the HDD: every node reboot degrades all 18 +volumes → full rebuild (slow; 30%+ IO-delay). Roll reboots one node at a time; wait +`robustness=healthy` before the next."* The rebuild is the thing that hurts, not the degradation: +rebuild writes land on the same partition as the embedded etcd WAL (`/data/k3s` and `/data/longhorn` +share one device), and that is the mechanism behind the 2026-09-07 `k3s-prod-1` NotReady outage. So +the question to answer is never just "why is it degraded" but "is the rebuild about to take a node +out". Cross-read `/runbooks/node-io`. +### Verify the alert can even fire (measured 2026-09-13) + +Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`, +`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`, +`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`, +`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`, +`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`. +There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this +rule depends on before you trust its silence: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \ + sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query' +``` + +**This alert is currently INERT.** `count(longhorn_volume_robustness)` returned **0 series** on prod +on 2026-09-13: the whole `platform-health.longhorn` rule group has no data, because Longhorn ships no +ServiceMonitor here. That is a separate open defect, and the practical consequence is that **you will +probably learn about a degraded volume from an application symptom or from `volumes.longhorn.io`, not +from this alert**. Do not infer "no alert, so volumes are fine". +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +The truth about robustness, straight from the CRs (this works whether or not the metric does): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ + -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state,NODE:.status.currentNodeID,SIZE:.spec.size' +``` + +Which replicas are missing, and where: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get replicas.longhorn.io \ + -o custom-columns=NAME:.metadata.name,VOL:.spec.volumeName,NODE:.spec.nodeID,RUNNING:.status.currentState | sort -k2' +``` + +Which PVC/workload is affected: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SC:.spec.storageClassName' +``` + +Are the nodes and their disks schedulable, and is a rebuild in flight? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o wide; echo; \ + kubectl --context k3s-prod -n longhorn-system get settings.longhorn.io concurrent-replica-rebuild-per-node-limit -o custom-columns=NAME:.metadata.name,VALUE:.value' +``` + +And the pressure that rebuild is causing — the reason this alert matters: + +``` +ssh k3s-prod-1 'cat /proc/pressure/io' +``` + +## Likely causes, most likely first + +### 1. A node rebooted or was briefly NotReady (overwhelmingly the most common) +Expected, and it self-heals by rebuilding. Your job is to keep the rebuild from taking a node down. +Remediation (operator, mutating) — freeze rebuilds while replicas will **reattach** (an in-place +bounce): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \ + concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"' +``` + +Thaw to `"1"` — never higher — once every volume reads `healthy`. Use `1` rather than `0` when the +replicas are genuinely gone (an OS re-provision), so replenishment can actually proceed. +`concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and +held**; do not "fix" this by setting a knob that is already set. `replicaRebuildingBandwidthLimit` is +**v2/SPDK-only** and does nothing here; `rebuildConcurrentSyncLimit` is already at its floor of 1. +What was actually tightened after that incident was every *other* Longhorn I/O source: +`concurrentVolumeBackupRestorePerNodeLimit` and `snapshotHeavyTaskConcurrentLimit` 5 → 1. + +### 2. A node is still down +Then the rebuild cannot complete and this will not clear. Go to `/runbooks/node-down` first; there is +nothing to fix in Longhorn. + +### 3. A disk was replaced and Longhorn refuses it +Signature: `DiskFilesystemChanged` — *"record diskUUID doesn't match the one on the disk"*. +Remediation, in exactly this order (the validating webhook enforces it): **disable the disk → remove +the disk entry → re-add it**. Seen during the 2026-09-07 drive swap. + +### 4. No node can hold the replica (anti-affinity or space) +Hard anti-affinity means a 3-replica volume needs three distinct schedulable nodes. With one node +cordoned or its disk over `storageMinimalAvailablePercentage: 25`, the third replica cannot be +placed and the volume stays degraded indefinitely. Check `nodes.longhorn.io` scheduling and +`/runbooks/longhorn-capacity`. + +### 5. `replicaAutoBalance: best-effort` is shuffling replicas +Normal background behaviour, transient. If degradation follows a pattern of replicas moving and never +settling, that is worth an operator's attention. + +## When to escalate or stop + +- **If more than one volume is degraded and I/O-PSI on any node is climbing, freeze rebuilds first + and then get the operator.** That combination is the 2026-09-07 failure mode in progress. +- **Never reboot, cordon or re-provision a second node while any volume is degraded.** This is the + single most important rule in this runbook. +- **Do not `kubectl drain`.** This cluster deliberately does not drain: there is no Longhorn + `nodeDrainPolicy` configured anywhere in the repo, the upgrade plan rejects + system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD Longhorn", and the + node-loss conformance test evicts only its own writer pod. +- **Do not delete a degraded volume's replicas to "force a clean rebuild"** unless the operator agrees + — with 3 replicas and one already missing you are one mistake from `/runbooks/longhorn-faulted`. +- Escalate for: a node that will not come back, a disk replacement, or a volume degraded for more than + a few hours with all three nodes Ready (that is a Longhorn bug, not an operational state). + +--- +Source of truth: `runbooks/longhorn-degraded.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` "2026-08-31 — Post-upgrade stabilization" (the reboot tax) and +"2026-09-07 — k3s-prod-1 NotReady"; `docs/plans/cluster-upgrade-plan.md` §6.2/§7 (the freeze/thaw). diff --git a/runbooks/longhorn-faulted.md b/runbooks/longhorn-faulted.md new file mode 100644 index 0000000..7a3a5d0 --- /dev/null +++ b/runbooks/longhorn-faulted.md @@ -0,0 +1,164 @@ +# longhorn-faulted — A Longhorn volume is FAULTED (data at risk, volume offline) + +**Alert:** `LonghornVolumeFaulted` (critical, 1m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn` +**Expression:** `longhorn_volume_robustness == 3` + +## What this means here + +Faulted means Longhorn has **no usable replica**. The volume is offline; the workload's writes are +failing. This is the most serious storage state on the cluster and the `for: 1m` is deliberate — there +is no benign transient version of this. + +Understand what is and is not recoverable before you touch anything: + +- The volumes on this cluster are 3-replica with **hard** node anti-affinity, so faulted normally + means two or three replicas are gone at once — two nodes lost, or one node lost while another + volume's rebuild had not finished, or a disk that ate its replica. +- **Standbys and replicas are not backups.** Recorded verbatim for the databases and true for all of + this: *"Standbys replicate corruption and deletions faithfully; only the Barman archive is a + backup."* For a CNPG volume (`keycloak-db`, `nextcloud-db`, `semaphore-db`) the real recovery path + is the object-store archive, not Longhorn. For everything else it is Velero. +- If this is a **`longhorn-vm`** volume (RWX Block, `migratable: true`) the VM is down, not just the + disk. If it is **`longhorn-scratch`** (single replica, CDI scratch space) losing it is expected and + harmless — re-run the import. +### Verify the alert can even fire (measured 2026-09-13) + +Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`, +`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`, +`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`, +`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`, +`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`. +There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this +rule depends on before you trust its silence: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \ + sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query' +``` + +**This alert is currently INERT.** `count(longhorn_volume_robustness)` returned **0 series** on prod +on 2026-09-13 — Longhorn ships no ServiceMonitor here, so the whole `platform-health.longhorn` group +has no data. A faulted volume today announces itself as an application outage. That is a known open +defect; treat `volumes.longhorn.io` as the source of truth, not this alert's silence. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Stop and take inventory before changing anything. Which volume, which workload, which replicas exist: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ + -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state,NODE:.status.currentNodeID | grep -Ev "healthy"' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get replicas.longhorn.io \ + -o custom-columns=NAME:.metadata.name,VOL:.spec.volumeName,NODE:.spec.nodeID,STATE:.status.currentState,FAILED:.spec.failedAt | sort -k2' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SC:.spec.storageClassName | grep ' +``` + +What recovery material exists — **fully-qualified names, or you will be told there is nothing**: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get backups.velero.io -A; echo; \ + kubectl --context k3s-prod get backups.postgresql.cnpg.io -A; echo; \ + kubectl --context k3s-prod -n longhorn-system get backups.longhorn.io' +``` + +`kubectl get backup -A` prints "No resources found" on this cluster even when six Velero and 27 CNPG +backups exist. Never make a recovery decision from the short name. + +Node and disk state: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get nodes; kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o wide' +``` + +## Likely causes, most likely first + +### 1. Two nodes were unavailable at once +Usually a second reboot or cordon while the first node's rebuilds were still running. Remediation: +bring the nodes back. Longhorn will reattach if the replica data survived on disk — which is why you +must not delete replicas before checking. + +### 2. A disk failed and took its replica with it +`k3s-prod-1`'s original 2011 SSD had 61,440 reallocated sectors before it was replaced; the dev-era +Seagate had 448 current-pending plus 448 offline-uncorrectable. Check SMART on the node holding the +failed replica (`/runbooks/node-io` has the command). If the disk is bad, the replica is gone — the +path forward is restore, not repair. + +### 3. All replicas were evicted or deleted +Check whether a disk was disabled, a node removed from Longhorn, or a `replicas.longhorn.io` object +deleted. `staleReplicaTimeout: 30` (minutes) means Longhorn eventually discards replicas it considers +stale. + +### 4. It is a single-replica volume by design +`longhorn-scratch` is `numberOfReplicas: 1`. Losing it is not an incident — delete the DataVolume and +re-import. + +## Remediation, in order + +1. **Do not delete the volume, the PVC, or any replica.** A faulted volume with a stale replica is + recoverable; a deleted one is not. +2. If a node is down, restore the node (`/runbooks/node-down`). That is the highest-probability fix. +3. If Longhorn shows a replica with data but marked failed, an operator can salvage it from the + Longhorn UI (*Volume → Salvage*) — this is the documented path for "all replicas failed but the + data is still there". +4. If the volume backs a **CNPG database**, do not try to salvage Postgres files. Recover the Cluster + from the Barman archive (`/runbooks/cnpg-archiving` for how to confirm the archive is real first). +5. Otherwise restore from Velero. Note honestly: `dr-restore` is **authored and has never been + executed** (`conformance.home.k3s/never-executed`), and the R2 restore path has not been rehearsed. + Expect to be the first person to run it, and read `/runbooks/backup-failure` for the R2 plugin + caveats before you rely on a backup's metadata. + +## When to escalate or stop + +- **Get the operator now.** This is a critical, data-at-risk alert; it is not a runbook-and-carry-on + situation. Everything above is diagnosis, and steps 3–5 are operator actions. +- **Stop before any destructive command.** Specifically: no `delete pvc`, no `delete + volumes.longhorn.io`, no `delete replicas.longhorn.io`, no touching files under `/data/longhorn`. +- **Do not reboot anything** while a volume is faulted. +- If the affected volume is `keycloak-db`, expect a full SSO outage as a knock-on: kubectl OIDC, + Grafana, Headlamp, Gitea and forward-auth all depend on it. The certificate-based kubeconfig is an + independent break-glass path and still works. + +--- +Source of truth: `runbooks/longhorn-faulted.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` (drive failures 2026-08-22 and 2026-09-07; `dr-restore` never executed). diff --git a/runbooks/monitoring-capacity.md b/runbooks/monitoring-capacity.md new file mode 100644 index 0000000..3a56dad --- /dev/null +++ b/runbooks/monitoring-capacity.md @@ -0,0 +1,133 @@ +# monitoring-capacity — The monitoring stack's own PVC is filling up + +**Alert:** `MonitoringVolumeFillingUp` (warning, 15m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.monitoring-self` +**Expression:** `kubelet_volume_stats_available_bytes{namespace="monitoring"} / kubelet_volume_stats_capacity_bytes{namespace="monitoring"} < 0.15` + +## What this means here + +"Don't let the monitor blind itself." Prometheus or Loki is running out of disk. If it fills, you lose +the ability to see every other alert in this set — which is why `monitoring` gets its own rule instead +of sharing `/runbooks/pvc-capacity` (the two files partition the namespace space between them; the +thresholds are deliberately identical, 15% free for 15m). + +Live shape on prod: + +- `prometheus-...-prometheus-0` PVC **50Gi**, `retention: 30d`, `retentionSize: 45GiB` +- `storage-loki-0` PVC **50Gi** +- both on the `longhorn` class, 3 replicas + +`retentionSize: 45GiB` against a 50Gi PVC is intentional: Prometheus is *supposed* to plateau near the +high-water mark and then stay there. **So a Prometheus TSDB approaching 45 GiB is normal, not a leak.** +That is also why the predictive rule in `pvc-capacity-alerts.yaml` deliberately **excludes** the +`monitoring` namespace: a TSDB filling toward its retention limit looks exactly like an unbounded leak +to `predict_linear` until it plateaus. + +So the real question is *which* PVC and *whether it should be plateauing*: read the +`persistentvolumeclaim` label first. Loki has no size cap analogous to `retentionSize`, so a log +volume growing past 85% is the more likely genuine problem of the two. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring get pvc; echo; \ + kubectl --context k3s-prod -n monitoring get prometheus -o jsonpath="{.items[0].spec.retention} {.items[0].spec.retentionSize}{\"\n\"}"' +``` + +Actual on-disk usage from inside the pods: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- df -h /prometheus; \ + kubectl --context k3s-prod -n monitoring exec loki-0 -- df -h /var/loki' +``` + +Is Prometheus at its retentionSize (normal) or past it (a problem)? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + for q in "prometheus_tsdb_storage_blocks_bytes" "prometheus_tsdb_head_series" "rate(prometheus_tsdb_compactions_failed_total[1h])"; do \ + printf "%s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]])"; done' +``` + +Series growth is the usual driver — find the biggest contributors: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s "localhost:19090/api/v1/status/tsdb" | python3 -m json.tool | head -40' +``` + +## Likely causes, most likely first + +### 1. Prometheus has reached its retention high-water mark (expected) +`retentionSize: 45GiB` on a 50Gi PVC leaves 10% headroom, which is close to the 15% trigger. If +`prometheus_tsdb_storage_blocks_bytes` is near 45 GiB and steady, **this is the design working** and +the correct response is either to accept it or to raise both the PVC and `retentionSize` together in +`platform/monitoring/kube-prometheus-stack.yaml`. Raising retention without raising the PVC is how you +get a genuinely full disk. + +### 2. Loki log volume growth +Loki has no retentionSize equivalent configured. A noisy workload (a crashlooping pod, a chatty +reconciler) can grow it quickly. Remediation: find the noisy stream in Grafana → Explore, fix the +source, and set/lower Loki retention in git. + +### 3. Series cardinality grew after a chart or workload change +`serviceMonitorSelectorNilUsesHelmValues` and `podMonitorSelectorNilUsesHelmValues` are both `false` +here, which means **Prometheus selects every ServiceMonitor/PodMonitor in the cluster regardless of +labels**. That is convenient (no `additionalLabels` needed anywhere) and it also means a new +component's monitor is picked up automatically, cardinality and all. Check +`/api/v1/status/tsdb` for the top label pairs. + +### 4. Compaction is failing +`rate(prometheus_tsdb_compactions_failed_total[1h]) > 0` means blocks are not being merged or expired. +Usually an I/O problem underneath — cross-read `/runbooks/node-io`. + +### 5. The volume is on a node that is itself short of space +`/runbooks/longhorn-capacity` and `/runbooks/node-disk`. + +## When to escalate or stop + +- **Do not delete Prometheus TSDB blocks or Loki chunks by hand.** Lower retention and let the + component expire them; hand-deleting blocks can leave the TSDB unopenable. +- **Do not raise `retention` or `retentionSize` without raising the PVC in the same change.** +- If Prometheus is already full and crashlooping, you are blind: check the alerts you care about + directly against the CRs (`clusters.postgresql.cnpg.io`, `volumes.longhorn.io`, `kubectl get nodes`) + rather than waiting for alerts that cannot be evaluated. +- Escalate for PVC expansion — it needs node headroom, and this cluster has been memory- and + disk-constrained before (Prometheus was already raised 2 → 3Gi of RAM after the kps 65→88 upgrade). + +--- +Source of truth: `runbooks/monitoring-capacity.md`. Index: `/runbooks/` +Records: `platform/monitoring/pvc-capacity-alerts.yaml` (why monitoring is excluded from the +predictive rule); `DECISIONS-LOG.md` (Grafana OOM / Prometheus 2→3Gi during kps 65→88). diff --git a/runbooks/node-cpu.md b/runbooks/node-cpu.md new file mode 100644 index 0000000..3a5e4a6 --- /dev/null +++ b/runbooks/node-cpu.md @@ -0,0 +1,125 @@ +# node-cpu — CPU pressure or run-queue depth high on a node + +**Alerts:** `NodeCPUPressureHigh` (warning, 15m) and `NodeCPULoadHigh` (warning, 15m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes` +**Expressions:** +`rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.15` +`node_load5 / count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1` + +## What this means here + +`NodeCPUPressureHigh` is PSI "some" for CPU: the fraction of wall-clock time during which at least +one task was stalled waiting for a CPU. Under 5% is healthy. Averaged CPU% — what `kubectl top` +shows — **hides this entirely**, which is why the alert uses PSI: it is the metric that surfaces the +"spikes cause blips of delay" symptom operators actually notice. + +`NodeCPULoadHigh` is 5-minute load average divided by core count. Sustained above 1 means the run +queue is longer than the number of CPUs. + +**Read them together.** CPU-PSI high *and* load high = genuine CPU contention. Load high *and* CPU-PSI +normal = the queue is full of tasks blocked on **I/O**, not CPU — go to `/runbooks/node-io`, which on +this cluster is the more dangerous of the two (etcd WAL and Longhorn replicas share one partition). + +Measured history: steady CPU PSI ran **15–23%** with load at or above core count, which is what drove +the node resize from **6 to 8 vCPU**; afterwards PSI measured ~4.5–7.7%. So 0.15 is not an arbitrary +threshold, it is the level the cluster actually lived at before the resize. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Distinguish CPU contention from I/O contention — do this first: + +``` +ssh k3s-prod-1 'cat /proc/pressure/cpu; echo; cat /proc/pressure/io; echo; uptime; nproc' +``` + +Who is burning it: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod top nodes; echo; kubectl --context k3s-prod top pods -A --sort-by=cpu | head -20' +``` + +``` +ssh k3s-prod-1 'top -b -n1 | head -20' +``` + +Are there VMs, and are any of them emulated (the worst case)? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get vmi -A -o wide' +``` + +``` +ssh k3s-prod-1 'grep -c -E "vmx|svm" /proc/cpuinfo; ls -l /dev/kvm' +``` + +## Likely causes, most likely first + +### 1. It is actually I/O, not CPU +Most common misread. A node blocked on disk has a long run queue and normal CPU-PSI. Confirm with +`/proc/pressure/io` and go to `/runbooks/node-io`. On 2026-09-07 the node that died had I/O-PSI full +at 11.36 while CPU looked unremarkable. + +### 2. A KubeVirt VM running under TCG emulation +`useEmulation: true` **silently forces TCG on KubeVirt v1.9** — it is set to `false` on purpose. A VM +that lost `/dev/kvm` or was authored with emulation burns a whole core per vCPU. Heavy sustained TCG +load has also degraded the Cilium datapath into intermittent LB-VIP unreachability (cleared by a +restart), so an "intermittent networking" report alongside this alert is probably one cause, not two. +Check `grep -c -E "vmx|svm" /proc/cpuinfo` (nested KVM is verified working; Proxmox nodes use +`cpu_type=host`). + +### 3. A Longhorn rebuild's checksumming +Rebuilds are I/O-heavy but not free on CPU. Same remediation as `/runbooks/longhorn-degraded`: freeze +with `concurrent-replica-rebuild-per-node-limit` `0`, thaw to `1`. + +### 4. Prometheus rule evaluation or a Loki query +`PersistentVolumeFillingUpPredicted` runs `predict_linear` over a 24h window across every PVC; a +Grafana panel doing the same interactively is heavier still. Bursty, self-limiting, usually harmless. + +### 5. Genuine capacity +If nothing is anomalous the node is simply undersized again. The 6 → 8 vCPU resize is the precedent. +A bpg cores change **reboots the VM**, so roll **one node at a time, waiting for Longhorn +`robustness=healthy` between each**. + +## When to escalate or stop + +- **If I/O-PSI is also elevated, stop treating this as a CPU problem** and switch to + `/runbooks/node-io`. Wrong runbook is the main risk here. +- Alone and steady, this alert is a capacity signal — schedule it, do not page on it. That is why it + is a warning with a 15m `for`. +- **Do not drain or cordon** to shed CPU load; this cluster does not drain. +- Escalate to the operator for a vCPU change (it reboots the node) or to kill a VM you did not create. + +--- +Source of truth: `runbooks/node-cpu.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` (node vCPU 6→8 and the post-resize PSI measurements; KubeVirt +`useEmulation` finding). diff --git a/runbooks/node-disk.md b/runbooks/node-disk.md new file mode 100644 index 0000000..9c4b06c --- /dev/null +++ b/runbooks/node-disk.md @@ -0,0 +1,131 @@ +# node-disk — A node filesystem is filling up + +**Alert:** `NodeDiskFillingUp` (warning, 15m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes` +**Expression:** `node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{...} < 0.10` + +## What this means here + +Less than 10% free on a real filesystem on a node, for 15 minutes. Read the `mountpoint` label +first, because on these nodes it tells you which of two very different problems you have. + +**`/data` is the one that matters.** Both the embedded **etcd WAL** (`/data/k3s`) and **every +Longhorn replica** (`/data/longhorn`) live on that single partition — +`node_baseline_etcd_source: shared-data-partition`, not a dedicated device, and the exit condition +for splitting them is still open (`infra/PROD-PROVISIONING.md` §2). If `/data` fills: + +- etcd cannot write its WAL. A control-plane member with a full WAL partition does not degrade + gracefully; it stops, and this is a 3-member cluster. +- Longhorn stops being able to hold replicas and volumes go degraded, which triggers rebuilds, which + need *more* space. + +So `/data` at 10% free is not a warning, it is the last comfortable moment to act. + +**`/` filling up** is usually logs, container images or a stuck journal. Lower stakes, but there is +history: in the dev era the host root filled to ~99% because VM disks sat under a snapper-snapshotted +subvolume, and the hypervisor **paused the k3s VMs on I/O error**. Recovery was deleting 11 snapper +timeline snapshots plus `virsh destroy` on the paused VMs (which were holding rsync-unlinked inodes): +99% → 30%. + +Longhorn's own guard rails are already tight and are why this alert usually fires before anything +breaks: `storageReservedPercentageForDefaultDisk: 20`, +`storageMinimalAvailablePercentage: 25`, `storageOverProvisioningPercentage: 200`. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which filesystem, on which node: + +``` +ssh k3s-prod-1 'df -h -x tmpfs -x overlay; echo; du -xh --max-depth=1 /data 2>/dev/null | sort -h | tail -10; echo; du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -10' +``` + +What Longhorn thinks it has: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,SCHED:.spec.allowScheduling; echo; \ + kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,SIZE:.spec.size,ACTUAL:.status.actualSize,ROBUST:.status.robustness' +``` + +Journal and image bloat on `/`: + +``` +ssh k3s-prod-1 'sudo journalctl --disk-usage; sudo crictl images | wc -l; sudo du -sh /var/lib/rancher/k3s/agent/containerd 2>/dev/null' +``` + +## Likely causes, most likely first + +### 1. Longhorn replicas grew, or replicas landed here after a rebuild +Hard node anti-affinity (`replicaSoftAntiAffinity: false`) means one replica per node per volume, but +`replicaAutoBalance: best-effort` moves them. Prod sits around 269 GiB scheduled against an 829 GiB +ceiling, so this is rarely tight — but a rebuild storm plus snapshots can spike it. +Remediation: check for orphaned replicas and stale snapshots in the Longhorn UI. Note +`orphanAutoDeletion: true` in `platform/longhorn/helmrelease.yaml` **has been silently inert since +Longhorn 1.12** — it was renamed `orphanResourceAutoDeletion` (a list) and the chart drops unknown +`defaultSettings` keys with no error. So orphan cleanup is manual. Verify the key's current state in +the file before you trust it. + +### 2. A CNPG database is not recycling WAL +The single most dangerous cause, because it is unbounded. With `archive_mode=on` and a failing +`archive_command`, Postgres **refuses to recycle** unarchived WAL segments and `pg_wal` grows forever +— on prod's `keycloak-db` that was measured at ~6.8 MiB/h, 17 segments / ~268 MiB, about 29 days to +fill a 5 GiB PVC. `pg_wal` shares the data PVC (no `walStorage:` is configured on any Cluster). +Check `/runbooks/cnpg-archiving` **before** you free any space, and **never delete WAL by hand** — +that destroys the database. + +### 3. Journal / container images on `/` +`journalctl --vacuum-size=200M` and pruning unused images are safe. Do not prune images for +workloads that may need to restart while the registry is unreachable. + +### 4. A Velero or CDI staging write +Velero's kopia uploads and CDI's scratch space both stage locally. `longhorn-scratch` +(single-replica) exists precisely because the 3-replica default failed the UniFi 60Gi import on +insufficient storage. + +### 5. The disk is smaller than the platform now needs +`docs/plans/longhorn-ssd-tiering.md` holds the planned hot tier (~51 GB: Prometheus, CNPG, Loki, +Gitea). Working set measured ~79 GB against ~323 GB free on SSD, so the "won't fit" blocker was a +measurement artifact. This is operator planning work, not an incident fix. + +## When to escalate or stop + +- **If the mountpoint is `/data` and free space is under 5%, treat it as an imminent control-plane + outage** and get the operator. etcd and Longhorn are both on it. +- **Never delete anything under `/data/longhorn` or a Postgres `pg_wal` directory by hand.** Both are + irreversible. Free space elsewhere, or fix the cause. +- **Do not `kubectl drain`** the node to move load off it — this cluster does not drain. +- If the cause is a CNPG archive failure, this runbook is the wrong one: go to + `/runbooks/cnpg-archiving`, which is the actual fix. + +--- +Source of truth: `runbooks/node-disk.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` "Disk/drive incident + k3s relocated to NVMe (2026-08-22)"; +`platform/cnpg/alerts.yaml` header (the WAL disk-full clock). diff --git a/runbooks/node-down.md b/runbooks/node-down.md new file mode 100644 index 0000000..6974d70 --- /dev/null +++ b/runbooks/node-down.md @@ -0,0 +1,156 @@ +# node-down — A node's node-exporter is unscrapable (the node may be gone) + +**Alert:** `NodeExporterDown` (critical, 5m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes` +**Expression:** `up{job="node-exporter"} == 0` + +## What this means here + +Prometheus has not scraped node-exporter on that instance for 5 minutes. The alert is worded +carefully — it says the *exporter* is down — because on this cluster the two cases look identical +from Prometheus and have opposite urgency: + +- **The node is genuinely gone** (powered off, kernel panic, network loss). +- **The node is up and the kubelet stopped posting status.** This is what actually happened on + **2026-09-07 at 06:05:21Z**: `k3s-prod-1` went NotReady with "Kubelet stopped posting node status" + while the host was up, pingable and SSH-able. Root cause was disk-I/O starvation of the embedded + etcd, not a dead machine. If this alert is firing alongside `node-io`, read `/runbooks/node-io` + first — that is the cause and this is the symptom. + +**Why it is critical rather than a warning.** All three nodes are control-plane + etcd members +(`k3s-prod-1` .10, `k3s-prod-2` .11, `k3s-prod-3` .12, kube-vip API VIP `192.168.2.9`). Losing one +leaves a 2-member etcd with no fault tolerance: the next loss ends quorum and the API server with it. +Losing a node also degrades every Longhorn volume that had a replica there, and every degraded volume +starts rebuilding — which loads the same disks etcd fsyncs to. One node down is not a routine event +on a 3-node cluster; it is one step from an outage. + +Two node-specific roles matter when you read which node it is: +- `k3s-prod-3` is the **sole VPN egress gateway**. Cilium OSS has **no egress-gateway failover**, so + while that node is down, VPN-classed egress is an outage, not a degradation. See `/runbooks/vpn-down`. +- `k3s-prod-2` is the cluster **founder** (`[k3s_prod_primary]` in `inventory/hosts.ini`). Do not + re-provision it from the inventory without reading `/runbooks/node-io` — re-provisioning a node + still listed as primary founds a **second cluster**. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Is the node actually NotReady, or is only the exporter unscrapable? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get nodes -o wide; echo; kubectl --context k3s-prod -n monitoring get pods -l app.kubernetes.io/name=prometheus-node-exporter -o wide' +``` + +Is it the API server or the node? (If this hangs, you have a bigger problem than one node.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get --raw /readyz?verbose 2>&1 | tail -20' +``` + +Is the host alive at all? Prod is directly on `192.168.2.0/24` — no NAT hop: + +``` +ping -c3 192.168.2.10 ; ssh k3s-prod-1 'uptime; systemctl is-active k3s; cat /proc/pressure/io' +``` + +The three lines that distinguish "etcd starved" from "machine dead": + +``` +ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|Kubelet stopped|leader"' +``` + +Etcd membership and leader — do this before touching anything: + +``` +ssh k3s-prod-2 'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"' +``` + +What did the node take with it? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ + -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy; \ + kubectl --context k3s-prod get pods -A --field-selector spec.nodeName=k3s-prod-1 -o wide | head -30' +``` + +## Likely causes, most likely first + +### 1. Disk I/O starvation wedged the kubelet (host still up) +The signature is `slow fdatasync took=3.56s` / `apply request took too long took=8.30s` / +`etcdserver: too many requests` in `journalctl -u k3s`, with `/proc/pressure/io` full avg300 well +above 1. On 2026-09-07 node 1 read **11.36** against 0.75 on node 2 and 1.29 on node 3. +Remediation: follow `/runbooks/node-io`. Recovery that day required a manual +`systemctl restart k3s` on the node (operator, on-host), after which 18 Longhorn volumes rebuilt and +pressure only fell when the rebuilds finished. + +### 2. The node rebooted (planned or otherwise) +Check `uptime`. A reboot is not harmless here: **every node reboot degrades all ~18 Longhorn volumes +and triggers a full rebuild**. If a reboot is intentional, roll one node at a time and wait for +`robustness=healthy` on every volume before the next, and freeze rebuilds first (see +`/runbooks/longhorn-degraded`). Note two hardware quirks recorded in `infra/PROD-PROVISIONING.md` +§0b: `k3s-prod-2` **hangs on a warm reboot while staying powered**, and all three nodes carry the EFI +memattr kernel bug that once presented `k3s-prod-1` as having 2.6 GiB usable of 16 GB (needs kernel +≥ 6.13). + +### 3. node-exporter itself is unhealthy, node fine +If `kubectl get nodes` says Ready and the pod is CrashLoopBackOff or Pending, this is a monitoring +fault, not a node fault. Downgrade urgency, but do not ignore it — while that exporter is down you +are blind to `node-io`, `node-memory`, `node-disk` and `node-cpu` on that node, which is exactly the +blindness that let 2026-09-07 arrive as a surprise. Related: `/runbooks/targets-down`. + +### 4. Network loss to the node +`ping` fails, SSH fails. Check the switch and the node's `eno1` (prod-3's +`/etc/network/interfaces` also holds a commented-out wireless stanza — inert, but worth scrubbing). +A node reachable by ping but not by the API is usually Cilium, not the network: heavy sustained node +load has been observed to degrade the Cilium datapath into intermittent LB-VIP unreachability, +cleared by a restart. + +### 5. Hardware +`k3s-prod-1`'s original 2011 SSD had 61,440 reallocated sectors and was replaced 2026-09-07. Check +SMART (`/runbooks/node-io` has the command). Hardware is an operator job. + +## When to escalate or stop + +- **STOP if two nodes are affected.** Two of three etcd members down means no quorum, no API server, + and no GitOps. Nothing in this runbook applies; get the operator. +- **Do not reboot, drain or cordon a second node** while any Longhorn volume is not `healthy`. This + cluster deliberately does not `kubectl drain` — the auto-drain of system-upgrade-controller was + rejected precisely because it fights slow-disk Longhorn. +- **Do not delete the Node object** to "clean up". That triggers replica deletion in Longhorn. +- Escalate to the operator for: a power-cycle, a disk replacement, or re-provisioning a node (the + inventory's `[k3s_prod_primary]` trap above makes re-provisioning a two-person job). +- If the node returns but volumes stay degraded for more than an hour, go to + `/runbooks/longhorn-degraded`. + +--- +Source of truth: `runbooks/node-down.md` in `home-network-k3s-impl`. Index: `/runbooks/` +Incident record: `DECISIONS-LOG.md` "2026-09-07 — k3s-prod-1 NotReady"; `infra/PROD-PROVISIONING.md` §0b/§0c. diff --git a/runbooks/node-io.md b/runbooks/node-io.md new file mode 100644 index 0000000..6a85e04 --- /dev/null +++ b/runbooks/node-io.md @@ -0,0 +1,183 @@ +# node-io — Disk I/O pressure / starvation on a node + +**Alerts:** `NodeIOPressureHigh` (warning, 10m) and `NodeIOPressureCritical` (critical, 5m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes` +**Expression:** `rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05` (warning) / `> 0.10` (critical) + +## What this means here + +PSI **full** for I/O: the fraction of wall-clock time in which *every* runnable task on the node was +blocked on disk, i.e. the machine did no useful work. It is not "the disk is busy" — it is "the node +is stalled". + +This alert exists because of a real outage it would have caught and could not. On **2026-09-07 at +06:05:21Z, `k3s-prod-1` went NotReady** ("Kubelet stopped posting node status") while the host stayed +up, pingable and SSH-able. The k3s journal showed embedded etcd starved of disk: + +``` +slow fdatasync took=3.56s / 6.71s / 4.95s / 3.40s (etcd expects 1s) +apply request took too long took=8.30s (expects 100ms) +etcdserver: too many requests +Failed to check local etcd status for learner management: context deadline exceeded +``` + +`/proc/pressure/io` **full avg300** at the time: `k3s-prod-1` **11.36**, `k3s-prod-2` (PM9A1 NVMe) +**0.75**, `k3s-prod-3` (WD Blue) **1.29**, against a ~1% steady state. The stack alerted on CPU-PSI +and memory and had **no I/O alert at all**, so the first signal the operator got was a node already +out of the cluster. The thresholds here are those measurements, not guesses: 0.05 is 5x steady +state, 0.10 is the level node 1 actually held while etcd stalled. + +**Why this is self-reinforcing, and why it is urgent.** On every node the etcd WAL (`/data/k3s`) and +all Longhorn replicas (`/data/longhorn`) sit on **one partition** — `node_baseline_etcd_source: +shared-data-partition`, not a dedicated device, and §2's exit condition in +`infra/PROD-PROVISIONING.md` is still **open**. So: node stalls → drops out → its volumes go degraded +→ Longhorn rebuilds them → more writes to the same spindle etcd fsyncs to → worse. Recovery on +2026-09-07 needed a manual `systemctl restart k3s`, and I/O pressure only fell when the 18 rebuilds +finished. + +**There is no etcd fsync metric to check.** `etcd_disk_wal_fsync_duration_seconds` is not collected — +`kubeEtcd` is deliberately disabled in `platform/monitoring/kube-prometheus-stack.yaml` for k3s. PSI +is the only signal that exists. Do not go looking for the etcd dashboard; it is not there. + +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which node, and how bad: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get nodes -o wide' +``` + +Read PSI and per-device queue depth on the node itself (SSH direct — prod is on +`192.168.2.0/24` with no NAT hop; `.10` = prod-1, `.11` = prod-2, `.12` = prod-3): + +``` +ssh k3s-prod-1 'cat /proc/pressure/io; echo; iostat -x 1 3; echo; lsblk -o NAME,MOUNTPOINT,ROTA' +``` + +Is etcd already stalling? This is the line that decides urgency: + +``` +ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|leader"' +``` + +Is Longhorn rebuilding (the usual writer)? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ + -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,NODE:.status.currentNodeID | grep -v healthy' +``` + +Is the disk itself dying? (This is what it was in September.) + +``` +ssh k3s-prod-1 'sudo smartctl -a /dev/sda | grep -Ei "Reallocated|Pending|Uncorrectable|Wear|Power_On_Hours|Model"' +``` + +## Likely causes, most likely first + +### 1. A Longhorn replica rebuild storm (most common by far) +Every node reboot degrades all ~18 volumes and triggers a full rebuild. That is the normal cost of +Longhorn on these disks, and it is why reboots are rolled **one node at a time, waiting for +`robustness=healthy` on every volume before the next**. + +Remediation — freeze rebuilds while the node recovers (operator, mutating): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \ + concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"' +``` + +Thaw to `"1"` — never higher — once all volumes read `healthy`. Use `0` when replicas will +**reattach** (an in-place bounce); use `1` when they are genuinely gone (an OS re-provision). +`concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and +held** — do not "fix" this by setting a knob that is already set. + +### 2. The node's disk is failing +`k3s-prod-1`'s original 2011 `C400-MTFDDAC512MAM` had **61,440 reallocated sectors**, 31% endurance +consumed and no power-loss protection. The operator replaced it with a Samsung 850 EVO on 2026-09-07 +and the same 18-replica rebuild measured **3.79** instead of 11.36 — ~3x better, node stayed Ready. +The earlier dev-era drive (Seagate `ST4000DM000`: 29,408 reallocated, 448 pending, 448 offline +uncorrectable) drove etcd to 48% iowait. If SMART shows reallocated or pending sectors climbing, +**the fix is hardware**; nothing in software recovers it. + +When replacing a disk, two traps recorded from that rebuild: +- Longhorn refuses the new disk with `DiskFilesystemChanged` ("record diskUUID doesn't match the one + on the disk"). Remedy: **disable → remove the disk entry → re-add, in that order** (the validating + webhook enforces it). +- `inventory/hosts.ini` decides `--cluster-init` via `[k3s_prod_primary]`. Re-provisioning a node + that is still listed there **founds a second cluster**. The founder now sits on `k3s-prod-2`. + +### 3. A Velero backup or restore running against the same spindle +Velero's daily runs at `0 3 * * *` and reads Longhorn PVCs through kopia. Check +`backups.velero.io` (fully qualified — see above) for one in flight, and the Longhorn limits +`concurrentVolumeBackupRestorePerNodeLimit` / `snapshotHeavyTaskConcurrentLimit`, both pinned to 1 +for exactly this reason. + +### 4. A CDI image import or a TCG-emulated VM +A CDI import writes a whole disk image; heavy sustained node load from TCG-emulated VMs has also been +observed to degrade the Cilium datapath into intermittent LB-VIP unreachability (cleared by a +restart). Check `kubectl --context k3s-prod get vmi -A` and importer pods in `cdi`. + +### 5. Structural: etcd and Longhorn share one partition +This is the standing cause underneath the other four, and it is open work. `roles/etcd_io_priority` ++ `playbooks/etcd-io-priority.yml` apply BFQ on the shared disk, `nr_requests` 64→32 and +`IOWeight=10000` on `system.slice` and `k3s.service`, reading every value back. **It applies nothing +until an operator runs it** — run `k3s-prod-3` first, with `--limit` and `--list-hosts`. + +Four mechanisms are **proven inert on these nodes** — do not re-try them: +`IOSchedulingClass=realtime` on `k3s.service` is a footgun (ioprio is inherited across fork/exec and +containerd is a child of the k3s server process, so realtime for k3s means realtime for Longhorn +too); `IOSchedulingPriority` is a no-op under `mq-deadline`; `io.latency` is unavailable +(`CONFIG_BLK_CGROUP_IOLATENCY` is not built in Debian trixie); a bare `IOWeight=` drop-in is +decoration (blk-iocost compiled-but-disabled, BFQ not loaded). And no cgroup or scheduler control +arbitrates the **jbd2 journal** that etcd's `fdatasync` actually waits on. + +## When to escalate or stop + +- **Stop and get the operator** if `NodeIOPressureCritical` is firing *and* the journal shows + `slow fdatasync` or `apply request took too long`. The node is minutes from leaving the cluster. + With one of three etcd members already gone, a second loss ends quorum. +- **Do not reboot or drain a second node** while any volume is not `healthy`. That is how a + single-node problem becomes a data problem on a 3-replica cluster. +- **Do not `kubectl drain`.** This cluster does not drain: the posture is deliberate (the upgrade + plan rejects system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD + Longhorn"), and the node-loss conformance test evicts only its own writer pod. +- Hand over to the operator for anything physical (disk replacement) or for the Ansible role — both + are outside what can be done from git. + +--- +Source of truth: `runbooks/node-io.md` in `home-network-k3s-impl`. Index of all runbooks: `/runbooks/` +Incident record: `DECISIONS-LOG.md` "2026-09-07 — k3s-prod-1 NotReady: etcd starved of disk I/O by +Longhorn", and `infra/PROD-PROVISIONING.md` §0c / §2. diff --git a/runbooks/node-memory.md b/runbooks/node-memory.md new file mode 100644 index 0000000..78c6784 --- /dev/null +++ b/runbooks/node-memory.md @@ -0,0 +1,123 @@ +# node-memory — Memory pressure high on a node + +**Alert:** `NodeMemoryPressureHigh` (warning, 10m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes` +**Expression:** `(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90` + +## What this means here + +Available memory has been under 10% for ten minutes. These are HP EliteDesk 800 G5 Minis with +~22 GiB usable, running the whole platform plus KubeVirt VMs on three nodes, so the headroom is +genuinely thin and this alert is a capacity signal rather than a bug signal most of the time. + +Two cluster-specific facts change how you read it: + +**`SwapTotal == 0` is not evidence swap is off.** On 2026-09-07 all three prod nodes came back from a +reboot with **977 MiB of swap ACTIVE** (`/dev/sda4` on prod-1 and prod-3, `/dev/nvme0n1p4` on prod-2) +while the Ansible role's commented-out fstab line reported success. +`systemd-gpt-auto-generator` claims any GPT partition typed +`0657fd6d-a4ab-43c4-84e5-0933c84b4f4f` straight from the partition table and synthesises a `.swap` +unit into `/run/systemd/generator.late/` — it never reads `/etc/fstab`. The fix masks `swap.target` +(the single choke point for gpt-auto, fstab-generator and cryptsetup-generator). If a node is +swapping, the kubelet's memory accounting is lying to you and eviction will misbehave. Check +explicitly with `swapon --show`, never `SwapTotal`. + +**There is a kernel memory-reporting bug on all three nodes.** `infra/PROD-PROVISIONING.md` §0b +records an EFI memattr defect that once presented `k3s-prod-1` as having **2.6 GiB usable of 16 GB**. +It needs kernel ≥ 6.13 and all three nodes are exposed (prod-3 publishes `MEMATTR=0xbeece018` too). +If `MemTotal` looks absurd rather than merely tight, suspect this before suspecting a leak. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod top nodes; echo; kubectl --context k3s-prod get nodes -o custom-columns=NAME:.metadata.name,MEM:.status.capacity.memory,PRESSURE:.status.conditions[?\(@.type==\"MemoryPressure\"\)].status' +``` + +Who is using it — and is swap actually off? + +``` +ssh k3s-prod-1 'free -h; echo; swapon --show; echo "(empty output above = swap really off)"; echo; cat /proc/pressure/memory; echo; ps -eo pid,rss,comm --sort=-rss | head -15' +``` + +Biggest consumers in the cluster, and anything that has been OOM-killed: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod top pods -A --sort-by=memory | head -20; echo; \ + kubectl --context k3s-prod get events -A --field-selector reason=OOMKilling --sort-by=.lastTimestamp | tail -20' +``` + +## Likely causes, most likely first + +### 1. A monitoring component grew past its limit +This is the one with real history. When kube-prometheus-stack went 65 → 88 it shipped Grafana 13 and +**Grafana OOM-looped** — "the hidden blocker that stalled the whole kps upgrade and the platform +tier". Limits went 256 → 512 → **768Mi** for Grafana and Prometheus 2 → **3Gi** (commit `b2b1aa5`). +Prod keeps 30d retention / 45GiB retentionSize on a 50Gi PVC, so Prometheus is legitimately the +largest single consumer. Remediation: raise the limit in +`platform/monitoring/kube-prometheus-stack.yaml` and push; do not restart it in a loop. + +### 2. A KubeVirt VM, especially a TCG-emulated one +VMs are the largest discretionary consumer. `useEmulation: false` is set deliberately (on v1.9 +`true` silently forces TCG), but leaked test namespaces running emulated VMs have caused API blips +before and were deleted. Check `kubectl --context k3s-prod get vmi -A` and the `virt-launcher-*` pods' +memory. + +### 3. Swap silently back on after a reboot +See above. Verify with `swapon --show` and `systemctl is-enabled swap.target` (should be `masked`). +`node_health` now reports `memory.swap_persistence` for this. The fix is the Ansible role; an +operator runs it. + +### 4. Genuine capacity exhaustion +If nothing is anomalous, the node is simply full. The node vCPU count was already raised 6 → 8 for +CPU pressure; memory is next. A bpg/OpenTofu memory change **reboots the VM**, so roll **one node at +a time and wait for Longhorn `robustness=healthy` between each** — the same discipline as every other +reboot here. + +### 5. A leak in an app-tier workload +Nextcloud, Plex and the *arr stack are the usual suspects. Compare `top pods` against the previous +day in Grafana rather than guessing. + +## When to escalate or stop + +- **Escalate immediately if this is firing together with `node-io` or `node-down`** on the same node. + Memory pressure plus I/O pressure on an etcd member is the shape of the 2026-09-07 outage. +- **Do not add swap** to relieve this. Kubernetes does not want it here and the platform has already + been bitten by swap re-enabling itself. +- **Do not evict or drain** to free memory: this cluster does not drain (Longhorn rebuild cost). Kill + the specific offending pod instead, and only if you know what it is. +- Hand to the operator for anything requiring a VM resize or an Ansible run. + +--- +Source of truth: `runbooks/node-memory.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` (Grafana OOM, kps 65→88); commits `0e92d25`/`9d00a3f` (swap persistence); +`infra/PROD-PROVISIONING.md` §0b (EFI memattr). diff --git a/runbooks/pvc-capacity.md b/runbooks/pvc-capacity.md new file mode 100644 index 0000000..241d109 --- /dev/null +++ b/runbooks/pvc-capacity.md @@ -0,0 +1,144 @@ +# pvc-capacity — A PersistentVolumeClaim is filling up, nearly full, or predicted to fill + +**Alerts:** `PersistentVolumeFillingUp` (warning, 15m), `PersistentVolumeAlmostFull` (critical, 15m), +`PersistentVolumeFillingUpPredicted` (warning, 1h) +**File:** `platform/monitoring/pvc-capacity-alerts.yaml`, group `platform-health.pvc-capacity` +**Expressions:** `< 0.15` free / `< 0.05` free / `< 0.50` free **and** +`predict_linear(kubelet_volume_stats_available_bytes[24h], 7*24*3600) < 0`, each `and +kubelet_volume_stats_inodes{job="kubelet"} > 0` + +## What this means here + +These are the PVC fill alerts for everything **outside** the `monitoring` namespace +(`/runbooks/monitoring-capacity` owns that half). Before this file existed, **no PVC outside +`monitoring` had any fill alert at all**. Unlike the Longhorn rule group, these work: they are built +on `kubelet_volume_stats_*`, which is live (56 series on prod). + +Three design facts you need in order to read the labels correctly: + +- **Every rule pins `job="kubelet"`.** `kubelet_volume_stats_*` is published by *two* jobs with + different label shapes: `job="kubelet"` carries the PVC's real namespace, while `job="apiserver"` + reports `namespace="default"` with the real one in `exported_namespace`. Without the pin, + `namespace!="monitoring"` double-fires on monitoring's own PVCs and names the wrong namespace. +- **Every rule carries `and kubelet_volume_stats_inodes > 0`.** Block-mode PVCs — KubeVirt VM disks + on `longhorn-vm` — report available=0, used=0, inodes=0 against a real capacity. Without the guard, + `unifi/unifi-rootdisk` (40 GiB Block) would fire both warning **and** critical permanently. If you + are ever tempted to remove that clause, this is why it is there. +- **The predicted rule is the point of the exercise.** A level threshold is useless for a slow leak on + a mostly-empty volume: prod's `keycloak-db` sat at **93% free** while leaking ~6.8 MiB/h of + unrecyclable WAL, so `<15%` was ~29 days away and would have arrived as a surprise. The `< 0.50` + gate keeps it quiet on volumes that are not plausibly about to fill, and the 24h fit / 7d horizon is + chosen so a daily backup or log rotation does not make it flap. + +**If the PVC name matches `*-db-N`, read `/runbooks/cnpg-archiving` before doing anything else.** +That is by far the most likely cause on this cluster and the one where the obvious action is fatal. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which PVC, how full, on which node: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s -G --data-urlencode "query=topk(10, 1 - kubelet_volume_stats_available_bytes{job=\"kubelet\"} / kubelet_volume_stats_capacity_bytes{job=\"kubelet\"})" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "persistentvolumeclaim|namespace|\"1\"" ' +``` + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get pvc -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SIZE:.spec.resources.requests.storage,SC:.spec.storageClassName,STATUS:.status.phase' +``` + +Is it a CNPG data volume? (Fully-qualified `clusters.postgresql.cnpg.io` — the short name `cluster` is +ambiguous on clusters carrying more than one `Cluster` kind.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A' +``` + +If yes, the one question that decides everything — is WAL archiving working? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint' +``` + +What is actually consuming the space, from inside the pod: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c "df -h /var/lib/postgresql/data; du -sh /var/lib/postgresql/data/pgdata/pg_wal; ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | wc -l"' +``` + +## Likely causes, most likely first + +### 1. A CNPG database is not recycling WAL +The dominant cause. With `archive_mode=on` and a failing `archive_command`, Postgres **refuses to +recycle** unarchived WAL segments, so `pg_wal` grows until the data PVC is full and the postmaster +**PANICs** — taking the database, and for `keycloak-db` all of SSO, down. `pg_wal` shares the data +PVC on every Cluster here (no `walStorage:` is configured), which is why a *backup* fault presents as +a *capacity* alert. This actually happened: prod ran from bootstrap to 2026-09-07 with +`endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, all three Clusters at `ContinuousArchiving=False` with +`barman-cloud-wal-archive: exit status 4`, unnoticed for 38+ hours. +**Remediation: fix archiving (`/runbooks/cnpg-archiving`). Do NOT delete WAL by hand — that destroys +the database. Do NOT expand the PVC and call it done; that buys days.** + +### 2. Log or cache growth in an app-tier workload +Nextcloud, the *arr apps, Gitea. Prune inside the app, or expand the PVC in git (every Longhorn class +has `allowVolumeExpansion: true`). + +### 3. Media on the wrong storage class +Media is meant to be static NFS PVs from `192.168.2.102:/mnt/Storage/plex` (8.32 TiB measured), not +Longhorn. A media volume on `longhorn` is a misconfiguration; fix the class, do not grow the disk. + +### 4. A Velero or CDI staging write +Transient. CDI uses `longhorn-scratch` deliberately (single replica) because the 3-replica default +failed the UniFi 60Gi import on insufficient storage. + +### 5. The predicted rule caught a genuine slow leak +This is the rule working as designed and giving you a week. Find what is growing before the level +thresholds arrive. + +## When to escalate or stop + +- **`PersistentVolumeAlmostFull` on a `*-db-N` volume is an imminent database outage.** Get the + operator. A Postgres data volume at under 5% free will PANIC the postmaster. +- **Never delete WAL segments, and never delete anything under a Postgres data directory.** +- **Do not delete the PVC** to reclaim space. +- If `unifi-rootdisk` or another Block-mode volume appears in these alerts, the inodes guard has been + removed or broken — that is a monitoring bug, not a storage incident. +- Escalate for PVC expansion beyond what the node can hold (see `/runbooks/longhorn-capacity`). + +--- +Source of truth: `runbooks/pvc-capacity.md`. Index: `/runbooks/` +Records: `platform/monitoring/pvc-capacity-alerts.yaml` (the two-job and inodes findings); +`platform/cnpg/alerts.yaml` header (the 2026-09-06 archiving incident and the measured fill rate). diff --git a/runbooks/serving/deployment.yaml b/runbooks/serving/deployment.yaml new file mode 100644 index 0000000..bf8c0d5 --- /dev/null +++ b/runbooks/serving/deployment.yaml @@ -0,0 +1,151 @@ +--- +# The runbook static server. Deliberately the dumbest thing that can work. +# +# WHY A POD AT ALL, AND NOT A REDIRECT TO GITEA. The obvious cheaper option is an HTTPRoute whose +# only job is a RequestRedirect into the in-cluster Gitea web UI, which already renders these files. +# It was rejected for three reasons, in order of weight: +# 1. DEPENDENCY DIRECTION. A runbook is read when the platform is broken. Gitea is an app-tier +# workload on Longhorn; its own DR path and several of these runbooks' subjects sit UNDER it. +# `runbooks/longhorn-faulted` must be readable while Longhorn is faulted, and +# `runbooks/keycloak-db` while a CNPG database is down. Pointing the operator at Gitea makes +# the documentation fail in exactly the conditions that produce the alert. +# 2. NO SUFFIX REWRITE EXISTS. Gateway API's RequestRedirect path modifier offers ReplaceFullPath +# and ReplacePrefixMatch only. There is no "append .md", so mapping 34 extensionless names onto +# 34 `.md` blobs needs 34 hand-maintained ReplaceFullPath rules — MORE surface than this pod, +# and a silent 404 for every runbook someone forgets to add a rule for. +# 3. A 302 is not a document. The annotation's contract is that the URL returns the runbook; a +# redirect chain through a second app's auth and rendering is a worse contract. +# This pod reads only from a ConfigMap projected out of git. It depends on the CNI, the kubelet and +# the Gateway, and on nothing else in the platform. +# +# WHY NOT THE SERVICE CATALOG (catalog/*.yaml + render.py). The catalog is the idiomatic onboarding +# surface for a SERVICE, and it was the first thing checked — but its schema cannot express this +# route. `ingressRoutes[].host` is pattern-bound to `^[a-z0-9-]+\.(\$\{BASE_DOMAIN\}|home\.keithjackson\.dev)$`: +# a single DNS label plus the apex, so the BARE apex is unrepresentable, and there is no `path` field +# at all, so a path-prefix route cannot be authored. Bending the schema for one docs server would be +# the tail wagging the dog. The repo's OTHER established pattern is exactly right instead: a small +# hand-authored platform component reconciled by its own non-gating Flux Kustomization — what +# platform/homepage does for the apex `/` prefix, which this mirrors file for file. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: runbooks + namespace: runbooks + labels: + app.kubernetes.io/name: runbooks +spec: + # TWO replicas with anti-affinity, unlike homepage's one. This is the only workload in the + # cluster whose whole purpose is to be available during an incident, and a single replica is + # unavailable for the minutes a node reboot or a drain takes — which is when it is wanted. It + # costs 2 x 16Mi. + replicas: 2 + revisionHistoryLimit: 3 + strategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 0 + maxSurge: 1 + selector: + matchLabels: + app.kubernetes.io/name: runbooks + template: + metadata: + labels: + app.kubernetes.io/name: runbooks + spec: + automountServiceAccountToken: false + # Spread across nodes so losing one node never takes the runbooks with it. `preferred`, not + # `required`: on a 3-node cluster with one node cordoned for the very incident being debugged, + # a hard rule would leave a replica Pending instead of running somewhere. + affinity: + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + topologyKey: kubernetes.io/hostname + labelSelector: + matchLabels: + app.kubernetes.io/name: runbooks + # Tolerate a node that is under memory/disk pressure or unreachable for a while. An operator + # reading `runbooks/node-disk` is by definition dealing with a node under pressure, and the + # default 300s unreachable toleration is long enough; these exist so eviction pressure does + # not silently remove the last replica while the cluster is sick. + tolerations: + - key: node.kubernetes.io/unreachable + operator: Exists + effect: NoExecute + tolerationSeconds: 900 + securityContext: + runAsNonRoot: true + runAsUser: 101 # `nginx` in nginxinc/nginx-unprivileged + runAsGroup: 101 + fsGroup: 101 + seccompProfile: + type: RuntimeDefault + containers: + - name: nginx + # Renovate maintains this pin. nginx-unprivileged (not nginx) so the container can honour + # runAsNonRoot: the stock nginx image's entrypoint needs root to drop privileges and to + # bind :80. `stable-alpine` branch, pinned exactly. + image: nginxinc/nginx-unprivileged:1.30.4-alpine + imagePullPolicy: IfNotPresent + ports: + - name: http + containerPort: 8080 + protocol: TCP + resources: + requests: + cpu: 10m + memory: 16Mi + limits: + memory: 64Mi + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + # Achievable here, unlike homepage/headlamp: nginx only needs /tmp (its pid file, on the + # unprivileged image) and /var/cache/nginx writable, and both are emptyDirs below. + readOnlyRootFilesystem: true + livenessProbe: + httpGet: { path: /healthz, port: http } + initialDelaySeconds: 3 + periodSeconds: 15 + readinessProbe: + # Probe a REAL runbook, not just /healthz. nginx answers /healthz from its own config + # even if the ConfigMap volume failed to project, so /healthz alone would report a pod + # that serves 404 for every runbook as Ready. `node-down` is the canonical alert link + # (NodeExporterDown, platform/monitoring/alerts.yaml), so if this path 404s the pod is + # useless and must not take traffic. + httpGet: { path: /runbooks/node-down, port: http } + initialDelaySeconds: 3 + periodSeconds: 15 + volumeMounts: + - name: nginx-config + mountPath: /etc/nginx/conf.d/default.conf + subPath: default.conf + readOnly: true + # The runbooks themselves, one extensionless file per key. Whole-volume mount (NOT + # subPath): subPath mounts do not update when the ConfigMap changes, and here the + # ConfigMap name carries kustomize's content hash so a content change rolls the pod + # anyway — the whole-volume mount just means a stale projection is impossible. + - name: runbooks + mountPath: /usr/share/nginx/html/runbooks + readOnly: true + - name: tmp + mountPath: /tmp + - name: cache + mountPath: /var/cache/nginx + volumes: + - name: nginx-config + configMap: + name: runbooks-nginx + - name: runbooks + configMap: + # Generated by ../kustomization.yaml's configMapGenerator from runbooks/*.md. kustomize + # rewrites this reference to the hashed name, so the pods roll whenever a runbook changes. + name: runbooks-content + - name: tmp + emptyDir: {} + - name: cache + emptyDir: {} diff --git a/runbooks/serving/httproute.yaml b/runbooks/serving/httproute.yaml new file mode 100644 index 0000000..35c6869 --- /dev/null +++ b/runbooks/serving/httproute.yaml @@ -0,0 +1,87 @@ +--- +# North-south ingress for the runbooks at https:///runbooks/ — the exact prefix the alert +# annotations already point at (`runbook_url: "https://${BASE_DOMAIN}/runbooks/"`, 42 of them +# across platform/monitoring/alerts.yaml, platform/monitoring/pvc-capacity-alerts.yaml, +# platform/cnpg/alerts.yaml and platform/kubevirt/alerts.yaml). Nothing about those annotations +# changes; this route is what finally makes them resolve. +# +# NO `hostnames:` — DELIBERATE, and the reason this manifest needs no Flux variable substitution. +# An HTTPRoute with no hostnames inherits the hostname of every listener it attaches to, and this +# route attaches to exactly one: `https-apex`, whose hostname is `${BASE_DOMAIN}` +# (platform/gateway/gateway.yaml). So the route is correct on dev and on prod with no +# `postBuild.substituteFrom` anywhere in this tier — which matters a great deal here, because the +# SAME Flux Kustomization renders the runbook markdown into a ConfigMap. With substitution enabled, +# Flux's envsubst would run over the runbook prose and silently EMPTY every `${...}` in every shell +# example a runbook contains (it fails open — DECISIONS-LOG 2026-09-01, the Cilium k8sServiceHost +# incident). Declaring no hostname removes the last reason to substitute, so the docs ship verbatim. +# See clusters/base/runbooks.yaml. +# +# PATH PRECEDENCE AGAINST HOMEPAGE. platform/homepage/httproute.yaml already binds `https-apex` with +# PathPrefix `/`. Gateway API resolves a conflict between rules on the same hostname by preferring +# the LONGEST path match, so `/runbooks` wins over `/` for these requests and homepage keeps +# everything else. The two routes coexist by specification, not by luck — and the live 200s recorded +# in the commit prove it on this Gateway implementation (Cilium). +# +# ============================ AUTH: THIS ROUTE IS PUBLIC ============================== +# auth-mode=none, deliberately, and it is the one route in this repo where that is the SAFE choice. +# +# THE CIRCULARITY. Every other exposed UI here declares forward-auth (oauth2-proxy -> Keycloak). +# Put these pages behind that and `runbooks/keycloak-down` becomes unreadable precisely when +# Keycloak is down; `runbooks/keycloak-db` unreadable when its CNPG database is down; +# `runbooks/keycloak-client-secrets` unreadable during the exact failure mode that broke all 12 +# OIDC clients on 2026-09-06 while every health signal stayed green. An on-call who cannot log in +# is the primary reader of this route. Gating documentation on the identity provider it documents +# is a loop with no exit. +# +# WHAT IS ACTUALLY EXPOSED. Operational procedure: metric names, thresholds, kubectl invocations, +# component topology. No credentials, no secrets, no tokens, no internal IPs beyond the LAN ranges +# already written in the public spec repo. The content is the same text any reader of the git repo +# can see. The real cost is reconnaissance value — it names what runs here and how it fails — and +# that is accepted knowingly, against the certainty of a locked-out operator staring at a login page +# during an outage. If that trade is ever revisited, the correct fix is NOT forward-auth on this +# route; it is to keep it public and move sensitive specifics out of the runbooks. +# +# HOW forward-auth IS ENFORCED HERE — and why this route is genuinely different. Enforcement is NOT +# a Gateway ext-authz attachment; it is PER-POD. Each forward-auth service runs an oauth2-proxy +# sidecar, its Service publishes BOTH the app port and 4180, and the service's HTTPRoute targets +# :4180 — verified on prod for all five that declare `auth: forward-auth` in the catalog (pihole, +# jackett, qbittorrent, radarr, sonarr). Ingress traffic therefore always traverses oauth2-proxy, +# which performs the Keycloak exchange and the group check. The app port stays on the Service so +# in-cluster clients can talk service-to-service without SSO (that is how sonarr/radarr reach +# qBittorrent's API), which is deliberate, not a leak: Cilium default-deny decides who may use it. +# +# So this route is NOT "no more open than its neighbours" — its neighbours are closed and this one is +# open, on purpose, for the lockout reason argued above. It has no oauth2-proxy sidecar and targets +# its app port directly. If a Gateway-level ext-authz attachment is ever introduced, this route must +# be left OUT of it ON PURPOSE, and this comment is the record of that decision so nobody "fixes" it +# later and re-closes the loop. (Headlamp is also excluded from forward-auth, for a different and +# equally deliberate reason: it runs native Keycloak OIDC and forwards the logged-in user's id_token +# to the apiserver for per-user RBAC, with clusterRoleBinding.create=false so its ServiceAccount can +# do nothing on its own.) +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: runbooks + namespace: runbooks + labels: + app.kubernetes.io/name: runbooks + annotations: + homelab.keithjackson.dev/auth-mode: none + homelab.keithjackson.dev/auth-rationale: >- + Public on purpose. An operator locked out of Keycloak is this route's primary reader; + gating it behind Keycloak would make runbooks/keycloak-down unreadable exactly when it is + needed. Content is operational procedure only — no secrets. Do NOT add this route to the + Gateway ext-authz attachment when forward-auth enforcement lands. +spec: + parentRefs: + - name: shared + namespace: gateway + sectionName: https-apex # the apex listener; the wildcard `https` listener won't match it + rules: + - matches: + - path: + type: PathPrefix + value: /runbooks + backendRefs: + - name: runbooks + port: 80 diff --git a/runbooks/serving/namespace.yaml b/runbooks/serving/namespace.yaml new file mode 100644 index 0000000..85ec480 --- /dev/null +++ b/runbooks/serving/namespace.yaml @@ -0,0 +1,16 @@ +--- +apiVersion: v1 +kind: Namespace +metadata: + name: runbooks + labels: + # Serve the runbooks at https:///runbooks/ — the APEX host, on a `/runbooks` path + # prefix. The apex has its own `https-apex` listener on the shared Gateway + # (platform/gateway/gateway.yaml); this label opts the namespace in to attach HTTPRoutes to it + # (serving/httproute.yaml). Same opt-in as homepage, which owns the apex `/` prefix. + gateway-access: "shared" + # NOT homelab.keithjackson.dev/app-namespace=true. This is a PLATFORM component (like homepage and + # headlamp), so Kyverno does not generate the app default-deny baseline here and the app-scoped + # auth-mode / egress-class policies do not apply. The default-deny NetworkPolicy and the + # Gateway-only CiliumNetworkPolicy are shipped explicitly in serving/networkpolicy.yaml instead. + app.kubernetes.io/name: runbooks diff --git a/runbooks/serving/networkpolicy.yaml b/runbooks/serving/networkpolicy.yaml new file mode 100644 index 0000000..9a249ad --- /dev/null +++ b/runbooks/serving/networkpolicy.yaml @@ -0,0 +1,57 @@ +--- +# Default-deny baseline (R-NET-01): deny all ingress + egress, allow only DNS. `runbooks` is a +# PLATFORM namespace (no app-namespace label) so Kyverno does not auto-generate this — it is shipped +# explicitly, defence in depth, same shape as the Kyverno-generated app baseline and as +# platform/homepage/networkpolicy.yaml. Cilium enforces the UNION of this NetworkPolicy and the +# CiliumNetworkPolicy below (allow if ANY policy allows). +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-deny + namespace: runbooks + labels: + homelab.keithjackson.dev/baseline: default-deny +spec: + podSelector: {} + policyTypes: + - Ingress + - Egress + ingress: [] + egress: + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 +--- +# Gateway ingress allow (R-NET-01/02): only the shared Gateway (the Cilium `ingress` entity) may +# reach the runbook server, on container port 8080. The server makes NO outbound calls whatsoever — +# every byte it serves comes from a ConfigMap projected by the kubelet — so the baseline's DNS-only +# egress is already more than it needs and no egress CNP is added. +# +# CONSEQUENCE WORTH STATING: because egress is DNS-only, these pages cannot pull a stylesheet, a +# font or a markdown renderer from anywhere. That is why the content is served as text/plain +# (serving/nginx-config.yaml) rather than as HTML with a CDN dependency, and it is a feature: the +# runbooks render with zero network beyond the request that fetched them. +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: runbooks-allow-gateway-ingress + namespace: runbooks + labels: + app.kubernetes.io/name: runbooks +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: runbooks + ingress: + - fromEntities: + - ingress + toPorts: + - ports: + - port: "8080" + protocol: TCP diff --git a/runbooks/serving/nginx-config.yaml b/runbooks/serving/nginx-config.yaml new file mode 100644 index 0000000..6ca8b9f --- /dev/null +++ b/runbooks/serving/nginx-config.yaml @@ -0,0 +1,81 @@ +--- +# nginx server config for the runbook static server. Hand-authored ConfigMap (same plain-manifest style +# as platform/homepage), mounted over /etc/nginx/conf.d/default.conf. +# +# WHY THE URI NEEDS NO REWRITE. The alert annotations are `.../runbooks/` with NO `.md` suffix, and +# this repo must not change them (they are spread across four alerts.yaml files that other work is +# editing). Rather than rewriting the URI, the ConfigMap generated from runbooks/*.md maps each file to an +# EXTENSIONLESS key (`node-down=node-down.md` in ../kustomization.yaml) and is mounted at /runbooks/. +# So `/runbooks/node-down` is a plain static file lookup: no rewrite, no fallback chain, nothing to get +# subtly wrong at 3am. +# +# CONTENT TYPE. An extensionless file has no MIME mapping, so nginx would serve it as +# application/octet-stream and a browser would DOWNLOAD it instead of showing it. The empty `types {}` +# block removes the mapping table entirely inside these locations and `default_type` forces +# text/plain;charset=utf-8 — markdown read as plain text, which is what markdown is for, and which works +# identically in a browser, in `curl`, and in a terminal with no JavaScript, no CDN and no egress. This +# namespace has DNS-only egress, so a client-side markdown renderer fetched from a CDN could not load +# anyway; plain text is the honest choice here, not a shortcut. +apiVersion: v1 +kind: ConfigMap +metadata: + name: runbooks-nginx + namespace: runbooks + labels: + app.kubernetes.io/name: runbooks +data: + default.conf: | + # Replaces the image's stock conf.d/default.conf. Port 8080 because the base image is + # nginx-unprivileged (runs as uid 101 and cannot bind 80). + server { + listen 8080; + server_name _; + root /usr/share/nginx/html; + + # Never leak the server version on an error page. + server_tokens off; + # Keep the 301 below relative so it works through the Gateway without knowing the public host. + absolute_redirect off; + + # Probes. Kept OUT of the access log so a 15s kubelet probe does not bury the one request an + # operator actually made. + location = /healthz { + access_log off; + default_type text/plain; + return 200 "ok\n"; + } + + # /runbooks and /runbooks/ both land on the index (runbooks/README.md, mounted as the + # extensionless key `README`). + location = /runbooks { + return 301 /runbooks/; + } + location = /runbooks/ { + types { } + default_type "text/plain; charset=utf-8"; + try_files /runbooks/README =404; + } + + # The runbooks themselves. One static file per alert runbook_url. + location /runbooks/ { + types { } + default_type "text/plain; charset=utf-8"; + # A ConfigMap volume is a tree of symlinks into ..data/; nginx must follow them. + disable_symlinks off; + try_files $uri =404; + add_header Cache-Control "public, max-age=300"; + # An unknown name gets a 404 with a POINTER, not a blank page: the likeliest cause is an + # alert whose runbook was never written, and the reader needs to know where to look next. + error_page 404 = @no_such_runbook; + } + location @no_such_runbook { + default_type "text/plain; charset=utf-8"; + return 404 "No runbook by that name.\n\nIndex of all runbooks: /runbooks/\nSource of truth: runbooks/.md in the home-network-k3s-impl repo.\nIf an alert linked you here, its runbook_url has no matching runbooks/.md, and the\nconformance check alert-runbooks-resolvable should have caught it before merge.\n"; + } + + # Nothing else is served from this pod. The apex `/` prefix belongs to homepage. + location / { + default_type "text/plain; charset=utf-8"; + return 404 "This host serves /runbooks/ only.\n"; + } + } diff --git a/runbooks/serving/service.yaml b/runbooks/serving/service.yaml new file mode 100644 index 0000000..771a9be --- /dev/null +++ b/runbooks/serving/service.yaml @@ -0,0 +1,19 @@ +--- +# ClusterIP fronting the runbook server. The shared Gateway routes the apex `/runbooks` prefix here +# (serving/httproute.yaml); Service port 80 -> container port 8080. +apiVersion: v1 +kind: Service +metadata: + name: runbooks + namespace: runbooks + labels: + app.kubernetes.io/name: runbooks +spec: + type: ClusterIP + selector: + app.kubernetes.io/name: runbooks + ports: + - name: http + port: 80 + targetPort: http + protocol: TCP diff --git a/runbooks/targets-down.md b/runbooks/targets-down.md new file mode 100644 index 0000000..73643ca --- /dev/null +++ b/runbooks/targets-down.md @@ -0,0 +1,138 @@ +# targets-down — More than 10% of Prometheus scrape targets are down + +**Alert:** `PrometheusTargetsMissing` (warning, 15m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.monitoring-self` +**Expression:** `(count(up == 0) / count(up)) > 0.10` + +## What this means here + +A tenth of everything Prometheus is supposed to scrape is unreachable. This is a **blind-spot** alert: +its purpose is to tell you that other alerts may be silent because their data is missing, not because +the thing they watch is healthy. + +Live baseline on prod, measured 2026-09-13: **55 targets across 19 jobs** — +`apiserver`, `cdi-prometheus-metrics`, `coredns`, `keycloak-discovery`, `keycloak-service`, +`keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`, `kube-prometheus-stack-grafana`, +`kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`, `kube-state-metrics`, `kubelet`, +`kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`, `node-exporter`, `oauth2-proxy`, +`semaphore-db/semaphore-db`, `velero`, `velero/node-agent`. So 10% is roughly **6 targets** — one +node's worth of exporters, or one CNPG cluster's three pods. + +**The far more dangerous failure this alert CANNOT see: a target that was never configured.** `up` +only counts targets Prometheus knows about. A missing ServiceMonitor produces no `up == 0` series and +no alert at all. This cluster has been bitten by that repeatedly, and these are the known live gaps as +of 2026-09-13 — every one of them means the corresponding alerts are **inert**: + +| Missing scrape job | Alerts that cannot fire | +|---|---| +| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornVolumeCapacityHigh`, `LonghornNodeStorageHigh` | +| cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` | +| Flux controllers (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` | +| Hubble / Cilium (`hubble_drop_total`) | `HubbleDeniedFlowSpike` | +| etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all | + +Precedent for how costly this is: Velero shipped with three backup alerts and +`metrics.serviceMonitor.enabled: false`, so *"a nightly backup could fail — or silently stop running — +for weeks with nothing to say so."* That ServiceMonitor now exists. The rest of the table does not. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which targets are down, grouped by job: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s -G --data-urlencode "query=up == 0" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "\"job\"|\"instance\"|\"namespace\"" ' +``` + +The scrape-error text, which usually names the cause outright: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ + curl -s "localhost:19090/api/v1/targets?state=active" | python3 -c " +import json,sys +for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]: + if t[\"health\"]!=\"up\": print(t[\"labels\"].get(\"job\"), t[\"scrapeUrl\"], t[\"lastError\"][:120])"' +``` + +Which monitors exist at all (the gap this alert cannot see): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get servicemonitors,podmonitors -A' +``` + +## Likely causes, most likely first + +### 1. A node is down or NotReady +One node takes out its `node-exporter` and `kubelet` targets together — roughly the 10% threshold on +its own. Go to `/runbooks/node-down`; this alert is the echo. + +### 2. A workload was rolled, scaled to zero, or crashlooped +Targets follow pods. A CNPG cluster restarting takes three targets with it. Transient; if it persists, +the workload is the problem, not the monitoring. + +### 3. A NetworkPolicy blocks the scrape +Default-deny is the baseline everywhere (R-NET-01) and a new component's monitor will be scraped the +moment it exists — `serviceMonitorSelectorNilUsesHelmValues: false` means Prometheus selects every +monitor in the cluster — so a newly added workload can produce a permanently-down target purely +because nothing allows ingress from `monitoring`. The `lastError` will be a timeout. Fix is an +explicit allow in the component's CiliumNetworkPolicy, in git. + +### 4. TLS or auth on the scrape endpoint +`lastError` will say so (certificate or 401/403). Note a related trap from this cluster: after the +Cilium upgrade, an in-cluster call to a `*.home` hostname is re-checked against the destination's real +backend identity, so `toEntities: world` no longer covers it — see `/runbooks/denied-flows`. + +### 5. The exporter is genuinely broken +CDI once had a broken HTTP ServiceMonitor that caused a false `CDIOperatorDown`, and +`KubeVirtControlPlaneDown` once used a non-existent metric with the wrong job label. Both were fixed. +An exporter that is up but publishing the wrong series is the hardest case — see +`/runbooks/cnpg-exporter` for the canonical example (the CNPG exporter can be `up=1` while its whole +`cnpg_pg_stat_*` family is missing). + +## When to escalate or stop + +- **Treat this alert as "trust nothing else for now".** Before acting on the *silence* of any other + alert, confirm its series exists (`count()`), because the table above shows five whole rule + groups are currently evaluating against nothing. +- Escalate if the down targets are `apiserver` or `kubelet` — that is a control-plane problem, not a + monitoring one. +- Adding a missing ServiceMonitor is a git change (a new `ServiceMonitor` next to the component, or a + chart value). Do not "fix" a blind spot by silencing the alert. +- The Longhorn, cert-manager and Flux monitor gaps are known open work. Note them; do not try to + hand-apply monitors into the cluster — every durable change goes through git. + +--- +Source of truth: `runbooks/targets-down.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` / commit `2d5171e` (Velero had no ServiceMonitor, so three critical backup +alerts could never fire); commits `6a7e9d1`, `b57b3e9` (the false CDI/KubeVirt alerts). diff --git a/runbooks/update-rollback.md b/runbooks/update-rollback.md new file mode 100644 index 0000000..2f49e77 --- /dev/null +++ b/runbooks/update-rollback.md @@ -0,0 +1,150 @@ +# update-rollback — A HelmRelease failed its health gate and is rolling back + +**Alert:** `UpdateHealthGateFailed` (critical, 10m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.automated-updates` +**Expression:** `max by (namespace, name) (gotk_reconcile_condition{kind="HelmRelease", type="Ready", status="False"}) == 1` + +## What this means here + +helm-controller holds a HelmRelease at Ready=False while a failed upgrade remediates and rolls back to +the last-good revision. This is the **R-UPD-03 safety rail firing** — the bad-update auto-rollback +working — so the alert is a report that the platform protected itself, not necessarily that something +is broken right now. + +Read the expression honestly, though: it matches **any** HelmRelease at Ready=False, not only one that +is mid-rollback. The rule's own comment flags this as a `TODO(bring-up)`: tighten it with the `reason` +label (`UpgradeFailed` / `RolledBack`) once confirmed against the deployed helm-controller. So a +HelmRelease that simply cannot install — a bad value, a missing CRD, a crashlooping pod — lights this +`critical` too. Your first job is to tell those two cases apart, because they need opposite responses: +a rollback is self-healing and wants a git revert; a failed *install* is a stall and may be gating an +entire tier (see `/runbooks/flux-reconcile`). + +Real precedent that rollback does work here: during the 2026-09-01 Cilium incident the Helm release +history shows `v8 e83bf695 → v9 eebd6c88 (the bad render) → v10 e83bf695 (identical to v8)`, and the +release secret for v9 reads `failed`. Helm's wait genuinely failed and the release was remediated — a +claim that remediation never engaged was **measured wrong**. That is why the cluster self-healed in +about six minutes. +**This alert is currently INERT.** `count(gotk_reconcile_condition)` returned **0 series** on prod on +2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every `gotk_*` rule +evaluates against nothing. Read the HelmRelease CRs directly with the commands below, and see +`/runbooks/targets-down` for the full list of inert rule groups. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Which release, and what happened: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get helmreleases.helm.toolkit.fluxcd.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,REASON:.status.conditions[?\(@.type==\"Ready\"\)].reason,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,LASTSUCCESS:.status.lastAppliedRevision' +``` + +The Helm revision history — this is what distinguishes "rolled back" from "never installed": + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n get secrets -l owner=helm --sort-by=.metadata.creationTimestamp \ + -o custom-columns=NAME:.metadata.name,STATUS:.metadata.labels.status,VERSION:.metadata.labels.version' +``` + +The controller's reasoning: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n flux-system logs deploy/helm-controller --tail=120 2>&1 | grep -iE "|rollback|remediat|upgrade failed" | tail -30' +``` + +What the workload itself is doing: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n get pods -o wide; \ + kubectl --context k3s-prod -n get events --sort-by=.lastTimestamp | tail -20' +``` + +If the release touches backups or databases, verify them with **fully-qualified names** before deciding +anything is fine — `kubectl get backup -A` prints "No resources found" on this cluster even with six +Velero and 27 CNPG backups present: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get backups.velero.io -A; echo; \ + kubectl --context k3s-prod get backups.postgresql.cnpg.io -A; echo; \ + kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A' +``` + +## Likely causes, most likely first + +### 1. A chart upgrade shipped an incompatible image or value +The recorded instances: **kube-prometheus-stack 65 → 88 brought Grafana 13, which OOM-looped** and +became "the hidden blocker that stalled the whole kps upgrade and the platform tier" (fixed by raising +Grafana 256 → 512 → 768Mi and Prometheus 2 → 3Gi, commit `b2b1aa5`); and **headlamp-0.40.1** shipped a +`-session-ttl` chart/image desync that crashlooped. Remediation: pin the previous version in git +(`update.pin` is the documented escape hatch, docs/11) and push. The rollback has already restored +service; the git change stops it happening again next interval. + +### 2. Not a rollback at all — a first install that cannot succeed +`REASON` will be an install/dependency failure and there will be no earlier `deployed` Helm secret. If +the release is inside the gating `platform` tier, everything downstream is frozen behind it — go to +`/runbooks/flux-reconcile`, which covers the "no matches for kind" / CRD-ordering class. + +### 3. A timeout, not a failure +`platform` has `timeout: 5m` while Longhorn's HelmRelease has `timeout: 15m` — **the tier can time out +while a legitimate Longhorn upgrade is still running.** Give it another interval before declaring a +failure, and check whether pods are still progressing. + +### 4. The rollback itself is stuck +Helm cannot roll back cleanly if the upgrade already mutated something immutable (a StatefulSet +selector, a PVC). Then Ready=False is permanent until an operator intervenes. + +### 5. A post-upgrade health check is failing for an unrelated reason +A node under I/O pressure makes pods slow to become Ready and the upgrade's wait expires. Cross-read +`/runbooks/node-io`. + +## When to escalate or stop + +- **Get the operator if the release is `cilium` or anything in the gating `platform` tier.** A CNI + rollback affects every node's datapath. +- **Do NOT enable `driftDetection.mode: enabled` on a CNI.** During the 2026-09-01 incident the Helm + storage manifest *was* the bad render, so drift correction would have actively re-asserted the broken + value. This is a recorded conclusion, not a preference. +- **Do not `helm rollback` by hand.** helm-controller owns the release; a manual rollback creates drift + it will fight. Use git. +- **Do not suspend the HelmRelease to silence the alert** — that trades a visible failure for + `/runbooks/flux-suspended` and silent drift. +- If the rolled-back release is CNPG or Keycloak, expect knock-on SSO effects and read + `/runbooks/keycloak-down`; note also that a CNPG update under `primaryUpdateMethod: switchover` + cannot change `imageName` and a postgresql parameter in the same reconcile — the operator rejects it. + +--- +Source of truth: `runbooks/update-rollback.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` "2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium" (the measured +Helm revision history and the drift-detection conclusion); the Grafana OOM and headlamp desync. diff --git a/runbooks/vpn-down.md b/runbooks/vpn-down.md new file mode 100644 index 0000000..caa7765 --- /dev/null +++ b/runbooks/vpn-down.md @@ -0,0 +1,187 @@ +# vpn-down — The egress VPN tunnel is down + +**Alert:** `VPNTunnelDown` (critical, 5m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.networking` +**Expression:** +``` +absent(node_network_receive_bytes_total{device="tun0"}) +and on() ( + (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0) + or (count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0) +) +``` + +## What this means here + +There is **no in-cluster egress-gateway pod** on this platform (ADR-0015). The tunnel is a **host +systemd service** — `vpn-egress-tunnel`, OpenVPN to PIA — running on one node, and steering is Cilium +eBPF via a `CiliumEgressGatewayPolicy` (`vpn-egress-steering`). OpenVPN creates and destroys `tun0` with +the connection, so the `tun0` byte-counter series exists **if and only if** the tunnel is up. The rule +keys on that presence rather than `node_network_up`, because `tun0` reports `operstate=unknown` and +`node_network_up` is therefore 0 even when the tunnel is fine. + +Live state, measured 2026-09-13: **the tunnel is up** (`node_network_receive_bytes_total{device="tun0"}` += 1 series) and the gateway node is labelled (`kube_node_labels{...egress_gateway="true"}` = 1 series), +so this alert is armed and quiet. The gateway is **`k3s-prod-3`** (192.168.2.12, `eno1`). + +**The two facts that decide your response:** + +1. **Cilium OSS has no egress-gateway failover.** Gateway-node loss is an outage, not a degradation. + The ADR-0016 supersede drafted on 2026-09-06 is still open. Worse, Cilium OSS picks *"the first node + in lexical ordering"* among nodeSelector matches — so labelling `k3s-prod-1` would **silently steal** + the gateway role and the public egress IP from `k3s-prod-3`. `[vpn_egress_gateway_prod]` is a + separate inventory group holding only `k3s-prod-3` for this reason; run the role as + `-e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3`, with `--list-hosts` first, + every time. +2. **The `absent()` was originally evaluated cluster-wide and became a permanent false alarm.** On a + cluster with no gateway node it could never clear, and from 2026-09-05T21:23Z it fired continuously + into `email-critical` — unbounded noise on the one channel that must stay trustworthy. The two + guards OR'd above are the fix: the alert arms itself the moment a node is labelled, and stays armed + through a kube-state-metrics outage. Before that, the very first version keyed on + `absent(gluetun_vpn_up)`, a metric gluetun's control API never exported at all — a pre-existing + perma-false-alarm. **If this alert is firing, verify the tunnel by hand before believing it.** +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Confirm on the host, not from Prometheus: + +``` +ssh k3s-prod-3 'systemctl status vpn-egress-tunnel --no-pager | head -20; echo; ip -br addr show tun0; echo; ip link show tun0 | head -2' +``` + +Which node is the gateway, and is it labelled? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink; echo; \ + kubectl --context k3s-prod get ciliumegressgatewaypolicies.cilium.io -o yaml | grep -A6 "nodeSelector\|egressIP\|interface"' +``` + +**The check that catches the silent failure** (see cause 1 below) — does Cilium's datapath point at the +interface that actually exists? + +``` +ssh k3s-prod-3 'ip -o link | awk -F: "{print \$1, \$2}" | grep tun0' +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg bpf egress list' +``` + +If the `Egress Ifindex` in that map is not the live `tun0` ifindex, egress is broken **even though this +alert is quiet**. + +Is traffic actually leaving through the tunnel? + +``` +ssh k3s-prod-3 'curl -s --interface tun0 --max-time 10 https://ipinfo.io/ip; echo; nft list table inet vpn_killswitch 2>/dev/null | grep -E "counter|killswitch_" | head' +``` + +## Likely causes, most likely first + +### 1. The tunnel is UP but Cilium is steering at a dead interface (the silent one) +This is the richest failure on the platform and it does **not** set off this alert. +**Cilium resolves a `CiliumEgressGatewayPolicy`'s `interface: tun0` exactly once**, caches +(egress IP, ifindex), and never re-resolves. On `k3s-dev-3`, `cilium-dbg bpf egress list` showed +`Egress Ifindex 3` — **there is no ifindex 3 on that node**; the live `tun0` was **1780**. Egress was +dead for about 80 minutes with nothing alerting and the conformance ledger still reading green. +`egress-gateway-reconciliation-trigger-interval: 1s` does **not** cover it (netlink link events are not +Kubernetes object events). + +Measured behaviour, so you can pick a lever that works: +| action | result | +|---|---| +| annotate the CEGP | map → 1780 in 8s, PIA IP returns | +| `systemctl restart vpn-egress-tunnel` | tun0 1780→1985, **map stays 1780**, still broken | +| the exact `vpn-killswitch` stop/30s/start sequence | still broken 20s+ later | +| set **any** unrelated label on the gateway **node** | recovers within 10s | + +The durable fix is in `roles/vpn_egress_gateway`: the OpenVPN up-script publishes a node label +`homelab.keithjackson.dev/egress-uplink=-` on every connect (via +`systemctl start --no-block`, so it can never block the tunnel), plus a 2-minute convergence timer for +boot. A node label, not a CEGP annotation — a node writing to GitOps state is a drift fight. +**Nothing monitors this coupling yet**; that gap is exactly why dev sat dead for 80 minutes. + +### 2. PIA handed two gateways the same tunnel address +Standing rule: **one PIA server per gateway.** The vaulted `.ovpn` pins one server, so when prod came up +green, dev went down: both gateways dialled `158.173.241.112:1198` and PIA gave both the tunnel address +`10.72.192.7`. Each reconnect steals it — a measured ~2-minute mutual-eviction cycle (dev restarting at +22:57:05, 22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC). Fix: the role var +`vpn_egress_gateway_pia_remote` ("` `") rewrites only the `.ovpn` `remote` line; prod is +pinned to **`158.173.241.135`** (pool `158.173.241.{34,55,112,135,239}`). *"A single green probe is not +a stability check."* + +### 3. PIA "unreachable" — suspect a stale server list first +The 2026-08-22/23 chain went port collision → UDP/53 → 8080 → "MTU" (Cilium VXLAN pod MTU 1450 is real +but irrelevant to a **14-byte** handshake) → "the network blackholes PIA" (**wrong**) before the actual +cause: **gluetun's stale bundled server list**, dialling retired `46.246.x`/`212.112.x` addresses when +live SE Stockholm is `158.173.241.x`. Lesson recorded verbatim: when a client "can't reach" the +provider, suspect the stale server list **first**. + +### 4. The gateway node is down +`absent()` also fires when the gateway node or its node-exporter is gone, so this alert covers +gateway-node-down by design. Go to `/runbooks/node-down`. Remember: **there is no failover** — while +`k3s-prod-3` is down, VPN-classed egress is an outage. + +### 5. Nothing is wrong and the alert is lying +Given the two false-alarm post-mortems above, always confirm `ip -br addr show tun0` on the host before +acting. + +## What the kill-switch does and does not do + +Proven with counters on `k3s-prod-3`, because the intuitive reading is wrong: with +**`bpf.masquerade: true`** (which Egress Gateway requires) SNAT and egress happen in eBPF and +`bpf_redirect()` **skips netfilter POSTROUTING**. Measured: `forward` saddr=pod-CIDR **0**, +`postrouting` oif WAN saddr=pod-CIDR **0**. **What actually fails closed is Cilium's own drop** — +tcpdump during tunnel stop and restart showed **0 packets leaving either device**. The nft table is +kept as defence-in-depth for node-originated tunnel-sourced traffic and was proven both ways: +`ping -I 10.72.192.7` with the ip-rule → 3/3 at 144 ms with `killswitch_accept` advancing; rule removed +→ `killswitch_drop` 0→3 packets / 252 bytes, 100% loss. Every rule carries a `counter`, so: +**if `killswitch_drop` ever counts in normal operation, the datapath has changed.** A reported "leak +window" was **measured as no WAN leak** — a stale egress IP follows the cached ifindex, not a routing +lookup. Note the `.nft` file uses `table` + `flush table`, which clears rules but leaves named-set +elements intact. + +## When to escalate or stop + +- **Escalate immediately if the gateway node is down.** No failover exists; this is an outage with no + in-cluster remedy. +- **Never label a second node `egress-gateway=true`.** Lexical ordering means `k3s-prod-1` would + silently take the role and the public egress IP. +- **Do not restart the tunnel expecting Cilium to follow** — measured above, it does not. If the map is + stale, touching a node label is the lever that works (operator action). +- **Do not remove or "simplify" the nft kill-switch** because the counters read zero. Zero on the pod + path is the expected, measured result of eBPF masquerade; the table guards a different path. +- A leak concern is a security matter (docs/07) — escalate rather than experimenting. + +--- +Source of truth: `runbooks/vpn-down.md`. Index: `/runbooks/` +Records: `DECISIONS-LOG.md` "2026-09-12 — VPN egress on PROD (k3s-prod-3), and three things the repo +believed that are not true"; the 2026-08-22/23 gluetun chain; commit `155df8e` (R-NET-06 re-proven on +prod, recovery ≤3s).