Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

merge: 35 runbooks, served, with a guard that they resolve
gate / gate (push) Has been cancelled
Details
gate / conformance (push) Has been cancelled
Details

Browse Source 
All 46 runbook_url annotations pointed at https://<apex>/runbooks/<name>
and every one returned 404 — there was no runbooks/ directory and no route.
Every alert that reached a human handed them a dead link at the moment they
needed it.

Runbooks are now served from their own non-gating tier at the apex prefix,
PUBLIC on purpose: gating them behind Keycloak would make keycloak-down
unreadable exactly when Keycloak is down, which is a loop with no exit. The
new L0 guard checks both directions — a runbook_url with no file, and a file
not wired into the generator — and it caught a live one on contact with
main: CNPGPrimariesCoLocated had shipped hours earlier pointing at a runbook
that did not exist.
This commit is contained in:
jakman85 committed 2026-09-13 00:11:03 -05:00
2 parents d616fd4025 bce8778096
commit e49b86eb41
47 files changed
+6768
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

clusters/base/kustomization.yaml
+7
View File
@@ -50,6 +50,13 @@ resources:
# reverse-proxy sidecar (Option B). POST-platform tier (dependsOn platform-config, wait:false); same
# non-gating leaf-UI rationale as headlamp/homepage/oauth2-proxy (a UI must never gate the platform).
- crafty.yaml # Flux Kustomization -> ./platform/crafty (dependsOn platform-config)
# Alert runbooks — POST-platform tier (dependsOn platform-config, wait:false). Serves the 34
# runbooks the alert rules' `runbook_url` annotations point at, at <apex>/runbooks/<name>; before
# this tier existed that path was a 404 and every alert handed the operator a dead link. Same
# non-gating leaf rationale as headlamp/homepage/crafty. Carries NO postBuild substitution on
# purpose — it renders documentation containing `${...}` shell examples that envsubst would empty;
# see clusters/base/runbooks.yaml.
- runbooks.yaml # Flux Kustomization -> ./runbooks (dependsOn platform-config)
# Velero backup controller — POST-platform tier (dependsOn platform-config, wait:false). A backup
# controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy.
# Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name).
clusters/base/runbooks.yaml
+57
View File
@@ -0,0 +1,57 @@
---
# Flux Kustomization: alert runbooks — reconciles ./runbooks.
#
# WHAT THIS SERVES. Every alert rule in this repo carries
# `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names.
# Until this tier existed that path returned 404 on the live cluster, so every alert that reached a
# human handed them a dead link at the moment they needed help. This Kustomization renders the
# markdown under ./runbooks into a ConfigMap and a two-replica nginx that serves it at that exact
# prefix. See runbooks/README.md and runbooks/serving/httproute.yaml.
#
# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false) — the same placement and
# the same reasoning as homepage/headlamp/oauth2-proxy. It needs the wildcard TLS cert, the shared
# Gateway (including the `https-apex` listener) and split-horizon DNS, all of which arrive with
# `platform` + `platform-config`; and a leaf web server must never be able to stall the platform
# chain, which is the whole lesson of DECISIONS-LOG's "Headlamp gating stall". wait:false guarantees
# its health can never gate anything even if something is later mistakenly pointed dependsOn it.
#
# NO postBuild.substituteFrom — THE ONE DELIBERATE DIVERGENCE FROM EVERY OTHER TIER, AND THE MOST
# IMPORTANT LINE IN THIS FILE.
# This tier renders the runbook PROSE into a ConfigMap, and runbooks are full of shell: `$POD`,
# `$(kubectl ...)`, and — unavoidably, since they document these very annotations — the literal text
# `${BASE_DOMAIN}`. Flux's postBuild substitution runs envsubst over the ENTIRE built stream, and it
# FAILS OPEN: an undefined `${X}` becomes the empty string while the Kustomization still reports
# Ready (DECISIONS-LOG 2026-09-01 — that is how `k8sServiceHost` was emptied and Cilium broke on 2
# of 3 dev nodes). Enabled here it would silently mutilate the documentation: every braced expansion
# in every command example blanked, in a file nobody re-reads after writing. Worse, the L0 check
# `conformance/tests/l0-static/flux-postbuild-substitution` scans only files reachable through
# kustomization `resources:`, not `configMapGenerator` inputs, so the damage would be invisible to
# the gate as well.
# The fix is to have nothing to substitute. The only knob this tier would have wanted is the apex
# hostname, and runbooks/serving/httproute.yaml avoids needing it by declaring NO `hostnames:` and
# inheriting the `https-apex` listener's own hostname instead. So this tier is
# environment-independent by construction and R-ENV-02 ("authored once, promoted dev -> prod") holds
# without substitution rather than because of it.
# IF A FUTURE KNOB EVER GENUINELY NEEDS SUBSTITUTION HERE: do not add substituteFrom to this
# Kustomization. Split the serving manifests into their own substituted tier and leave the content
# ConfigMap in an unsubstituted one, or the runbooks lose their shell examples.
#
# NO SOPS decryption stanza: this tier carries no encrypted Secret (markdown plus five plain
# manifests), so — like homepage — neither clusters/dev nor clusters/prod patches it.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: runbooks
namespace: flux-system
spec:
interval: 10m
retryInterval: 2m
timeout: 5m
prune: true
wait: false
dependsOn:
- name: platform-config # wildcard TLS + Gateway (incl. the https-apex listener) first
sourceRef:
kind: GitRepository
name: flux-system
path: ./runbooks
conformance/tests/l0-static/alert-runbooks-resolvable/chainsaw-test.yaml
+89
View File
@@ -0,0 +1,89 @@
# =============================================================================
# L0 STATIC — every alert's runbook_url resolves to a runbook that is actually served
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# Every platform-health alert rule in this repo carries
# `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations across four files, over 34
# distinct names. For the whole of phases 4-6 there was NO runbooks/ directory and NO HTTPRoute serving
# that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404.
# So an alert reaching a human handed them a dead link at the one moment documentation is read, while the
# annotation made the problem look solved — the same green-surface-over-absent-mechanism class as the
# import Job that reported success while storing placeholder secrets, and the guard whose glob silently
# stopped matching.
#
# The link is served now (runbooks/, clusters/base/runbooks.yaml). This test is the guard that keeps it
# true, and it asserts BOTH halves of the claim because the regression is silent in both directions: an
# alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its
# configMapGenerator entry is in git, reviewed, merged — and still 404s in the cluster. "The file exists"
# and "the URL resolves" are different claims; only the second one helps an operator.
#
# GIVEN every `runbook_url` annotation on every PrometheusRule in the repo,
# WHEN each is parsed out of git and resolved against runbooks/,
# THEN (1) each URL has EXACTLY the shape the server resolves — https://${BASE_DOMAIN}/runbooks/<name>,
# the ${BASE_DOMAIN} token (never a literal hostname), no `.md`, no trailing slash, no query;
# (2) every referenced <name> has a runbooks/<name>.md in git;
# (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the
# dev-container kubectl invocation with --context k3s-prod — so a stub cannot satisfy (2) and
# make the link look fixed while still helping nobody;
# (4) every runbook is PROJECTED INTO THE SERVER by runbooks/kustomization.yaml's
# configMapGenerator under the extensionless key the URL asks for;
# (5) route, listener and server agree with the annotations (PathPrefix == the annotations' prefix,
# a sectionName that platform/gateway/gateway.yaml actually defines, the namespace opted in to
# the Gateway, and an nginx location for that same prefix with a text/plain default_type);
# (6) the Flux tier is wired into clusters/base AND declares NO postBuild.substituteFrom —
# substitution would run envsubst over the runbook prose and silently empty every `${...}`
# shell example in it (it fails open: DECISIONS-LOG 2026-09-01, the emptied Cilium
# k8sServiceHost), and the flux-postbuild-substitution check cannot see it because that check
# scans files reachable through `resources:`, not configMapGenerator inputs;
# AND (7) the check is LOAD-BEARING: this Test DECLARES
# `conformance.home.k3s/scope: whole-repo`, so ci/l0.sh Stage 3b's
# `--include-scope whole-repo` unions it into EVERY gate invocation rather than only the runs
# whose touched requirements happen to include R-OBS-04; and ci/gate.sh sets the merge block
# when the l0 stage fails. A SKIP of a scope-declaring check is UNWIRED, never green.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library only);
# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test wraps the SAME
# script.
#
# NOT proven (named in check.sh's header): this does not FETCH the URL. That it returns HTTP 200 with
# real content was proven by an in-cluster probe recorded in the commit message; a live probe needs a
# cluster, which L0 by definition does not have. What is asserted here is every static precondition for
# that 200. It also does not judge whether a runbook is correct or current — (3) is a floor, not a
# review — and it does not prove Alertmanager delivers the annotation to a human (that is the L3
# `alerting` test). R-OBS-04 is tagged for the narrow slice only: the platform's own health alerts carry
# runbook links that resolve to real, served procedure rather than to a 404.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: alert-runbooks-resolvable
labels:
conformance.home.k3s/R-OBS-04: "true"
annotations:
conformance.home.k3s/requirements: "R-OBS-04"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE — this is what makes the check unconditional. Stage 3b of ci/l0.sh runs
# `conformance/run.sh --layer l0 --include-scope whole-repo`, which UNIONS every test carrying this
# annotation into the selection regardless of which requirements the change touched. That union is
# load-bearing here: ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" —
# NOT R-OBS-04 — and maps platform/kubevirt/* and runbooks/* to nothing at all, so the single most
# likely breaking change (adding an alert with a runbook_url, or adding a runbook without its
# configMapGenerator entry) attributes a requirement set that would otherwise FILTER THIS CHECK OUT
# of exactly the change that introduces the bug. Declaring the scope replaces the hand-written
# per-test ci/l0.sh stage that this repo used to need (and that nobody remembered to add).
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every runbook_url on every alert rule resolves to a runbook that exists in git, clears a quality
floor, and is actually projected into the server that answers that path — so an alert can never
hand an operator a dead link.
spec:
steps:
- name: audit-every-alert-runbook-link
description: >-
Parse every runbook_url out of every PrometheusRule and assert the runbook exists, is real, is
served at that exact path, and that the serving tier is wired and unsubstituted.
try:
- script:
content: bash ./check.sh
conformance/tests/l0-static/alert-runbooks-resolvable/check.sh
Executable
+410
View File
@@ -0,0 +1,410 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — alert-runbooks-resolvable
# Serves: R-OBS-04 ("The platform alerts on its own health (nodes, storage, GitOps, certs,
# VPN)." — spec docs/10-observability-and-alerting.md)
#
# WHY THIS EXISTS — A DEAD LINK IN AN ALERT IS WORSE THAN NO LINK
# --------------------------------------------------------------
# Every platform-health alert rule in this repo carries
# runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
# 42 such annotations across four files, over 34 distinct names. For the whole of phases 4-6
# there was NO runbooks/ directory and NO HTTPRoute serving that path: a live in-cluster probe
# of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. So an alert reaching a
# human at 3am handed them a link that fails, at the one moment the documentation is read. The
# annotation LOOKED like the problem was solved, which is the specific failure this repo keeps
# finding (an import Job that reports success while storing placeholder secrets; a guard whose
# glob silently stopped matching) — a green-looking surface over an absent mechanism.
#
# The link is now served (runbooks/, clusters/base/runbooks.yaml). This check is the guard that
# keeps it true, because the regression is SILENT IN BOTH DIRECTIONS and nothing else catches it:
# * add an alert with a new runbook_url and forget the runbook -> that alert links to a 404,
# and no test, lint, render or policy notices;
# * write runbooks/<name>.md but forget to add it to the ConfigMap generator -> the file is in
# git, reviewable, apparently done, and the URL still 404s in the cluster. "The file exists"
# and "the URL resolves" are different claims and only the second one helps an operator.
# Both halves are asserted below, which is why this check reaches past the markdown into the
# kustomization, the HTTPRoute, the nginx config and the Flux tier.
#
# GIVEN the repository as authored,
# WHEN every `runbook_url` annotation on every PrometheusRule is parsed out of git,
# THEN (1) each URL has EXACTLY the shape the server resolves —
# https://${BASE_DOMAIN}/runbooks/<name>, `${BASE_DOMAIN}` as the token (never a literal
# hostname: a literal is how prod was authored to serve dev's hostnames, see the
# ServiceDefinition schema's note), no `.md` suffix, no trailing slash, no query;
# (2) every referenced <name> has a runbooks/<name>.md in git;
# (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the
# dev-container kubectl invocation — so a one-line stub cannot satisfy (2) and make the
# link look fixed while still helping nobody (the failure mode this check is most likely
# to be defeated by is a placeholder file, not a missing one);
# (4) every runbook markdown file is actually PROJECTED INTO THE SERVER by
# runbooks/kustomization.yaml's configMapGenerator, under the EXTENSIONLESS key the URL
# asks for — the "exists in git but 404s in the cluster" half;
# (5) the route and the server agree with the annotations: the HTTPRoute's PathPrefix is the
# prefix the URLs use, it attaches to a Gateway listener that actually exists, its
# namespace is opted in to that Gateway, and the nginx config answers that same prefix;
# (6) the Flux tier that reconciles all of it is wired into clusters/base AND declares NO
# postBuild.substituteFrom — substitution here would run envsubst over the runbook prose
# and silently empty every `${...}` shell example in it (it fails open: DECISIONS-LOG
# 2026-09-01, the emptied Cilium k8sServiceHost);
# AND (7) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and ci/gate.sh sets the
# merge block when the l0 stage fails.
#
# WHY (7) IS AN UNCONDITIONAL ci/l0.sh STAGE and not left to the requirement-scoped run: Stage 3b
# scopes `conformance/run.sh --layer l0` to the TOUCHED requirements (R-TEST-03), and
# ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" — NOT R-OBS-04 — and
# maps platform/kubevirt/* and runbooks/* to nothing at all. So the single most likely breaking
# change (adding an alert with a runbook_url to platform/monitoring/alerts.yaml, or adding a
# runbook and forgetting the generator entry) attributes a requirement set that would FILTER THIS
# CHECK OUT. Declaring `conformance.home.k3s/scope: whole-repo` makes ci/l0.sh Stage 3b union it into
# EVERY gate invocation instead; a SKIP of a scope-declaring check is UNWIRED, never green.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
# - It does not fetch the URL. That it returns HTTP 200 with real content was proven by an
# in-cluster probe (recorded in the commit message and DECISIONS-LOG); a live probe needs a
# cluster, which L0 by definition does not have. What is asserted here is every static
# precondition for that 200: the file, its projection, the key name, the route, the prefix.
# - It does not judge whether a runbook is CORRECT or current. (3) is a floor, not a review: it
# can tell a stub from a runbook, not a good runbook from a stale one. Nothing automatable can.
# - It does not prove Alertmanager actually delivers the annotation to a human. That is the L3
# `alerting` test.
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06) and is not consulted.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
# Honest SKIP (never green) if there are no alert rules at all — nothing to audit yet.
if ! grep -rl 'kind: PrometheusRule' "${ROOT}/platform" >/dev/null 2>&1; then
echo "check SKIPPED: no PrometheusRule found under platform/ — no alert rules to audit" >&2
exit 3
fi
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
"""Static audit: every alert runbook_url resolves to a runbook that is actually served.
Pure standard library, no PyYAML — the same constraint and the same style as the neighbouring
L0 checks (lib/coverage.py, lib/status.py). Every parse contract is documented at its parser;
all of them are line-constrained shapes in hand-authored manifests.
"""
from __future__ import annotations
import re
import sys
from pathlib import Path
ROOT = Path(sys.argv[1])
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
FAILURES: list[str] = []
ADVISORIES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def read(p: Path) -> str:
try:
return p.read_text()
except (OSError, UnicodeDecodeError):
return ""
# =============================================================== (1) parse the annotations
# Parse contract: `runbook_url:` is an annotation on a Prometheus alerting rule, authored one per
# line as `runbook_url: "<url>"` (single or double quoted, or bare). Only files that declare
# `kind: PrometheusRule` are scanned, so a runbook_url quoted inside this check or inside prose
# elsewhere in the repo is not mistaken for an alert annotation. Commented-out lines are skipped:
# platform/monitoring/alerts.yaml carries a `# runbook_url points at a docs base` header comment.
SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo"
RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""")
rule_files = sorted(
p for p in ROOT.rglob("*.yaml")
if ".git" not in p.parts and "kind: PrometheusRule" in read(p)
)
check(len(rule_files) > 0,
"no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
"this check would silently audit nothing")
# url -> [ "<relpath>:<lineno>", ... ]
refs: dict[str, list[str]] = {}
for f in rule_files:
rel = f.relative_to(ROOT)
for lineno, line in enumerate(read(f).splitlines(), 1):
if line.lstrip().startswith("#"):
continue
m = RUNBOOK_URL_RE.match(line)
if m:
refs.setdefault(m.group(1), []).append(f"{rel}:{lineno}")
n_annotations = sum(len(v) for v in refs.values())
check(n_annotations > 0,
"no runbook_url annotation parsed out of any PrometheusRule — the parser contract broke; "
"this check would pass vacuously while every alert link rots")
# ---- URL SHAPE. The server resolves exactly one shape. Anything else is a link that 404s even
# though a file with a plausible name exists, so shape is a failure, not a style note.
URL_RE = re.compile(r"^https://\$\{BASE_DOMAIN\}/(runbooks)/([a-z0-9][a-z0-9-]*)$")
names: dict[str, list[str]] = {} # runbook name -> annotation sites
prefixes: set[str] = set()
for url, sites in sorted(refs.items()):
m = URL_RE.match(url)
check(m is not None,
f"runbook_url '{url}' ({', '.join(sites)}) is not the shape this platform serves. "
f"Required: https://${{BASE_DOMAIN}}/runbooks/<name> — the ${{BASE_DOMAIN}} TOKEN (never a "
f"literal hostname: a literal is how prod got authored to serve dev's public names), a "
f"lowercase-hyphen <name>, and NO '.md' suffix, trailing slash or query. The runbook "
f"server maps the URL path straight onto an extensionless ConfigMap key, so any other "
f"shape is a 404 no matter which files exist")
if m:
prefixes.add(m.group(1))
names.setdefault(m.group(2), []).extend(sites)
check(len(prefixes) <= 1,
f"alert annotations use more than one runbook path prefix ({sorted(prefixes)}); the single "
f"HTTPRoute + nginx location can only serve one, so the others are dead links")
PREFIX = (sorted(prefixes) or ["runbooks"])[0]
# =================================================== (2)(3) the runbooks exist and are real
RUNBOOK_DIR = ROOT / PREFIX
check(RUNBOOK_DIR.is_dir(),
f"{PREFIX}/ is not a directory in this repo, so every one of the {n_annotations} "
f"runbook_url annotations is a dead link")
# The quality floor. (2) alone is satisfiable by `touch runbooks/node-down.md`, which makes the
# link resolve and still abandons the operator — a worse outcome than the 404, because the 404 at
# least told the truth. These four sections are the operator's path through an incident: what the
# alert means HERE, what to type first, what is probably wrong, and when to stop. `dev.sh` +
# `--context k3s-prod` are required because kubectl does not work on the workstation at all
# (the toolchain lives only in the pinned dev container) and because dev.sh defaults its
# current-context to k3s-DEV — a runbook that omits the context teaches the reader to answer
# questions about the wrong cluster during an incident on this one.
REQUIRED_SECTIONS = (
"## What this means here",
"## First commands",
"## Likely causes, most likely first",
"## When to escalate or stop",
)
MIN_BYTES = 1500
for name, sites in sorted(names.items()):
md = RUNBOOK_DIR / f"{name}.md"
if not md.is_file():
check(False,
f"runbook '{name}' does not exist: {md.relative_to(ROOT)} is missing, but "
f"{len(sites)} alert annotation(s) link to it ({', '.join(sites[:3])}"
f"{', ...' if len(sites) > 3 else ''}). An alert firing now hands the operator a 404. "
f"Write {PREFIX}/{name}.md (see {PREFIX}/README.md for the required sections)")
continue
body = read(md)
check(len(body) >= MIN_BYTES,
f"{md.relative_to(ROOT)} is {len(body)} bytes, under the {MIN_BYTES}-byte floor — a stub "
f"that makes the link resolve while still helping nobody is worse than the 404 it "
f"replaced, because it looks fixed")
for sec in REQUIRED_SECTIONS:
check(sec in body,
f"{md.relative_to(ROOT)} has no '{sec}' section — the four operator sections are what "
f"make this a runbook rather than a restatement of the alert name")
check("dev.sh" in body,
f"{md.relative_to(ROOT)} never mentions dev.sh — kubectl does not exist on the "
f"workstation; every command in a runbook here must be runnable as "
f"`infra/devcontainer/dev.sh run bash -c '...'` and the runbook must say so")
check("--context k3s-prod" in body,
f"{md.relative_to(ROOT)} never passes --context k3s-prod — dev.sh merges both clusters "
f"and picks a current-context PER INVOCATION — a default that has already changed once "
f"(k3s-dev until dev was decommissioned on 2026-09-13, k3s-prod now). A context-less command "
f"silently answers about whichever cluster the default happens to be, in the middle of an "
f"incident")
# ADVISORY, not a failure: a runbook nothing links to yet. Writing the runbook BEFORE the alert is
# the right order, and the repo's precedent for "authored but unreferenced" is advisory (see the
# dead-knob advisory in flux-postbuild-substitution/check.sh).
served_extra = {"README"}
if RUNBOOK_DIR.is_dir():
for md in sorted(RUNBOOK_DIR.glob("*.md")):
stem = md.stem
if stem in served_extra or stem in names:
continue
ADVISORIES.append(f"{md.relative_to(ROOT)} is a runbook no alert links to (written ahead of "
f"its alert, or the alert was removed — advisory, not a failure)")
# ================================= (4) every runbook is PROJECTED INTO THE SERVER, not just in git
# THIS IS THE HALF A FILE-EXISTENCE CHECK WOULD MISS. The pod serves a ConfigMap, not the git tree.
# Parse contract: runbooks/kustomization.yaml declares one configMapGenerator whose `files:` entries
# are `- <key>=<file>.md`, one per line. The key is EXTENSIONLESS on purpose (the URL carries no
# `.md`, and the ConfigMap volume mount turns each key into a file the nginx root resolves directly).
KUST = RUNBOOK_DIR / "kustomization.yaml"
check(KUST.is_file(), f"{PREFIX}/kustomization.yaml is missing — the Flux tier renders nothing and "
f"every runbook 404s no matter what is in git")
GEN_FILE_RE = re.compile(r"^\s*-\s*([A-Za-z0-9._-]+)=([A-Za-z0-9._/-]+\.md)\s*(?:#.*)?$")
generated: dict[str, str] = {}
kust_text = read(KUST)
if kust_text:
check("configMapGenerator:" in kust_text,
f"{PREFIX}/kustomization.yaml has no configMapGenerator — the runbook markdown is not "
f"rendered into the ConfigMap the server mounts")
for line in kust_text.splitlines():
if line.lstrip().startswith("#"):
continue
m = GEN_FILE_RE.match(line)
if m:
generated[m.group(1)] = m.group(2)
check(len(generated) > 0,
f"{PREFIX}/kustomization.yaml declares no `<key>=<file>.md` generator entries — the "
f"parser contract broke, or the markdown is being added with implicit keys (which would "
f"carry the '.md' suffix the alert URLs do NOT have, so every link would 404)")
for name in sorted(names):
check(generated.get(name) == f"{name}.md",
f"runbook '{name}' is not projected into the server: {PREFIX}/kustomization.yaml has no "
f"configMapGenerator entry `- {name}={name}.md`. The file can be in git, reviewed and "
f"merged and the URL still returns 404 — 'the file exists' and 'the URL resolves' are "
f"different claims")
# And the reverse, so a generator entry cannot point at a file that is not there.
for key, fname in sorted(generated.items()):
check((RUNBOOK_DIR / fname).is_file(),
f"{PREFIX}/kustomization.yaml generates key '{key}' from {PREFIX}/{fname}, which does not "
f"exist — `kustomize build` fails, so the WHOLE tier stops reconciling and every runbook "
f"goes stale at once")
# ============================== (5) the route, the listener and the server agree with the URLs
ROUTE = RUNBOOK_DIR / "serving" / "httproute.yaml"
NGINX = RUNBOOK_DIR / "serving" / "nginx-config.yaml"
NS = RUNBOOK_DIR / "serving" / "namespace.yaml"
GATEWAY = ROOT / "platform" / "gateway" / "gateway.yaml"
route_text = read(ROUTE)
check(bool(route_text), f"{PREFIX}/serving/httproute.yaml is missing — nothing routes the "
f"/{PREFIX} prefix to the server")
# Parse contract: a single HTTPRoute with one PathPrefix match and one parentRefs entry naming a
# listener via sectionName; hand-authored, one key per line.
m_path = re.search(r"^\s*type:\s*PathPrefix\s*$\n\s*value:\s*(\S+)\s*$", route_text, re.M)
check(m_path is not None and m_path.group(1) == f"/{PREFIX}",
f"{PREFIX}/serving/httproute.yaml does not match PathPrefix /{PREFIX} "
f"(found {m_path.group(1) if m_path else 'no PathPrefix match'}). The alert annotations point "
f"at /{PREFIX}/<name>; if the route's prefix drifts from the annotations the links 404 while "
f"both files look individually correct")
m_sec = re.search(r"^\s*sectionName:\s*(\S+)", route_text, re.M)
check(m_sec is not None,
f"{PREFIX}/serving/httproute.yaml declares no parentRefs sectionName. The apex host needs the "
f"dedicated apex listener: the wildcard `*.${{BASE_DOMAIN}}` listener does NOT match the bare "
f"apex, so a route without sectionName binds nothing and serves nothing")
if m_sec:
listener = m_sec.group(1)
gw = read(GATEWAY)
check(re.search(rf"^\s*-\s*name:\s*{re.escape(listener)}\s*$", gw, re.M) is not None,
f"{PREFIX}/serving/httproute.yaml attaches to Gateway listener '{listener}', which "
f"platform/gateway/gateway.yaml does not define — the route would report NotAllowed and "
f"never serve")
# An HTTPRoute with no `hostnames:` inherits the listener's hostname, which is how this tier avoids
# needing Flux substitution (see (6)). Either form is routable; a LITERAL hostname is not acceptable,
# for the same reason the schema forbids it in a ServiceDefinition.
m_host = re.search(r"^\s*hostnames:\s*$\n\s*-\s*(\S+)", route_text, re.M)
check(m_host is None or "${BASE_DOMAIN}" in m_host.group(1),
f"{PREFIX}/serving/httproute.yaml hardcodes hostname '{m_host.group(1) if m_host else ''}' — "
f"use the ${{BASE_DOMAIN}} token, or omit `hostnames:` entirely and inherit the listener's")
ns_text = read(NS)
check('gateway-access: "shared"' in ns_text or "gateway-access: shared" in ns_text,
f"{PREFIX}/serving/namespace.yaml does not carry gateway-access=shared — the shared Gateway "
f"selects namespaces by that label, so without it the HTTPRoute is refused attachment")
nginx_text = read(NGINX)
check(f"location /{PREFIX}/" in nginx_text,
f"{PREFIX}/serving/nginx-config.yaml has no `location /{PREFIX}/` block — the route would "
f"deliver the request and the server would answer 404 for every runbook")
check("default_type" in nginx_text and "text/plain" in nginx_text,
f"{PREFIX}/serving/nginx-config.yaml does not force a text/plain default_type. The ConfigMap "
f"keys are EXTENSIONLESS, so with the stock MIME table nginx serves them as "
f"application/octet-stream and a browser DOWNLOADS the runbook instead of showing it")
# ====================================== (6) the Flux tier is wired, and is NOT substituted
FLUX = ROOT / "clusters" / "base" / f"{PREFIX}.yaml"
BASE_KUST = ROOT / "clusters" / "base" / "kustomization.yaml"
flux_text = read(FLUX)
check(bool(flux_text),
f"clusters/base/{PREFIX}.yaml is missing — nothing reconciles {PREFIX}/ into the cluster, so "
f"the runbooks exist in git and the URL still 404s")
check(re.search(rf"^\s*path:\s*\./{re.escape(PREFIX)}\s*$", flux_text, re.M) is not None,
f"clusters/base/{PREFIX}.yaml does not set `path: ./{PREFIX}`")
check(re.search(r"^\s*wait:\s*false\s*$", flux_text, re.M) is not None,
f"clusters/base/{PREFIX}.yaml is not wait:false — a leaf web server that can be legitimately "
f"unhealthy must never gate the reconcile chain (DECISIONS-LOG: 'Headlamp gating stall')")
# Strip comments before testing: this file's own comment block EXPLAINS at length why
# substituteFrom must never be added here, and a naive substring test would fire on the warning.
flux_code = "\n".join(l for l in flux_text.splitlines() if not l.lstrip().startswith("#"))
check("substituteFrom" not in flux_code,
f"clusters/base/{PREFIX}.yaml declares postBuild.substituteFrom. It MUST NOT: this tier "
f"renders documentation into a ConfigMap, and Flux's envsubst would run over the prose and "
f"silently EMPTY every `${{...}}` shell example in every runbook — it fails open, so the "
f"Kustomization would still report Ready (DECISIONS-LOG 2026-09-01, the emptied Cilium "
f"k8sServiceHost). The flux-postbuild-substitution check cannot see this: it scans files "
f"reachable through `resources:`, not configMapGenerator inputs")
check(re.search(rf"^\s*-\s*{re.escape(PREFIX)}\.yaml\b", read(BASE_KUST), re.M) is not None,
f"clusters/base/kustomization.yaml does not list {PREFIX}.yaml — the Flux Kustomization is "
f"authored and never applied, which is the 'looks done, is absent' class this repo has been "
f"bitten by repeatedly")
# ============================================ (7) this check is load-bearing in the merge gate
L0 = read(ROOT / "ci" / "l0.sh")
GATE = read(ROOT / "ci" / "gate.sh")
# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism, in two halves.
# It is not enough that check.sh exists: the merge gate must SELECT it on every run. The repo used to
# get that with a hand-written "UNCONDITIONAL" stage per whole-repo check in ci/l0.sh, hardcoding the
# path; those stages are gone, because the next whole-repo invariant only got one if somebody
# REMEMBERED to write it. The property is now DECLARED by the test and HONOURED by the gate, so both
# halves are asserted — neither alone is sufficient. The mechanism itself (that the union really
# selects, and that a scope-declared SKIP exits 3 rather than passing green) is proven end-to-end and
# falsifiably by conformance/tests/l0-static/whole-repo-invariants-wired.
SELF_YAML = (ROOT / "conformance" / "tests" / "l0-static" / "alert-runbooks-resolvable"
/ "chainsaw-test.yaml")
SELF = read(SELF_YAML)
check(re.search(SCOPE_DECL_RE, SELF, re.M) is not None,
"alert-runbooks-resolvable/chainsaw-test.yaml does not declare "
"conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
"change happens to touch a requirement it is labelled with. ci/touched-requirements.sh maps "
"platform/monitoring/* to 'R-OBS-01 R-OBS-02' (NOT R-OBS-04) and maps platform/kubevirt/* and "
"runbooks/* to nothing at all — so adding an alert with a new runbook_url, or adding a runbook "
"without its configMapGenerator entry, would be filtered out of exactly the change that "
"introduces the dead link")
check("--include-scope" in L0,
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
"purely requirement-scoped run again and the scope annotation above is inert decoration — this "
"check would then run only by coincidence")
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
"ci/gate.sh does not set the merge block when the l0 stage fails")
# ======================================================================== report
print(f"Alert runbook inventory: {n_annotations} runbook_url annotation(s) across "
f"{len(rule_files)} PrometheusRule file(s) -> {len(names)} distinct runbook(s), "
f"prefix /{PREFIX}")
missing = sorted(n for n in names if not (RUNBOOK_DIR / f"{n}.md").is_file())
print(f" runbooks present: {len(names) - len(missing)}/{len(names)} "
f"projected into the ConfigMap: {sum(1 for n in names if generated.get(n) == f'{n}.md')}/{len(names)}")
if missing:
print(f" MISSING: {', '.join(missing)}", file=sys.stderr)
if ADVISORIES:
print("\nADVISORIES (not failures):")
for a in ADVISORIES:
print(f" - {a}")
print()
if not FAILURES:
print(f"check OK: alert-runbooks-resolvable — {CHECKS} assertions (R-OBS-04)")
raise SystemExit(0)
print(f"CHECK FAIL (alert-runbooks-resolvable): {len(FAILURES)} failed, "
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
for f in FAILURES:
print(f" FAIL {f}", file=sys.stderr)
raise SystemExit(1)
PY
runbooks/README.md
+142
View File
@@ -0,0 +1,142 @@
# Alert runbooks — home-network-k3s prod
This is the index page served at `https://home.keithjackson.dev/runbooks/`.
Every alert rule in this platform carries an annotation of the form
runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
46 of them, over the 35 names below. Until this directory existed, that path returned **HTTP 404** on the
live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
help. These pages are what makes the link resolve.
## How to run the commands in these runbooks
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang
is not the cluster being down.
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod`. `dev.sh` merges BOTH clusters into one kubeconfig and picks a
current-context per invocation, and **that default has already changed once** — k3s-dev until dev was
decommissioned on 2026-09-13, k3s-prod now. A command that names its context still means the same thing
next month; one that relies on the default does not, and the failure is silent. `CLUSTER=prod
infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner
naming the cluster — read it. Do not run two `dev.sh` invocations at once; they share one container.
**Use fully-qualified resource names, always.** Three CRDs on this cluster register the kind `Backup` —
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1`, `velero.io/v1` — and the bare short name resolves to
Longhorn's, which holds zero objects. `kubectl get backup -A` therefore prints **"No resources found"** on
a cluster holding 6 Velero and 27 CNPG backups (verified on prod 2026-09-13). Acting on that false
negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
`backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
`volumes.longhorn.io`, `nodes.longhorn.io`.
`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git.
## Alerts you should not trust the silence of (measured 2026-09-13)
Prometheus scrapes 19 jobs on prod. There is **no** Longhorn, cert-manager, Flux-controller or
Hubble/Cilium job, so these rule groups currently evaluate against nothing and **cannot fire**:
| Missing scrape job | Inert alerts |
|---|---|
| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornVolumeCapacityHigh`, `LonghornNodeStorageHigh` |
| cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` |
| Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
| Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
| etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.
## The runbooks
### Nodes
- **`node-down`** — a node's node-exporter is unscrapable; the node may be gone, or its kubelet may have
stopped posting status while the host stays up (the 2026-09-07 shape).
- **`node-io`** — disk I/O pressure/starvation. The etcd WAL shares a partition with Longhorn replicas;
this is the alert that would have called the 2026-09-07 outage and did not exist at the time.
- **`node-memory`** — memory pressure. Includes why `SwapTotal == 0` is not evidence swap is off.
- **`node-disk`** — a node filesystem filling up. `/data` is the one that matters.
- **`node-cpu`** — CPU PSI and run-queue depth, and how to tell CPU contention from I/O contention.
### Storage
- **`longhorn-degraded`** — a volume has lost redundancy. Normal after any reboot, and the rebuild is the
expensive part. Freeze/thaw commands here.
- **`longhorn-faulted`** — no usable replica. Data at risk; stop before anything destructive.
- **`longhorn-capacity`** — a volume or a node's Longhorn disk near capacity.
- **`pvc-capacity`** — PVC filling up / almost full / predicted to fill. These rules work.
- **`monitoring-capacity`** — Prometheus or Loki running out of disk; don't let the monitor blind itself.
### GitOps and updates
- **`flux-reconcile`** — a Flux resource stuck at Ready=False. Which tier decides the blast radius.
- **`flux-suspended`** — reconciliation turned off and forgotten. Drift becomes silent.
- **`update-rollback`** — a HelmRelease failed its health gate and is rolling back.
- **`image-automation`** — the git write-back engine is failing (feature is disabled here).
- **`image-policy-stalled`** — an ImagePolicy/ImageRepository cannot resolve a tag.
### Certificates
- **`cert-issuance`** — ACME issuance failing. The DNS-01 self-check is the usual cause.
- **`cert-expiry`** — a certificate is expiring. Almost always a late symptom of `cert-issuance`.
### Networking
- **`vpn-down`** — the egress VPN tunnel. Includes the silent egress-gateway staleness that this alert
cannot see, and what the kill-switch actually does.
- **`denied-flows`** — policy-denied flow spike. The Cilium hairpin/identity regression class.
### Backups and DR
- **`backup-failure`** — a Velero backup failed. The R2 object-tagging incompatibility is documented here.
- **`backup-partial`** — a Velero backup completed with partial failures, and which skips are deliberate.
- **`backup-missed`** — no successful Velero backup in 24h.
- **`cnpg-archiving`** — WAL archiving stuck / no recoverability point. **An availability incident, not
just a backup one.**
- **`cnpg-backup-missed`** — no recent CNPG base backup. Velero does not cover these databases.
- **`cnpg-wal-growth`** — `pg_wal` not being recycled. Never delete WAL by hand.
- **`cnpg-exporter`** — the CNPG SQL exporter erroring: a blind spot that looks exactly like health.
- **`cnpg-primary-spread`** — two or more CNPG primaries on one node. Blast radius, not an outage,
and the rebalance costs ~9s of writes per database — one cluster at a time.
### Identity
- **`keycloak-down`** — the IdP is down. Blast radius, and the break-glass path that still works.
- **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
every green signal missed.
### Virtualisation
- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
- **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps.
- **`kubevirt-crashloop`** — a virt-launcher pod restarting repeatedly.
- **`cdi-import-stuck`** — a DataVolume import not progressing. Check node I/O before restarting it.
## Editing a runbook
Each page is `runbooks/<name>.md` in the `home-network-k3s-impl` repo. Edit it, push to `main`, and Flux
re-renders the ConfigMap and rolls the two-replica server; kustomize's content hash on the generated
ConfigMap name is what triggers the roll.
Adding a **new** alert with a `runbook_url` requires the matching `runbooks/<name>.md` **and** its
`<name>=<name>.md` entry in `runbooks/kustomization.yaml`. Both are enforced by
`conformance/tests/l0-static/alert-runbooks-resolvable`, which runs unconditionally in the merge gate —
"the file exists in git" and "the URL resolves in the cluster" are different claims, and it checks both.
Every page must carry the four operator sections (*What this means here*, *First commands*, *Likely
causes, most likely first*, *When to escalate or stop*), mention `dev.sh`, and pass `--context k3s-prod`.
The check enforces that too, because a stub that makes the link resolve while helping nobody is worse than
the 404 it replaced — it looks fixed.
## Why this route has no login
Deliberately public. The UIs that declare forward-auth (pihole, jackett, qbittorrent, radarr, sonarr)
really are closed: enforcement is per-pod, each one's HTTPRoute targeting its oauth2-proxy sidecar on
:4180 rather than the app port. If
these pages did too, `keycloak-down` would be unreadable exactly when Keycloak is down,
`keycloak-db` unreadable when its database is down, and `keycloak-client-secrets` unreadable during the
precise failure that broke all 12 OIDC clients on 2026-09-06 behind an entirely green cluster. An
operator who cannot log in is this route's primary reader. The content is operational procedure — metric
names, thresholds, kubectl invocations, topology — and no secrets. See
`runbooks/serving/httproute.yaml` for the full reasoning, including how forward-auth is actually
enforced here and the instruction to leave this route **out** of any Gateway-level ext-authz attachment
if one is ever introduced.
The running narrative of real incidents on this cluster is `DECISIONS-LOG.md`.
runbooks/backup-failure.md
+173
View File
@@ -0,0 +1,173 @@
# backup-failure — A Velero backup failed
**Alert:** `VeleroBackupFailure` (critical, 5m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups`
**Expression:** `increase(velero_backup_failure_total[1h]) > 0`
## What this means here
A Velero backup in the last hour ended in `phase=Failed`. Off-cluster DR (R-STATE-03) is at risk. Velero
backs up the **cluster's Kubernetes objects and its Longhorn PVC data**; it does **not** provide the DR
path for the three CloudNativePG databases — CNPG owns those via its own Barman archive, because a
Velero snapshot of a live Postgres is not a backup. So a Velero failure and a CNPG failure are separate
incidents with separate runbooks (`/runbooks/cnpg-archiving`).
## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE
`kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**.
Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and
`velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified
on prod 2026-09-13, where the fully-qualified queries returned **6 Velero backups** and **27 CNPG
backups** while the short name returned nothing.
This has already caused two false "the backups are missing" conclusions in one night. **A destructive
recovery action taken on that false negative is how a monitoring quirk becomes real data loss.** Use:
```
backups.velero.io backups.postgresql.cnpg.io backups.longhorn.io
schedules.velero.io scheduledbackups.postgresql.cnpg.io clusters.postgresql.cnpg.io
```
## Where the backups go
Cloudflare **R2**, since the operator's 2026-09-05 decision (commit `32d6766`). Prod cannot reach the
on-Proxmox MinIO at all — measured from a prod pod, *connection timed out after 15s* — and that MinIO
ran on the same physical host as dev's VMs, so dev's "off-cluster" backup shared a failure domain.
Live config: endpoint `https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`, region
**`auto`**, bucket `k3s-prod-backup`, prefix `velero`, schedule `0 3 * * *`, retention `720h`.
`backupstoragelocation.velero.io/default` should read **Available**.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Which backup failed, and why — fully-qualified, always:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get backups.velero.io -A \
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,START:.status.startTimestamp,ERRORS:.status.errors,WARN:.status.warnings'
```
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n velero get backups.velero.io <name> -o yaml | sed -n "/status:/,\$p"'
```
The storage location and the controller's log (the R2 error text is in the log, not the CR):
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n velero get backupstoragelocations.velero.io,schedules.velero.io; echo; \
kubectl --context k3s-prod -n velero logs deploy/velero --tail=150 2>&1 | grep -iE "error|NotImplemented|InvalidArgument|tagging|checksum" | tail -25'
```
Did the **volume data** upload even though the backup failed? (This distinction is the whole point —
see cause 1.)
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n velero get podvolumebackups.velero.io \
-o custom-columns=NAME:.metadata.name,BACKUP:.metadata.labels.velero\\.io/backup-name,PHASE:.status.phase,BYTES:.status.progress.bytesDone'
```
The plugin pin (cause 1 depends entirely on this):
```
grep -n "velero-plugin-for-aws\|checksumAlgorithm\|region" platform/velero/helmrelease.yaml
```
## Likely causes, most likely first
### 1. The R2 object-tagging incompatibility (the known, specific, repo-documented one)
`velero-plugin-for-aws` **v1.14.2** sets PutObject's Tagging unconditionally —
`Tagging: aws.String(o.tagging)` with `o.tagging == ""` — so the SDK emits `x-amz-tagging: ` (empty) on
**every** PutObject. **R2 does not implement object tagging at all** and answers:
```
NotImplemented: Header 'x-amz-tagging' with value '' not implemented
```
Observed live on prod **twice**, on the write of `velero/backups/<name>/velero-backup.json`. The shape
of the failure is the dangerous part: **kopia uploads the volume DATA successfully (its own S3 client),
then the Backup goes `phase=Failed` with NO metadata object — an unrestorable backup.**
**No configuration avoids it** (`config.tagging` with a real value sends a tag set R2 also rejects). The
fix is upstream PR #309, which landed on release-1.14 *after* v1.14.2 was cut, so the image is pinned to
**`velero/velero-plugin-for-aws:v1.14.3-rc.2`**.
**EXIT CONDITION: move to v1.14.3 the moment it ships GA. Do NOT "clean this up" back to v1.14.2 — that
silently breaks every prod backup's metadata.** If this alert fires and the pin has drifted, that is
your answer.
Two more mandatory R2 settings in the same file: `config.checksumAlgorithm: ""` (R2 rejects the AWS
SDK's `x-amz-checksum-*` headers with **HTTP 400 InvalidArgument** — harmless against MinIO, fatal
against R2) and `region` exactly **`auto`** (it was once hardcoded `us-east-1`, correct-by-accident for
MinIO).
### 2. A placeholder or wrong endpoint
`clusters/prod` deliberately carried the **loud** placeholder
`https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` rather than inheriting the base (dev) value,
because *"an omitted op means prod silently inherits a dev address — far worse than a failed apply"*.
Check `S3_BACKUP_ENDPOINT`, `S3_BACKUP_REGION`, `VELERO_BACKUP_BUCKET`, `VELERO_BACKUP_PREFIX` and
`BACKUP_CREDENTIALS_SUFFIX` in `clusters/base/cluster-vars.yaml` and the prod overlay. A
`backupstoragelocation` that is not `Available` points straight here.
### 3. Credentials
The R2 token lives in a SOPS-encrypted Secret (`velero-cloud-prod.sops.yaml`), and the `velero` tier
carries its own `decryption` stanza. An expired or mis-scoped token gives 403s in the log.
### 4. A volume could not be read
Check `podvolumebackups.velero.io` for a specific volume. Note that NFS-sourced volumes are
**deliberately skipped** by the `velero-volume-policies` ConfigMap — see `/runbooks/backup-partial`,
which is where that belongs.
### 5. The node or storage was unhealthy during the run
Velero's daily runs at 03:00 and reads Longhorn PVCs. If a node was under I/O pressure or a volume was
degraded, the read can fail. Cross-read `/runbooks/node-io` and `/runbooks/longhorn-degraded`.
## When to escalate or stop
- **Escalate promptly.** This is a `critical` because it is a DR alarm: the cluster may currently have no
recent restorable backup.
- **Do not delete a Failed backup to "clean up".** A Failed Velero backup can leave **orphaned kopia
blobs** in the bucket — prod has two such backups from 2026-09-12 (`prod-proof-20260912`,
`prod-vp-probe-20260912`) whose orphaned blobs are **pending owner consent to delete**. Deleting
objects in R2 is an operator decision, not a tidy-up.
- **Do not assume a Completed backup is restorable** if the plugin pin has drifted — the failure mode is
precisely "data uploaded, metadata missing".
- **Be honest about restore**: `dr-restore` is **authored and has never been executed**
(`conformance.home.k3s/never-executed`), so the R2 restore path is unrehearsed. R2 egress is free,
which is the whole reason restore rehearsals are affordable — but nobody has done one yet.
- The three CNPG databases are **not** covered by Velero. If the concern is a database, go to
`/runbooks/cnpg-archiving`.
---
Source of truth: `runbooks/backup-failure.md`. Index: `/runbooks/`
Records: commit `2e259a8` and `platform/velero/helmrelease.yaml` lines ~68-95 (the R2 tagging
incompatibility, `checksumAlgorithm`, `region: auto`); commit `32d6766` (the move to R2); commit
`2d5171e` (Velero shipped with no ServiceMonitor, so these alerts could never fire).
runbooks/backup-missed.md
+161
View File
@@ -0,0 +1,161 @@
# backup-missed — No successful Velero backup in 24 hours
**Alert:** `VeleroNoRecentBackup` (critical, 1h)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups`
**Expression:** `(time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)`
## What this means here
A Velero **schedule** has not produced a successful backup in over a day. This is the alert that catches
the quiet failure — not "a backup broke" (that is `/runbooks/backup-failure`) but "backups stopped
happening and nobody noticed". The `{schedule!=""}` filter means it only watches scheduled backups, so
ad-hoc ones do not mask a dead schedule.
Live shape on prod: two schedules, both `0 3 * * *` —
`schedule.velero.io/velero-daily` and `schedule.velero.io/velero-daily-resources-only` — retention
`720h`, target Cloudflare R2 (bucket `k3s-prod-backup`, prefix `velero`, region `auto`).
**Why this alert is worth its `critical`:** the precedent is that these rules *could not fire at all*.
Velero shipped with three backup alerts and the chart default `metrics.serviceMonitor.enabled: false`,
confirmed live on prod — `kubectl -n velero get servicemonitor` returned no resources. Recorded verbatim:
*"A nightly backup could fail — or silently stop running — for weeks with nothing to say so."* The
ServiceMonitor now exists (`velero` and `velero/node-agent` are both live scrape jobs as of 2026-09-13),
which is the only reason this alert is real.
## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE
`kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**.
Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and
`velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified
on prod 2026-09-13: the fully-qualified queries returned **6 Velero backups** and **27 CNPG backups**
while the short name returned nothing. This false negative has already produced two wrong "the backups
are missing" conclusions in one night. **A destructive recovery action taken on it is how a CLI quirk
becomes real data loss.** Always:
```
backups.velero.io backups.postgresql.cnpg.io backups.longhorn.io
schedules.velero.io scheduledbackups.postgresql.cnpg.io clusters.postgresql.cnpg.io
```
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Are the schedules enabled, and when did each last run?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n velero get schedules.velero.io \
-o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,PAUSED:.spec.paused,LASTBACKUP:.status.lastBackup,PHASE:.status.phase'
```
What backups actually exist (fully-qualified — the short name lies):
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get backups.velero.io -A \
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,SCHEDULE:.metadata.labels.velero\\.io/schedule-name,START:.status.startTimestamp --sort-by=.status.startTimestamp'
```
Is the storage location usable at all?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n velero get backupstoragelocations.velero.io \
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,LASTVALIDATED:.status.lastValidationTime,DEFAULT:.spec.default'
```
Is Velero's own Flux tier reconciling, or was it suspended?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
-o custom-columns=NAME:.metadata.name,SUSPENDED:.spec.suspend,READY:.status.conditions[?\(@.type==\"Ready\"\)].status | grep -Ei "velero|NAME"'
```
And the controller:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n velero get pods; \
kubectl --context k3s-prod -n velero logs deploy/velero --tail=120 2>&1 | tail -40'
```
## Likely causes, most likely first
### 1. The Velero Flux tier was suspended and nobody resumed it
This has happened: **Velero was found suspended live, with the suspension never recorded in git**, and
was resumed with `flux resume`. A suspended backup controller is a silent DR gap and is precisely the
class this alert exists for. Remediation (operator): `flux resume kustomization velero`, and see
`/runbooks/flux-suspended` for why an undocumented live suspension is itself a problem.
### 2. The schedule is paused
`schedules.velero.io` has `spec.paused`. Note from the live listing that
`velero-daily-resources-only` shows an empty PAUSED column while `velero-daily` shows `false` — read the
field, not the column width. A paused schedule is a git change to unpause.
### 3. Every run is failing rather than not running
Then `backups.velero.io` will show recent `Failed` entries and this alert is a downstream symptom. Go to
`/runbooks/backup-failure` — and in particular check the plugin pin
(`velero/velero-plugin-for-aws:v1.14.3-rc.2`), because the R2 tagging bug produces exactly this: volume
data uploads, the backup goes Failed, and no successful backup ever lands.
### 4. The BackupStorageLocation is not Available
A wrong endpoint, a bad credential, or the placeholder. `clusters/prod` deliberately carried the loud
placeholder `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` rather than inheriting the base (dev)
value, because *"an omitted op means prod silently inherits a dev address — far worse than a failed
apply"*. Check `S3_BACKUP_ENDPOINT`, `S3_BACKUP_REGION` (`auto`), `VELERO_BACKUP_BUCKET`,
`VELERO_BACKUP_PREFIX`, `BACKUP_CREDENTIALS_SUFFIX` (`-prod`).
### 5. Velero cannot schedule a pod
Node pressure, or the node-agent DaemonSet unhealthy. Cross-read `/runbooks/node-io`.
### 6. The metric, not the backup, is missing
If `velero_backup_last_successful_timestamp` has no series at all the rule cannot fire — which is the
historical state. Confirm the `velero` scrape job is up (`/runbooks/targets-down`) before trusting either
its silence or its firing.
## When to escalate or stop
- **Escalate: this is a DR alarm and 24h is already a real window of exposure.**
- **Do not create an ad-hoc backup and consider the alert answered.** The expression filters
`{schedule!=""}`, so a manual backup will not clear it, and more importantly it does not fix the
schedule. (An operator's ad-hoc run for reassurance is
`velero backup create --from-schedule velero-daily`.)
- **Do not conclude "no backups exist" from `kubectl get backup`.** See the trap above.
- **Remember the databases are separate.** `keycloak-db`, `nextcloud-db` and `semaphore-db` are covered
by CNPG's Barman archive, not by Velero. This alert being clear says nothing about them, and vice
versa — check `/runbooks/cnpg-backup-missed` too.
- **Be honest about restore:** `dr-restore` is authored and has **never been executed**, so the R2
restore path is unrehearsed. Free R2 egress makes a rehearsal cheap; it is still outstanding.
---
Source of truth: `runbooks/backup-missed.md`. Index: `/runbooks/`
Records: commit `2d5171e` (Velero had no ServiceMonitor, so these alerts could never fire);
`DECISIONS-LOG.md` (Velero found suspended with the suspension never in git; `dr-restore` never
executed); commit `32d6766` (R2 decision and endpoint knobs).
runbooks/backup-partial.md
+164
View File
@@ -0,0 +1,164 @@
# backup-partial — A Velero backup completed with partial failures
**Alert:** `VeleroBackupPartialFailure` (warning, 5m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups`
**Expression:** `increase(velero_backup_partial_failure_total[1h]) > 0`
## What this means here
The backup finished, but some resources or volumes were skipped or errored. It is a warning rather than a
critical because *most of it worked* — and that is exactly what makes it dangerous: a
`PartiallyFailed` backup looks like a backup in every listing, and the thing it silently omitted may be
the thing you need.
**The first question is always: was the skip deliberate?** On this cluster a great deal of skipping is
intentional and correct.
## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE
`kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**.
Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and
`velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified
on prod 2026-09-13: the fully-qualified queries returned **6 Velero backups** and **27 CNPG backups**
while the short name returned nothing. This false negative has already produced two wrong "the backups
are missing" conclusions in one night. **A destructive recovery action taken on it is how a CLI quirk
becomes real data loss.** Always:
```
backups.velero.io backups.postgresql.cnpg.io backups.longhorn.io
schedules.velero.io scheduledbackups.postgresql.cnpg.io clusters.postgresql.cnpg.io
```
## The deliberate exclusions (know these before calling anything a failure)
**NFS-sourced volumes are skipped on purpose, and the reason is a near-miss worth remembering.** The
opt-out file-system-backup schedule (`defaultVolumesToFsBackup: true`, excluding only `monitoring` and
`unifi`) would have read the NFS media library **three times** — `plex-media`, `sonarr-media` and
`radarr-media` are all bound to the same `192.168.2.102:/mnt/Storage/plex` export, **8.32 TiB
measured** — plus `nextcloud-legacy-archive` (270 GiB), pushing roughly **8.6 TiB to R2 nightly with
720h retention**. Namespace exclusion could not express the fix, because those namespaces also hold real
Longhorn PVCs that must be backed up.
The fix is a **per-volume opt-out by SOURCE TYPE**: the ConfigMap `velero-volume-policies` skips any
volume whose source is NFS, and the schedule references it. The per-pod
`backup-volumes-excludes` annotation was **rejected for failing OPEN** (forget the annotation and you
back up 8 TiB). Proven on prod: a Backup of namespace `nextcloud` produced exactly **one**
PodVolumeBackup (`data`, Longhorn, 875,417,102 bytes, Completed) and the NFS volume produced none.
**Residual gap, stated honestly: the rule matches NFS only.** A future non-NFS bulk source would not be
covered.
**The three CNPG databases are also out of scope by design.** `keycloak-db`, `nextcloud-db` and
`semaphore-db` have their own DR path (Barman WAL archive + base backups to R2) because a Velero snapshot
of a live Postgres is not a backup. If a `*-db-N` volume is skipped, that is correct — see
`/runbooks/cnpg-archiving` for the path that does cover them.
**`monitoring` and `unifi` are excluded** from file-system backup as well.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Which backup, and what exactly it warned about:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get backups.velero.io -A \
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,ERRORS:.status.errors,WARN:.status.warnings,START:.status.startTimestamp'
```
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n velero get backups.velero.io <name> -o yaml | sed -n "/status:/,\$p"'
```
Per-volume results — this tells you whether a skip was the NFS policy or a real failure:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n velero get podvolumebackups.velero.io \
-o custom-columns=NAME:.metadata.name,BACKUP:.metadata.labels.velero\\.io/backup-name,POD:.spec.pod.name,VOL:.spec.volume,PHASE:.status.phase,BYTES:.status.progress.bytesDone'
```
The volume policy that does the deliberate skipping:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n velero get cm velero-volume-policies -o yaml'
```
The controller's own account of what it skipped:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n velero logs deploy/velero --tail=200 2>&1 | grep -iE "skip|excluded|warn|error" | tail -30'
```
## Likely causes, most likely first
### 1. The deliberate NFS / CNPG / monitoring exclusions
Most common by far, and correct. Confirm against the `velero-volume-policies` ConfigMap and the
schedule's excluded namespaces. **No action.** If you are tempted to "fix" it, re-read the 8.6 TiB
near-miss above.
### 2. A resource Velero could not serialise
Usually a CRD whose API was unavailable at the moment of the backup, or an admission webhook refusing a
read. The log names the group/kind. Often transient; if it recurs for the same kind, that kind is
genuinely not backed up.
### 3. A volume was busy, degraded, or on a node under pressure
Velero's daily runs at 03:00 against Longhorn PVCs. A degraded volume or a node with high I/O-PSI can
fail one volume's read while the rest succeed. Cross-read `/runbooks/longhorn-degraded` and
`/runbooks/node-io`.
### 4. A partial R2 write
Related to the tagging/checksum incompatibilities in `/runbooks/backup-failure`: those normally produce a
hard `Failed`, but a partially-written backup is worth checking the plugin pin for
(`velero/velero-plugin-for-aws:v1.14.3-rc.2` — do **not** revert to v1.14.2).
### 5. A namespace was created or deleted mid-backup
Benign race. Leaked test namespaces have existed on this cluster and been deleted; a namespace
disappearing under Velero produces exactly this warning.
## When to escalate or stop
- **Decide, and record, whether the skipped items matter.** The honest failure mode here is shrugging at
a `PartiallyFailed` and discovering later that something was never covered. If the skip was not one of
the documented exclusions, treat it as an uncovered resource.
- **Do not add namespaces to the exclusion list to quiet the alert.** Those namespaces hold real
Longhorn PVCs; namespace-level exclusion is the mistake the source-type policy exists to avoid.
- **Do not switch to the per-pod `backup-volumes-excludes` annotation** — it was rejected for failing
OPEN, and the consequence of forgetting it is 8 TiB per night.
- **Do not delete a PartiallyFailed backup** — like a Failed one it can leave orphaned kopia blobs, and
deleting R2 objects is an operator decision.
- Escalate if the same resource or volume is skipped on consecutive nights: that is a coverage gap, which
is an R-STATE-03 matter and not a transient.
---
Source of truth: `runbooks/backup-partial.md`. Index: `/runbooks/`
Records: commit `2e259a8` (the 8.6 TiB near-miss, the source-type volume policy, and the rejection of the
fail-open annotation); `docs/BULK-NFS-STORAGE.md`.
runbooks/cdi-import-stuck.md
+172
View File
@@ -0,0 +1,172 @@
# cdi-import-stuck — A CDI DataVolume import is not progressing
**Alert:** `CDIDataVolumeImportStuck` (warning, 30m)
**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt`
**Expression:** `sum(kubevirt_cdi_import_pods_high_restart) > 0`
## What this means here
A CDI DataVolume import or clone has not completed after 30 minutes, so **the VM disk it backs cannot
start**. `kubevirt_cdi_import_pods_high_restart` counts importer pods stuck restarting; a sustained
positive value means the import is not making progress rather than merely being slow.
The 30-minute `for` is generous on purpose: on this hardware a genuine large import is legitimately slow.
The reference case is the **UniFi VM's 60 GiB import**, and the `apps-vms` tier is deliberately
**non-gating** (`wait: false`, `dependsOn: platform-config`) precisely because a VM's image import plus
in-guest install is legitimately long-not-Ready.
**Two things to know before triaging:**
1. **Imports are the single heaviest write this platform performs**, and the etcd WAL (`/data/k3s`) shares
one partition with every Longhorn replica (`/data/longhorn`). A 60 GiB import is exactly the class of
write load behind the 2026-09-07 `k3s-prod-1` NotReady outage. So "restart the import" is not a free
action — check `/runbooks/node-io` first.
2. **There is a known false-positive precedent in this area.** A broken HTTP CDI ServiceMonitor once caused
a false `CDIOperatorDown` (commit `6a7e9d1`), and the sibling `KubeVirtControlPlaneDown` once used a
non-existent metric with the wrong job label (`b57b3e9`). The metric name in this rule is flagged in
the file as one to confirm against the deployed CDI. `cdi-prometheus-metrics` **is** a live scrape job
on prod (measured 2026-09-13), so the target is up — but verify the series exists before trusting either
this alert's firing or its silence.
Platform versions: **CDI v1.66.0** with **KubeVirt v1.9.0**, on real nested KVM.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Which DataVolume, and how far it got:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get datavolumes.cdi.kubevirt.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PHASE:.status.phase,PROGRESS:.status.progress,RESTARTS:.status.restartCount'
```
The importer pod is where the real error is:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get pods -A | grep -E "importer|upload|clone"; echo; \
kubectl --context k3s-prod -n unifi logs -l app=containerized-data-importer --tail=100 2>&1 | tail -50'
```
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n unifi describe pod -l app=containerized-data-importer 2>&1 | sed -n "/Events:/,\$p"'
```
Is there anywhere to put it — and is the scratch class configured?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get pvc -A | grep -Ei "scratch|dv|rootdisk"; echo; \
kubectl --context k3s-prod get sc; echo; \
kubectl --context k3s-prod get cdis.cdi.kubevirt.io -o yaml | grep -A5 scratchSpace'
```
Is the node being flattened by the import? **Check this before restarting anything:**
```
ssh k3s-prod-1 'cat /proc/pressure/io; df -h /data'
```
Can the importer reach the image source at all? (Default-deny is the baseline everywhere.)
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
hubble observe --verdict DROPPED --last 100 -o compact | grep -i import'
```
Does the metric this alert depends on actually exist?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
for q in "up{job=\"cdi-prometheus-metrics\"}" "count(kubevirt_cdi_import_pods_high_restart)"; do \
printf "%-50s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \
| python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done'
```
## Likely causes, most likely first
### 1. Not enough schedulable storage for the target or the scratch space
The recorded failure, and the reason a whole StorageClass exists: **the 3-replica default class failed the
UniFi 60Gi import with insufficient storage / `ReplicaSchedulingFailure`.** The fix was a dedicated
single-replica **`longhorn-scratch`** class (`numberOfReplicas: 1`, `dataLocality: best-effort`),
referenced by the CDI CR's `scratchSpaceStorageClass`. If scratch is pointed at the 3-replica default,
this is your cause. Note that a 60 GiB import needing 3x replication plus scratch space is a very
different ask from 60 GiB.
### 2. An egress or default-deny block on the image source
Import pods are ordinary pods under the cluster's default-deny baseline (R-NET-01). The alert's own
description names this: *"often an egress/default-deny block on the image source, or a slow mirror."*
Remediation is an explicit allow in git; see `/runbooks/denied-flows` — and in particular the
post-Cilium-upgrade finding that `toEntities: world` no longer covers a destination that resolves to an
in-cluster backend.
### 3. The importer cannot open the raw Block device
`nonroot-devices: true` is set on the k3s servers because the **non-root CDI importer must open a raw
Block device that Kubernetes never chowns for it** (commit `df5979f`). A permission error in the importer
log against a Block-mode PVC is this. `longhorn-vm` is Block-mode only, so VM disks are always in scope.
### 4. The source is slow, unreachable, or the URL is wrong
A 404/403 or a stalled transfer appears plainly in the importer log. A slow mirror is a genuine cause — the
30m `for` already allows for it, so check throughput rather than assuming a fault.
### 5. Node I/O pressure is starving the import
Self-reinforcing: the import is heavy, the disk is shared with etcd, and a node under I/O pressure slows
the import which then runs longer. `/runbooks/node-io`.
### 6. Longhorn is degraded
An import onto a degraded volume competes with the rebuild. `/runbooks/longhorn-degraded`.
### 7. The alert is wrong
Given the CDI/KubeVirt false-positive precedents, if the DataVolumes all read `Succeeded` this is a
monitoring fault. Fixing the rule is a real change, not a shrug — a permanently-firing alert is a
recorded, expensive failure mode here.
## When to escalate or stop
- **Check node I/O pressure BEFORE restarting or recreating an import.** A 60 GiB re-import is hours of
write load on the partition the embedded etcd fsyncs to, and that combination took a node out of the
cluster on 2026-09-07. If I/O-PSI is already elevated, wait or freeze Longhorn rebuilds first
(`/runbooks/longhorn-degraded`).
- **Do not delete the target DataVolume/PVC casually** — that guarantees a full re-import.
- **Do not point `scratchSpaceStorageClass` at the 3-replica default** to "keep things consistent". The
single-replica class exists because the default one demonstrably failed.
- **Do not widen a NetworkPolicy to `world`** to let an import out; author the narrow allow.
- Scope it honestly: the VM tier is non-gating, so a stuck import does not affect the container platform.
The exception is the UniFi VM, which is the home network's controller.
- Escalate for k3s server-flag changes (`nonroot-devices`) or CDI CR changes — Ansible and operator work.
---
Source of truth: `runbooks/cdi-import-stuck.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` (the UniFi 60Gi import failing on the 3-replica default; `longhorn-scratch`);
commits `df5979f` (`nonroot-devices`), `6a7e9d1` and `b57b3e9` (false CDI/KubeVirt alerts).
runbooks/cert-expiry.md
+146
View File
@@ -0,0 +1,146 @@
# cert-expiry — A certificate is expiring soon, or imminently
**Alerts:** `CertificateExpiringSoon` (warning, <14d, 1h) and `CertificateExpiryImminent` (critical, <3d, 1h)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.certificates`
**Expression:** `(certmanager_certificate_expiration_timestamp_seconds - time()) < (14 * 24 * 3600)` / `(3 * 24 * 3600)`
## What this means here
cert-manager has a Certificate whose expiry is inside 14 days (warning) or 3 days (critical) and it has
not renewed. cert-manager normally renews at two-thirds of lifetime — about 60 days into a 90-day Let's
Encrypt certificate — so **by the time this fires, renewal has already been failing for weeks**. Treat
it as a late symptom of `/runbooks/cert-issuance`, not as a separate problem: the fix is almost always
"why is issuance failing", and expiry is the clock running out on that.
What is at stake concretely. There is effectively **one certificate that matters**: `wildcard-home` in
the `gateway` namespace, secret `wildcard-home-tls`, requesting **both** `*.${BASE_DOMAIN}` **and**
`${BASE_DOMAIN}` from `${ACME_CLUSTER_ISSUER}` (= `letsencrypt-prod` on prod). Every HTTPS listener on
the shared Gateway terminates with it — the wildcard `https` listener and the `https-apex` listener both
reference `wildcard-home-tls`. If it expires:
- every public hostname serves an untrusted certificate;
- **SSO breaks in-cluster**, because OIDC clients validate the issuer's TLS. This is not hypothetical:
while the wildcard was still LE-**staging**, in-cluster clients rejected it with
`x509: certificate signed by unknown authority`, which is what forced the move to `letsencrypt-prod`;
- and the page you are reading right now is served over that same certificate.
**There is no history of an actual expiry on this cluster.** The recorded history is all issuance
failures. Say so rather than inventing precedent — but note the corollary: the renewal path has never
been exercised under stress, so do not assume it will quietly work.
### Verify the alert can even fire (measured 2026-09-13)
**This alert is currently INERT.** `count(certmanager_clock_time_seconds)` and
`count(certmanager_certificate_expiration_timestamp_seconds)` both returned **0 series** on prod:
there is no cert-manager scrape job among Prometheus's 19 jobs, so all three certificate rules evaluate
against nothing. **Certificate expiry is unmonitored today.** Check expiry by hand with the commands
below, and see `/runbooks/targets-down` for the full list of inert rule groups. Adding cert-manager's
ServiceMonitor is open work.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Every certificate and its real expiry, straight from the CRs (works with no metrics):
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get certificates.cert-manager.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,NOTAFTER:.status.notAfter,RENEWAL:.status.renewalTime,ISSUER:.spec.issuerRef.name'
```
What the Gateway is actually serving (the answer that matters), read from outside the cluster with an
explicit SNI — testing HTTPS from **inside** a pod is misleading here, and an IP+SNI probe without the
right SNI returns curl exit 35:
```
curl -sv --resolve home.keithjackson.dev:443:192.168.2.20 https://home.keithjackson.dev/runbooks/ 2>&1 | grep -E "expire|subject|issuer|SSL certificate"
```
```
echo | openssl s_client -connect 192.168.2.20:443 -servername grafana.home.keithjackson.dev 2>/dev/null | openssl x509 -noout -subject -issuer -dates
```
Is a renewal in flight or stuck?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get certificaterequests.cert-manager.io,orders.acme.cert-manager.io,challenges.acme.cert-manager.io -A; echo; \
kubectl --context k3s-prod -n cert-manager logs deploy/cert-manager --tail=80 | tail -40'
```
## Likely causes, most likely first
### 1. Issuance has been failing since the last successful renewal
The dominant cause, and the one with real history. Go to `/runbooks/cert-issuance` — it covers the
DNS-01 self-check failure (fixed with `dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"` and
`dns01RecursiveNameserversOnly: true`), the Cloudflare token, and the ACME order/challenge chain. Fixing
issuance fixes expiry; nothing else does.
### 2. Two clusters racing the same `_acme-challenge` TXT record
A specific, recorded hazard for any future two-cluster window. `infra/PROD-PROVISIONING.md` §10 records
that after the 2026-09-04 incident, `platform-config` **would have** requested a
`*.home.keithjackson.dev` wildcard from letsencrypt-prod for a certificate dev already held; both
clusters would then have raced each other's `_acme-challenge` TXT records on one Cloudflare zone and
**neither would reliably solve DNS-01**. It never fired only because the tier was suspended. If a second
cluster exists, confirm which one owns the zone before renewing.
### 3. Let's Encrypt rate limits
One wildcard is well inside the limits, so this is unlikely — unless something has been retrying in a
loop. `cert-manager` logs will say `too many certificates already issued`. Remediation: stop the loop,
wait out the window; do **not** switch to letsencrypt-staging as a workaround, because in-cluster OIDC
clients reject the staging chain (that is the bug that forced LE-prod).
### 4. The issuer or its credential changed
`${ACME_CLUSTER_ISSUER}` is a cluster-vars knob; the Cloudflare API token is a SOPS-encrypted Secret.
Note the standing rule: **non-secret, environment-specific values must not live inside an encrypted
Secret** — `${BASE_DOMAIN}` substitution can never reach inside one, which is exactly how prod's OIDC
clients once ended up pointed at dev's Keycloak.
### 5. The Certificate is Ready but the Secret is stale
Check the Secret's actual notAfter (the `openssl` command above) rather than the CR's status. The Gateway
reads the Secret.
## When to escalate or stop
- **`CertificateExpiryImminent` on `wildcard-home` is a cluster-wide outage in under 3 days**, including
SSO and including this runbook server. Get the operator immediately.
- **Do not switch to `letsencrypt-staging`** to make issuance succeed. It will issue, and it will break
every in-cluster OIDC client with `x509: unknown authority`.
- **Do not delete the `wildcard-home-tls` Secret** to force a reissue while issuance is broken — you
would replace a certificate that still has days left with none at all.
- **Do not hand-edit the Secret.** cert-manager owns it; the durable change is in git.
- Verify DNS at the **authority** (`dig @betty.ns.cloudflare.com _acme-challenge.home.keithjackson.dev TXT`),
never at a resolver: 1.1.1.1 has served a deleted TXT record for minutes here and produced a false
failure diagnosis.
---
Source of truth: `runbooks/cert-expiry.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` "Fix 5 — cert-manager DNS-01 self-check via public resolvers" and the
LE-staging → LE-prod switch (Session 7); `infra/PROD-PROVISIONING.md` §10 (the queued ACME race).
runbooks/cert-issuance.md
+165
View File
@@ -0,0 +1,165 @@
# cert-issuance — A Certificate is not Ready (ACME issuance is failing)
**Alert:** `CertificateNotReady` (warning, 1h)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.certificates`
**Expression:** `certmanager_certificate_ready_status{condition="False"} == 1`
## What this means here
cert-manager cannot issue or renew a certificate. The one that matters is `wildcard-home` in the
`gateway` namespace — secret `wildcard-home-tls`, requesting **both** `*.${BASE_DOMAIN}` **and**
`${BASE_DOMAIN}` from `${ACME_CLUSTER_ISSUER}` (`letsencrypt-prod` on prod). Every HTTPS listener on the
shared Gateway terminates with it, so issuance failing is a slow-burning cluster-wide outage: it does
nothing today and takes down every hostname plus in-cluster SSO when the existing certificate expires.
The 1h `for` reflects that — a failing order is worth investigating today, not paging at 3am.
The solver is **DNS-01 against Cloudflare**, which matters because DNS-01 has a failure mode this
cluster has already hit:
**The self-check, not the challenge, is the usual failure.** On 2026-08-16 the wildcard stalled with the
challenge at `Presented=true` — the Cloudflare TXT record *was* created, so the API token worked — while
cert-manager's propagation self-check looped **"DNS record not yet propagated"**. The record was
publicly resolvable at 1.1.1.1 and 8.8.8.8; the NAT'd cluster's own DNS could not see it, because
split-horizon CoreDNS answers authoritatively for the zone and never forwards. The fix is live in
`platform/cert-manager/helmrelease.yaml`:
```yaml
dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"
dns01RecursiveNameserversOnly: true
```
The challenge went pending → valid and the certificate Issued. If you find those two lines missing or
changed, that is your answer.
### Verify the alert can even fire (measured 2026-09-13)
**This alert is currently INERT.** `count(certmanager_clock_time_seconds)` returned **0 series** on
prod — there is no cert-manager scrape job among Prometheus's 19 jobs, so all three certificate rules
evaluate against nothing. Certificate health must be read from the CRs, as below. See
`/runbooks/targets-down` for the full list of inert rule groups.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Walk the chain top-down — Certificate → CertificateRequest → Order → Challenge. The failure is almost
always visible at one specific level:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get certificates.cert-manager.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,NOTAFTER:.status.notAfter'
```
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get certificaterequests.cert-manager.io -A; echo ==; \
kubectl --context k3s-prod get orders.acme.cert-manager.io -A; echo ==; \
kubectl --context k3s-prod get challenges.acme.cert-manager.io -A -o wide'
```
The controller log, which carries the actual ACME error text:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n cert-manager logs deploy/cert-manager --tail=150 2>&1 | grep -iE "error|propagat|challenge|order|rate" | tail -30'
```
Is the TXT record really there? **Ask the authority, never a resolver** — 1.1.1.1 has served a deleted
TXT record here for minutes and produced a false failure diagnosis:
```
dig +short @betty.ns.cloudflare.com _acme-challenge.home.keithjackson.dev TXT
dig +short @1.1.1.1 _acme-challenge.home.keithjackson.dev TXT
```
Is the resolver override still configured?
```
grep -n "dns01Recursive" platform/cert-manager/helmrelease.yaml
```
The issuer and its credential:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get clusterissuers.cert-manager.io \
-o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
```
## Likely causes, most likely first
### 1. The DNS-01 propagation self-check cannot resolve the TXT record
The recorded failure. Signature: Challenge `Presented=true`, status looping on *"DNS record not yet
propagated"*, and `dig @betty.ns.cloudflare.com` **does** return the TXT. Cause: cert-manager is asking
the cluster's own resolver, which is split-horizon CoreDNS answering authoritatively for the zone and
never forwarding. Remediation: the two `dns01Recursive*` values above, in git.
### 2. The Cloudflare API token is wrong, expired, or scoped too narrowly
Signature: Challenge never reaches `Presented=true`, and the log carries a Cloudflare 403/6003. The
token lives in a SOPS-encrypted Secret. Note the standing rule from the OIDC incident: **non-secret
environment-specific values must never live inside an encrypted Secret** — `${BASE_DOMAIN}`
substitution cannot reach inside one, and that is precisely how prod's OIDC clients once ended up
pointed at dev's Keycloak. Keep hostnames out; keep only the token in.
### 3. `EXTERNALDNS_DOMAIN_FILTER` / zone confusion
Recorded as finding F-DNS1: the filter **must be the ZONE** (`keithjackson.dev`), not the subdomain.
A zone mismatch means the TXT record is created in the wrong place or not at all.
### 4. Two clusters racing the same `_acme-challenge` record
`infra/PROD-PROVISIONING.md` §10: after the 2026-09-04 incident `platform-config` would have requested a
`*.home.keithjackson.dev` wildcard from letsencrypt-prod for a certificate dev already held; both
clusters would then have raced each other's `_acme-challenge` TXT records on one Cloudflare zone and
**neither would reliably solve DNS-01**. Only the tier being suspended prevented it. If any second
cluster exists, establish which one owns the zone first.
### 5. Let's Encrypt rate limit
`too many certificates already issued` in the log. One wildcard is well inside the limits, so this means
something has been retrying in a loop. Stop the loop and wait out the window.
### 6. A new Certificate for a hostname that does not resolve
A newly added service whose DNS is not yet published cannot solve DNS-01. Check ExternalDNS first.
## When to escalate or stop
- **Escalate if `wildcard-home` has been not-Ready for more than a day**, and check
`/runbooks/cert-expiry` for how much runway the existing certificate has. Under 3 days of runway is an
operator page.
- **Never switch `${ACME_CLUSTER_ISSUER}` to letsencrypt-staging** to get a green certificate. Staging
issues fine and then breaks every in-cluster OIDC client with `x509: certificate signed by unknown
authority` — that is the bug that forced LE-prod in the first place.
- **Do not delete the existing `wildcard-home-tls` Secret** while issuance is broken. A stale but valid
certificate is far better than none.
- **Do not create certificates by hand.** cert-manager owns them; the durable change is in git.
- Rotating the Cloudflare token is an operator action (SOPS, DR path).
---
Source of truth: `runbooks/cert-issuance.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` "Fix 5 — cert-manager DNS-01 self-check via public resolvers"
(commit ~`59dd3d9`) and the LE-staging → LE-prod switch; `infra/PROD-PROVISIONING.md` §10.
runbooks/cnpg-archiving.md
+198
View File
@@ -0,0 +1,198 @@
# cnpg-archiving — WAL archiving is stuck, or a Postgres cluster has no recoverability point
**Alerts:** `CNPGWALArchiveBacklog` (warning, 1h) and `CNPGNoRecoverabilityPoint` (warning, 6h)
**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
**Expressions:**
```
max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 0
max by (namespace, job) (cnpg_collector_first_recoverability_point) == 0
```
## What this means here — and why it is an AVAILABILITY incident, not just a backup one
`pg_wal/archive_status` holds one marker per WAL segment: `.ready` = written but **not yet archived**,
`.done` = archived and safe to recycle. A healthy archiver drains `.ready` within seconds, so a
persistently non-zero ready count means `barman-cloud-wal-archive` is failing.
**Two consequences, and the second one is the emergency:**
1. There is no continuous archive, so **PITR is broken** (R-IDP-05 / R-STATE-03).
2. **`archive_mode=on` means Postgres will NOT recycle an unarchived WAL segment.** `pg_wal` grows
without bound until the data PVC fills and the postmaster **PANICs**. `pg_wal` shares the **data**
PVC on every Cluster here — no `walStorage:` is configured on any of them — so there is no separate
volume to absorb it.
This is measured, not theoretical. Prod ran **from bootstrap to 2026-09-07** with
`endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`: all three Clusters at `ContinuousArchiving=False` with
`barman-cloud-wal-archive: exit status 4` and `firstRecoverabilityPoint: null`, **for 38+ hours, with no
alert of any kind** — found by hand. On `keycloak-db` that was 17 unarchived segments / ~268 MiB on a
5 GiB PVC, growing **~6.8 MiB/h ≈ 29 days to disk-full** (about 579 MiB already burned by 2026-09-07).
The rule recorded verbatim: *"Never leave a placeholder here, and treat `ContinuousArchiving=False` on
this Cluster as an availability incident, not a backup one."*
`CNPGWALArchiveBacklog` is *"the one that would have caught the 2026-09-06 incident within the hour
instead of 38 hours later."* `> 0` with `for: 1h` — not a count threshold — because `for` requires the
condition at **every** evaluation for the full hour, so a write burst resets the timer while a genuinely
stuck archiver never returns to zero. A `> 5` threshold would have delayed `semaphore-db` indefinitely:
its backlog sits at 3 because it writes almost nothing.
## The two alerts are NOT the same statement
`CNPGWALArchiveBacklog` is about the **mechanism** and is per-**pod** (each instance has its own `pg_wal`
on its own PVC; a `> 0` shape is safe per-pod, and `pod` names the instance to go look at).
`CNPGNoRecoverabilityPoint` is about **DR coverage** and is per-**CLUSTER**, grouped
`by (namespace, job)`. That grouping is a fix, dated 2026-09-12: only the **primary** reads the backup
catalogue, so a replica reports a hard `0` forever. Grouped by `pod`, this `== 0` fired **permanently on
all six replicas** while all three Clusters were healthy. Verified live:
old `max by (namespace, pod) (...) == 0` → 6 series (`keycloak-db-1/-3`, `nextcloud-db-1/-3`,
`semaphore-db-1/-3`); new `max by (namespace, job) (...) == 0` → **empty** — and not vacuously so:
forcing the metric to 0 for one primary yields exactly one series. `job` is the grouping label because
the metric carries **no `cnpg.io/cluster` label**; the operator creates one PodMonitor per Cluster, so
prometheus-operator derives `job="<namespace>/<cluster>"`: `keycloak/keycloak-db`,
`nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`. `namespace` alone happens to work today but
would let two Clusters in one namespace mask each other.
**Practical note:** if you see this alert firing on *pods* rather than per cluster, the per-cluster
aggregation is not deployed on this cluster yet — check the deployed PrometheusRule before treating it
as a real loss of DR.
**Severity is `warning` on purpose.** A database heading for disk-full deserves weeks of lead time, and
this repo has been burned by a permanently-firing `critical` (the `VPNTunnelDown` post-mortem). The
`critical` tier for this failure mode is `PersistentVolumeAlmostFull`, which fires only when the outage
is genuinely imminent.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
The three Clusters' archiving and recoverability status — the single most informative command:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
```
The archiver's actual error text — `exit status 4` and its cause live here:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -i archive | tail -30'
```
The backlog on disk, per instance:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c \
"ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | grep -c ready; du -sh /var/lib/postgresql/data/pgdata/pg_wal; df -h /var/lib/postgresql/data"'
```
The backups that exist — **fully-qualified, or you will be told there are none**:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A; echo; \
kubectl --context k3s-prod get scheduledbackups.postgresql.cnpg.io -A'
```
`kubectl get backup -A` prints "No resources found" on this cluster even with 27 CNPG backups present:
three CRDs register the kind `Backup` (`longhorn.io`, `postgresql.cnpg.io`, `velero.io`) and the short
name resolves to Longhorn's, which holds zero. Verified on prod 2026-09-13. **Never decide a recovery
action from the short name.**
The object-store configuration, which is where the fault usually is:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak keycloak-db -o yaml | sed -n "/backup:/,/^ [a-z]/p"'
```
```
grep -nE "CNPG_BACKUP_PATH|S3_BACKUP_ENDPOINT|S3_BACKUP_REGION|BACKUP_CREDENTIALS_SUFFIX" clusters/base/cluster-vars.yaml clusters/prod/kustomization.yaml
```
## Likely causes, most likely first
### 1. A wrong or placeholder object-store endpoint / path / credential
The recorded cause of the 38-hour incident: prod ran with `endpointURL:
TBD-PROD-S3-BACKUP-ENDPOINT`. The placeholder was **deliberately loud** because *"base defaults are the
DEV values, so an OMITTED op means prod silently inherits a dev address — far worse than a failed
apply"*. Live prod values to expect: `CNPG_BACKUP_PATH = s3://k3s-prod-backup/barman`,
`S3_BACKUP_ENDPOINT = https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`,
`S3_BACKUP_REGION = auto`, `BACKUP_CREDENTIALS_SUFFIX = -prod`, retention `30d`, WAL compression gzip,
`maxParallel: 2`. Remediation: fix it in git and push; Flux applies and the archiver drains on its own.
### 2. R2-specific S3 incompatibilities
Cloudflare R2 rejects things MinIO tolerates. The two that are mandatory elsewhere in this platform and
worth checking here: `region` must be exactly `auto`, and AWS-SDK checksum headers are rejected with
**HTTP 400 InvalidArgument**. See `/runbooks/backup-failure` for the full R2 story, including the Velero
plugin's `x-amz-tagging` `NotImplemented` failure.
### 3. Credentials expired or mis-scoped
A SOPS-encrypted Secret selected by `BACKUP_CREDENTIALS_SUFFIX`. Standing rule from the OIDC incident:
**non-secret, environment-specific values must NOT live inside an encrypted Secret** — `${BASE_DOMAIN}`
substitution can never reach inside one. Keep endpoints and paths in cluster-vars, tokens in the Secret.
### 4. Egress is blocked
Default-deny is the baseline. The database namespaces need outbound HTTPS to R2. Cross-read
`/runbooks/denied-flows`.
### 5. A fresh Cluster that has not made its first base backup
`firstRecoverabilityPoint` reads null until the first base backup lands; the `for: 6h` exists exactly to
keep a legitimate cold start quiet. Base backups are staggered: keycloak 01:00, nextcloud 02:00,
semaphore 04:00 (Velero at 03:00). If a Cluster is hours old, wait.
### 6. Scaling out made it worse
Noted honestly: scaling a Cluster while `archive_mode=on` with a failing `archive_command`
**multiplies the disk-full clock**. Do not add instances as a remedy here.
## When to escalate or stop
- **Treat `ContinuousArchiving=False` as an availability incident with a countdown.** Compute the runway:
free bytes on the data PVC ÷ WAL growth rate. At ~6.8 MiB/h on a 5 GiB PVC that is weeks; on a busier
database it is days.
- **NEVER delete WAL segments by hand.** It destroys the database. This is the one irreversible mistake
available in this runbook.
- **Do not delete the `.ready` marker files.** Same outcome.
- **Do not expand the PVC and stop there.** It buys time and hides the fault.
- **Do not scale the Cluster** while archiving is failing.
- **Verify before any restore** that `firstRecoverabilityPoint` is non-null and
`backups.postgresql.cnpg.io` shows completed backups. Prod's healthy state on 2026-09-13 was:
all three Clusters `ContinuousArchiving=True`, `LastBackupSucceeded=True`, `firstRecoverabilityPoint`
2026-09-07, 27 backups (9 per cluster) completing daily, with one historical failure
(`semaphore-db-daily-20260905130918`). That is what "good" looks like.
- Escalate to the operator for credential or endpoint changes, and for any restore.
---
Source of truth: `runbooks/cnpg-archiving.md`. Index: `/runbooks/`
Records: `platform/cnpg/alerts.yaml` header (the 2026-09-06 incident, the measured fill rate, and the
2026-09-12 aggregation-scope fix with its live promtool proof);
`platform/keycloak/postgres-cluster.yaml`; commit `32d6766` (the R2 decision).
runbooks/cnpg-backup-missed.md
+163
View File
@@ -0,0 +1,163 @@
# cnpg-backup-missed — No recent CNPG base backup for a Postgres cluster
**Alert:** `CNPGNoRecentBaseBackup` (warning, 2h)
**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
**Expression:**
```
(time() - max by (namespace, job) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600)
and max by (namespace, job) (cnpg_collector_last_available_backup_timestamp) > 0
```
## What this means here
The newest completed **base backup** for one Postgres cluster is over 36 hours old — one missed daily run
plus slack. The daily `ScheduledBackup` has silently stopped producing backups, so the PITR window is
ageing out against the `30d` retention policy.
**Nothing else covers this.** The Velero rules in `platform/monitoring/alerts.yaml` watch Velero's own
schedules, and **Velero does not back up these databases** — CNPG owns their DR path, because a Velero
PVC snapshot of a live Postgres is not a backup. So `/runbooks/backup-missed` being clear says nothing
about the databases, and vice versa.
The three Clusters and their staggered base-backup times: **keycloak 01:00**, **nextcloud 02:00**,
**semaphore 04:00** (Velero's own daily runs at 03:00, deliberately between them).
## Two expression details that matter when you read the alert
**The `> 0` guard is load-bearing.** A cluster that has *never* had a backup reports `0`, and
`time() - 0` is about **57 years** — always true. Without the guard this rule would double-notify with
`CNPGNoRecoverabilityPoint`, which is the correct alert for "never had a backup". So this rule means
specifically: *this cluster has had at least one backup, and the newest one is stale.*
**It is aggregated `by (namespace, job)`, not `by (namespace, pod)`** — fixed 2026-09-12 alongside
`CNPGNoRecoverabilityPoint`. `cnpg_collector_last_available_backup_timestamp` has the same primary-only
defect: measured on prod the same day, `keycloak-db-2` = 3.1h old, `nextcloud-db-2` = 2.1h,
`semaphore-db-2` = 0.1h, and **all six replicas = 0**, i.e. `time() - 0` ≈ **56.7 years**. Unlike its
sibling this rule was *not* firing — because the `> 0` guard was the single thing standing between it and
six more permanent false positives, which is far too thin a margin for a DR alarm. Aggregating `pod` away
removes the dependence: `max` picks the primary's timestamp as the cluster's answer (a demoted primary
drops to 0 and can never win the `max`; a fresher backup always beats a staler one).
`job` is the grouping label because the metric carries **no `cnpg.io/cluster` label** — the operator
creates one PodMonitor per Cluster, so prometheus-operator derives `job="<namespace>/<cluster>"`:
`keycloak/keycloak-db`, `nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`.
**Severity is `warning` on purpose** — this repo has been burned by a permanently-firing `critical`
(the `VPNTunnelDown` post-mortem), and *"noise on a DR alarm is worse than no alarm at all"*.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
**Fully-qualified names, always.** `kubectl get backup -A` prints **"No resources found"** on this
cluster even with **27 CNPG backups** present: three CRDs register the kind `Backup`
(`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1`, `velero.io/v1`) and the bare short name resolves to
**Longhorn's**, which holds zero objects. Verified on prod 2026-09-13. Concluding "the backups are gone"
from that and acting on it is how a CLI quirk becomes data loss.
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CLUSTER:.spec.cluster.name,PHASE:.status.phase,START:.status.startedAt,STOP:.status.stoppedAt --sort-by=.metadata.creationTimestamp | tail -20'
```
Are the schedules there and enabled?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get scheduledbackups.postgresql.cnpg.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SCHEDULE:.spec.schedule,SUSPEND:.spec.suspend,CLUSTER:.spec.cluster.name,LAST:.status.lastScheduleTime'
```
The Cluster's own verdict (and the condition that tells you whether *archiving* is also broken):
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIMARY:.status.currentPrimary,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
```
Why the last attempt failed — the barman error is in the primary's logs:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -iE "barman|backup|archive" | tail -30; echo ====; \
kubectl --context k3s-prod -n cnpg-system logs deploy/cnpg-controller-manager --tail=120 2>&1 | grep -iE "backup|error" | tail -20'
```
## Likely causes, most likely first
### 1. The ScheduledBackup is suspended, missing, or unapplied
Check `spec.suspend`, and whether the object exists at all for that Cluster. A `ScheduledBackup` that was
never applied is the "looks done, is absent" failure class this repo keeps finding; verify the Flux tier
that owns it is Ready (`/runbooks/flux-reconcile`) and not suspended (`/runbooks/flux-suspended`).
### 2. The object store is rejecting the write
Same root cause family as `/runbooks/cnpg-archiving`: endpoint, path, region or credentials. Live prod
values to expect are `CNPG_BACKUP_PATH = s3://k3s-prod-backup/barman`,
`S3_BACKUP_ENDPOINT = https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`,
`S3_BACKUP_REGION = auto`, `BACKUP_CREDENTIALS_SUFFIX = -prod`. **If `ContinuousArchiving` is also
False, fix that first** — a cluster whose archiver is broken has a disk-full countdown running, which is
more urgent than a stale base backup.
### 3. A failed run that nobody noticed
There is precedent: one historical failure, `semaphore-db-daily-20260905130918`. A single failed run does
not clear itself; the next night's run should. If two consecutive nights fail, the cause is
configuration, not luck.
### 4. The primary changed and the new primary cannot write
The backup catalogue is the **primary's** job. After a switchover, a credential or path problem scoped to
one instance shows up here. Check which pod is `currentPrimary` and read that pod's logs, not another's.
### 5. Node or storage trouble during the backup window
Base backups are I/O-heavy and run at 01:00-04:00 alongside Velero. Cross-read `/runbooks/node-io`.
### 6. The metric is missing, not the backup
If `cnpg_collector_last_available_backup_timestamp` has no series, the rule cannot fire. Note that a
**related** blind spot is real and separate: the CNPG SQL exporter can fail while `up` stays 1 — see
`/runbooks/cnpg-exporter`.
## When to escalate or stop
- **A stale base backup narrows, but does not eliminate, recoverability** — provided WAL archiving is
healthy, PITR still works from the older base. That is why this is a warning. If **both** this and
`CNPGNoRecoverabilityPoint` / `CNPGWALArchiveBacklog` are lit, DR for that database is effectively
gone: escalate.
- **Never delete WAL, `.ready` markers, or anything under a Postgres data directory.**
- **Do not delete old CNPG Backup objects to tidy up.** Retention is `30d` and the operator manages it;
deleting a Backup object can orphan objects in R2.
- **Do not trigger a base backup as a reflex** on a cluster whose archiver is failing — it adds write
load to a volume that may already be filling.
- **Be honest about restore:** the CNPG restore path from R2 has not been rehearsed on this cluster
(`dr-restore` is authored and never executed). Free R2 egress makes a rehearsal cheap; it is
outstanding. A restore is an operator operation.
---
Source of truth: `runbooks/cnpg-backup-missed.md`. Index: `/runbooks/`
Records: `platform/cnpg/alerts.yaml` (the 2026-09-12 aggregation-scope fix with its live measurements,
the `> 0` guard rationale, and the staggered schedule).
runbooks/cnpg-exporter.md
+168
View File
@@ -0,0 +1,168 @@
# cnpg-exporter — The CNPG metrics collector is erroring (a database is invisible)
**Alert:** `CNPGMetricsCollectorErroring` (warning, 6h)
**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
**Expression:** `max by (namespace, pod) (cnpg_last_error) == 1`
## What this means here
**This is the monitor's own blind spot, for databases — a failure that looks exactly like health.**
CloudNativePG publishes **two independent metric families** and only one of them depends on SQL:
- `cnpg_collector_*` comes from the instance manager and is **always** present.
- `cnpg_pg_stat_*` (archiver, database, replication) comes from the exporter logging in to the
application database as the `cnpg_metrics_exporter` role.
When that login fails, `cnpg_last_error` goes to 1 and **the entire `cnpg_pg_stat_*` family blanks for
that instance — while `up` stays 1 and `cnpg_collector_up` stays 1.** Nothing looks broken. The scrape
target is green. The dashboards have gaps that read as "no data" rather than "broken".
**This is not hypothetical, and it shaped the whole alert file.** Measured live on 2026-09-06 on both
clusters, `cnpg_errors_total` carried:
```
pg_stat_archiver on db nextcloud: failed to connect to `user=cnpg_metrics_exporter database=nextcloud`:
FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
```
`nextcloud-db` had been publishing **no** `pg_stat_archiver` / `pg_stat_database` metrics at all. The
consequence recorded verbatim: *an archiving alert built on `pg_stat_archiver` — the obvious choice —
would have been silently blind to one of three prod databases.* That is why every DR rule in
`platform/cnpg/alerts.yaml` keys on `cnpg_collector_*` instead, and why this rule exists to alert on the
blind spot itself.
**Scope: `pod` is kept here, deliberately and unchanged.** `cnpg_last_error` is genuinely per-instance —
each instance's own exporter reports its own SQL-collection health — and prod proved the metric
discriminates rather than tracking role: all **three** `nextcloud-db` pods (including primary `-2`)
reported 1 while all six `keycloak`/`semaphore` pods reported 0. There is no primary-only defect to fix,
and per-pod is the right granularity because the remedy is a **GRANT applied per database** and you want
to see it clear on every instance.
`for: 6h` keeps a pod restart or a brief connection blip quiet. Severity stays `warning` so it never
becomes the permanently-firing `critical` this repo has been burned by.
**Live state, measured 2026-09-13:** `max by (namespace,pod) (cnpg_last_error) == 1` returned **empty**
across the 9 instances that publish `cnpg_last_error` — so the `nextcloud` grant appears to have been
addressed. Re-measure rather than assuming either way.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Which instances are erroring:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
curl -s -G --data-urlencode "query=max by (namespace,pod) (cnpg_last_error)" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "pod|\"1\""'
```
**The message itself** — `cnpg_errors_total` carries it in an `errorUserQueries` label. This is the one
command that tells you the cause outright:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
curl -s -G --data-urlencode "query=cnpg_errors_total" localhost:19090/api/v1/query | python3 -m json.tool | head -60'
```
Or straight from the instance's metrics endpoint:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n nextcloud-db exec nextcloud-db-2 -c postgres -- \
curl -s localhost:9187/metrics | grep -E "^cnpg_last_error|^cnpg_errors_total|^cnpg_collector_up" | head -20'
```
Confirm which family is actually missing — this is what makes the blind spot visible:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
for q in "count(cnpg_collector_up)" "count(cnpg_last_error)" "count(cnpg_pg_stat_archiver_archived_count)" "count(cnpg_pg_stat_database_xact_commit)"; do \
printf "%-50s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \
| python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done'
```
The exporter role's actual grants:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n nextcloud-db exec nextcloud-db-2 -c postgres -- \
psql -tAc "select rolname, rolcanlogin from pg_roles where rolname = \"cnpg_metrics_exporter\";" \
-c "select datname, has_database_privilege(\"cnpg_metrics_exporter\", datname, \"CONNECT\") from pg_database where datistemplate = false;"'
```
## Likely causes, most likely first
### 1. The exporter role lacks CONNECT on the application database
The recorded cause. Signature: `SQLSTATE 42501`, `permission denied for database "<name>"` in
`cnpg_errors_total`. Remedy: **grant the `cnpg_metrics_exporter` role CONNECT on the application
database.** Do it declaratively — CNPG can manage roles and the `managed.roles` / bootstrap surface in the
Cluster spec is where this belongs, so the grant survives a re-seed. A `psql` GRANT by hand fixes the
symptom on one instance and vanishes on the next bootstrap.
### 2. Custom monitoring queries reference something that does not exist
CNPG lets a Cluster attach extra collector queries via a ConfigMap. A query against a missing table,
extension or column fails the whole collection pass for that instance. `cnpg_errors_total`'s label names
the failing query.
### 3. The database is not accepting connections
A read-only/recovery state, `max_connections` exhausted, or the instance mid-restart. Cross-read
`/runbooks/keycloak-db` if the instance is `keycloak-db-*`. Note the deliberate setting
`dataDurability: preferred` (never `required`) exists so a degraded standby cannot block writes here.
### 4. An upgrade changed the exporter's expectations
CNPG **1.30.0** is the verified operator version on both clusters. A major operator or Postgres bump can
change the collector's queries; if this started at an upgrade, read `/runbooks/update-rollback`.
### 5. The instance is fine and the whole scrape is gone
Then `up` for that `job` would be 0 and this is the wrong runbook — `/runbooks/targets-down`. The precise
value of *this* alert is the case where `up` is **1** and the data is still missing.
## When to escalate or stop
- **Understand what you are currently blind to.** While this fires, `cnpg_pg_stat_archiver`,
`cnpg_pg_stat_database` and `cnpg_pg_stat_replication` are missing for that instance. The DR alerts in
`platform/cnpg/alerts.yaml` are deliberately built on `cnpg_collector_*` so they still work — **but do
not build any new alert or dashboard on `pg_stat_*` without checking this first.** That is the trap
this alert exists to mark.
- **Do not silence it.** It is a warning with a 6h `for` precisely so it can be left visible without
becoming noise, and it was a **true positive** for months.
- **Do not grant the exporter role broad privileges** to make it go away. CONNECT on the application
database is the whole requirement; anything more widens the blast radius of a metrics credential.
- **Do not restart the database** to clear it — the error is a grant, not a state.
- Escalate to the operator if the fix requires changing the Cluster's managed roles or bootstrap, since
that touches the database's declarative definition and its DR path.
---
Source of truth: `runbooks/cnpg-exporter.md`. Index: `/runbooks/`
Records: `platform/cnpg/alerts.yaml` (the 2026-09-06 `SQLSTATE 42501` measurement on `nextcloud-db`, the
two-metric-families finding, and the 2026-09-12 scope audit that deliberately kept `pod`).
runbooks/cnpg-primary-spread.md
+187
View File
@@ -0,0 +1,187 @@
# cnpg-primary-spread — Two or more CNPG primaries share one node
**Alert:** `CNPGPrimariesCoLocated` (warning, 30m)
**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
**Expression:** `count by (node) (<ready *-rw endpoints joined to cnpg_collector_up>) > 1`
**and** gated on every node being Ready:
`count(kube_node_status_condition{condition="Ready",status="true"} == 1) == count(kube_node_info)`
## What this means here
More than one CloudNativePG **primary** has been running on the same node for 30 minutes, while all
three nodes are Ready. There are exactly three CNPG Clusters on this platform — `keycloak/keycloak-db`,
`nextcloud-db/nextcloud-db` and `semaphore-db/semaphore-db` — so on a 3-node cluster the healthy shape is
one primary per node, and this alert says that shape has drifted.
**This is BLAST RADIUS, not an outage.** Each Cluster has `instances: 3` with one pod per node
(`podAntiAffinityType: required`) and one synchronous standby (`method: any`, `number: 1`,
`dataDurability: preferred`), so every one of them survives losing the shared node — you just get two or
three simultaneous PostgreSQL failovers instead of one. Nothing is broken right now, which is why it is a
`warning` with a long `for`.
**Why it matters anyway: this exact drift has already caused an outage.** All three Clusters used to run
`instances: 1` and all three primaries had drifted onto one node, and **on 2026-09-07 that node took SSO,
Nextcloud and Semaphore down together.** `instances: 3` means the same drift no longer causes an outage —
but if `keycloak-db`'s primary is co-located and that node dies, the SSO failover happens at the same
moment as two others, on nodes that are also absorbing Longhorn replica rebuilds.
**Why the drift happens and cannot be declared away: CNPG has no declarative primary-placement field.**
The primary lands wherever the last promotion put it — a node reboot, a switchover, an operator upgrade.
So this is avoidable drift that must be *corrected*, not prevented, and the alert re-arms itself so it
will be reported again rather than silently persisting.
**Why the node-readiness gate is in the expression.** During a real node outage, primaries legitimately
pile onto the survivors and there is nothing to do about it; alerting then would be noise at the worst
moment. The `and on ()` clause suppresses the rule unless every node is Ready, and re-arms when the node
returns.
This rule was proven in both directions on live prod data (2026-09-13): the expression returned
`{node="k3s-prod-1"} => 3` before the rebalance and a **throwaway twin with `for: 1m`** was observed going
`pending` at +70s and `firing` at +130s; after two switchovers the same expression returned an **empty
vector while the inner join still resolved all three primaries** — quiet because the condition is false,
not because the query went blind. It is also unit-tested in `alerts.promtool-test.yaml`
(run `./run-alert-tests.sh`) with four cases, mutation-checked so that deleting the `cnpg_collector_up`
join, the readiness gate, or shortening `for:` makes a test FAIL.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Where the primaries are — the one command this alert is about:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get pods -A -l cnpg.io/instanceRole=primary \
-o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName'
```
Are all three Clusters actually healthy enough to move one? (Fully-qualified:
`clusters.postgresql.cnpg.io` — `cluster` is an ambiguous short name.)
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase'
```
Is every node Ready (the gate the rule itself applies)?
```
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes -o wide'
```
Is there storage or I/O trouble that makes a switchover a bad idea right now?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
-o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy'
```
```
ssh k3s-prod-1 'cat /proc/pressure/io'
```
And confirm DR is intact before you deliberately move a primary — the primary owns the backup
catalogue, and **fully-qualified names matter**: `kubectl get backup -A` prints "No resources found" on
this cluster even with 27 CNPG backups present, because three CRDs register the kind `Backup` and the
short name resolves to Longhorn's (zero objects):
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A | tail -10; echo; \
kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
```
## Likely causes, most likely first
### 1. A node rebooted and the primaries never moved back
The dominant cause. Promotions during the reboot put two primaries on a survivor, and CNPG has no reason
to move them back afterwards. Remediation: a deliberate switchover, below.
### 2. A rolling operator or Postgres upgrade
`primaryUpdateMethod: switchover` is set deliberately (the CRD default `restart` is a
delete-and-recreate of the *primary*, i.e. tens of seconds with no Postgres — an SSO outage on
`keycloak-db`). Switchovers during an upgrade are correct behaviour and leave the primaries wherever
they landed.
### 3. One node was cordoned or unschedulable for a while
Same mechanism. Note that `podAntiAffinityType: required` means an instance stays **Pending** until a
node returns rather than co-locating — that is the accepted cost of the 2026-09-07 fix, and it is why
the *instances* stay spread even when the *primaries* do not.
### 4. A Longhorn or I/O event caused a health-check failover
A primary whose storage stalled gets demoted. Cross-read `/runbooks/node-io` and
`/runbooks/longhorn-degraded` — if that is why, fix the node before rebalancing.
### 5. The alert is firing during a genuine node outage
It should not be able to: the readiness gate suppresses it. If it fires anyway with a node NotReady, the
gate is broken — that is a monitoring bug worth fixing, since a rule that cannot stop is fatigue.
## Remediation — move one primary, deliberately
The cnpg plugin is **NOT in the devcontainer image**; fetch the `kubectl-cnpg` release matching the
operator version (CNPG **1.30.0** here) before you start.
```
kubectl cnpg promote -n <namespace> <cluster> <instance-on-an-idle-node>
```
**Do ONE CLUSTER AT A TIME and wait for `readyInstances: 3` before the next.** A switchover interrupts
writes to that database for about **9 seconds** — measured on both clusters moved on 2026-09-13. Nine
seconds of `keycloak-db` unavailability is a brief SSO interruption; doing two at once is an avoidable
compound one.
Pick the target by reading the primaries table above and choosing an instance that already sits on a node
holding no other primary. Then re-run the table to confirm the new distribution, and let the alert clear
on its own.
## When to escalate or stop
- **Do not rebalance during any other incident.** A switchover is a small, deliberate write interruption,
and it is the wrong thing to add to a node that is already under I/O pressure or to a cluster with
degraded Longhorn volumes. This alert has a 30-minute `for` precisely because it is never urgent.
- **Never switch over a Cluster whose `readyInstances` is below 3.** You would be promoting into a
cluster that has no standby left to fall back on.
- **Never change `dataDurability` to `required`** while thinking about durability here: it is a strict
quorum, and one degraded standby would then **block writes** entirely on the SSO database.
- **Do not `kubectl delete pod` the primary to force a failover.** That is `restart` semantics — the
thing `primaryUpdateMethod: switchover` exists to avoid.
- **Check `ContinuousArchiving` and `firstRecoverabilityPoint` first** (`/runbooks/cnpg-archiving`). If
archiving is broken, that is a real countdown to a disk-full PANIC and it outranks blast-radius
tidying completely.
- Escalate to the operator for the switchover itself if the affected Cluster is `keycloak-db`: the brief
write interruption is user-visible across SSO, kubectl OIDC, Grafana, Headlamp and Gitea.
---
Source of truth: `runbooks/cnpg-primary-spread.md`. Index: `/runbooks/`
Records: `platform/cnpg/alerts.yaml` (the rule's own both-directions proof on live prod data and its
promtool tests); commits `f338e91` (instances 3), `e8d8601` (required anti-affinity + switchover),
`cac413e` (synchronous replication, `dataDurability: preferred`); the 2026-09-07 single-node SSO outage.
runbooks/cnpg-wal-growth.md
+161
View File
@@ -0,0 +1,161 @@
# cnpg-wal-growth — pg_wal is not being recycled
**Alert:** `CNPGWALVolumeNotRecycling` (warning, 2h)
**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
**Expression:**
```
max by (namespace, pod) (cnpg_collector_pg_wal{value="count"})
> max by (namespace, pod) (cnpg_collector_pg_wal{value="max"})
```
## What this means here
`pg_wal` is holding more WAL segments than `max_wal_size` allows for, which means **checkpoints are no
longer reclaiming space**. On CNPG the units are **segments of 16 MiB**: `max` is `max_wal_size`
(**64 = 1 GiB** on this platform) and `count` is what is currently sitting in `pg_wal`.
This is the **storage escalation** of `/runbooks/cnpg-archiving`, not an independent fault.
`CNPGWALArchiveBacklog` is the day-one signal — it fires within an hour of the archiver getting stuck.
This one fires when the resulting backlog has grown large enough to be a *disk* problem. With
`archive_mode=on` a failing `archive_command` is exactly what this looks like after a while, and because
**`pg_wal` shares the data PVC on every Cluster here** (no `walStorage:` is configured on any of them)
it ends in the volume filling and the postmaster **PANICking**.
It does **not** fire today and is not meant to: measured, prod `keycloak-db` holds 17 segments against
max 64 (dev held 36 against 64), and across all nine prod instances counts ran 16-37 against 64. At
prod's observed ~6.8 MiB/h, `keycloak-db` would reach 64 segments roughly **40 days before** its 5 GiB
PVC actually ran out — so this alert leads the PVC-fill alerts by weeks.
**Why `pod` is kept here, and is required.** Every instance reports its own `count` and `max`, `pg_wal`
is per-instance on a per-instance PVC, and the comparison only means anything pod-against-itself —
aggregating `pod` away would compare one instance's `count` with another's `max`. A `>` shape is also
safe per-pod: no instance can sit permanently true. (Contrast the `== 0` and staleness rules in the same
file, which **must** aggregate `pod` away — that is the 2026-09-12 fix.)
**One metric that looks right and is not:** the CNPG-native
`cnpg_collector_pg_wal{value="volume_size"}` is **NaN on both clusters**, precisely because `pg_wal`
shares the data PVC rather than having its own. Do not build anything on it.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Which instance, and how far past `max_wal_size`:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
curl -s -G --data-urlencode "query=max by (namespace,pod) (cnpg_collector_pg_wal{value=\"count\"})" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "pod|\"1\""'
```
The same thing from inside the instance, which also shows you the archive backlog and the free space in
one view:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c \
"du -sh /var/lib/postgresql/data/pgdata/pg_wal; \
ls /var/lib/postgresql/data/pgdata/pg_wal | grep -c \"^0\"; \
ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | grep -c ready; \
df -h /var/lib/postgresql/data; \
psql -tAc \"show max_wal_size; show archive_mode; select pg_walfile_name(pg_current_wal_lsn());\""'
```
Is archiving the cause? (It almost always is.)
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint; echo; \
kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -i archive | tail -20'
```
Is a replication slot or a stale standby holding WAL instead?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- \
psql -x -c "select slot_name, active, wal_status, safe_wal_size, restart_lsn from pg_replication_slots;" \
-c "select application_name, state, sync_state, replay_lag from pg_stat_replication;"'
```
Compute the runway — free bytes on the PVC divided by the observed growth rate:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
curl -s -G --data-urlencode "query=predict_linear(kubelet_volume_stats_available_bytes{job=\"kubelet\",namespace=\"keycloak\"}[24h], 7*24*3600)" localhost:19090/api/v1/query'
```
## Likely causes, most likely first
### 1. WAL archiving is failing (dominant cause)
Go to `/runbooks/cnpg-archiving` — that is the actual fix. Fix the `archive_command` and Postgres recycles
the backlog on its own, without any intervention in `pg_wal`. Historical precedent: prod ran 38+ hours
from bootstrap with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, all three Clusters at
`ContinuousArchiving=False` with `barman-cloud-wal-archive: exit status 4`.
### 2. An inactive replication slot is pinning WAL
A standby that has been gone long enough leaves a slot holding `restart_lsn`, and Postgres keeps every
segment after it. `pg_replication_slots.wal_status` will read `extended` or `lost`. The healthy fix is to
bring the standby back (CNPG manages the slots), not to drop the slot by hand. Note
`podAntiAffinityType: required` means a lost instance stays **Pending** until a node returns — expected,
and accepted.
### 3. Genuinely heavy write load
Then `count` exceeding `max_wal_size` is checkpointing lagging, not a leak, and it self-corrects. Confirm
that `archive_status/*.ready` is near zero: a large `pg_wal` with an empty ready queue is a busy
database, not a stuck archiver.
### 4. `max_wal_size` was lowered
64 segments = 1 GiB is the configured value on both clusters. A reduction makes this rule fire without
anything actually being wrong. Check the Cluster's postgresql parameters — and note that under
`primaryUpdateMethod: switchover` the operator **rejects an update that changes `imageName` and a
postgresql parameter in the same reconcile**.
### 5. The instance is a replica that fell behind
Replay lag holds WAL on the primary. `pg_stat_replication.replay_lag` shows it.
## When to escalate or stop
- **NEVER delete WAL segments or `archive_status` marker files by hand.** It destroys the database. This
is the single irreversible mistake available here, and the alert's own description says so: *"Fix
archiving — do NOT delete WAL by hand."*
- **Do not drop a replication slot** to free space without the operator: you may make a lagging standby
unrecoverable and force a full re-seed.
- **Do not expand the PVC and stop there.** It buys days and hides the fault.
- **Do not scale the Cluster out** while archiving is failing — that **multiplies** the disk-full clock.
- Escalate when the computed runway is under a week, or when `PersistentVolumeAlmostFull`
(`/runbooks/pvc-capacity`) is also firing for the same PVC — at that point an outage is imminent and a
Postgres data volume under 5% free will PANIC the postmaster.
---
Source of truth: `runbooks/cnpg-wal-growth.md`. Index: `/runbooks/`
Records: `platform/cnpg/alerts.yaml` (the measured segment counts, the NaN `volume_size` finding, and the
2026-09-12 scope audit that deliberately KEPT `pod` here).
runbooks/denied-flows.md
+154
View File
@@ -0,0 +1,154 @@
# denied-flows — A spike in policy-denied network flows
**Alert:** `HubbleDeniedFlowSpike` (warning, 10m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.networking`
**Expression:** `sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5`
## What this means here
Cilium is dropping more than 5 flows/second on policy. East-west traffic is **default-deny** everywhere
(R-NET-01): a bare workload reaches DNS and nothing else, and every allow is declared per
source → destination → port (R-NET-02). So a denied-flow spike is almost always **a NetworkPolicy
regression** — something legitimate that is no longer allowed — and only rarely a probe.
### Verify the alert can even fire (measured 2026-09-13)
**This alert is currently INERT.** `count(hubble_drop_total)` returned **0 series** on prod: there is no
Hubble/Cilium scrape job among Prometheus's 19 jobs, so Hubble metrics are not being collected and this
rule evaluates against nothing. **Policy drops are unmonitored today** — you will hear about them as
"service X cannot reach service Y", and you diagnose them with `hubble observe`, below. See
`/runbooks/targets-down` for the full list of inert rule groups.
### The regression class that has actually bitten this cluster
**Cilium's CVE-2025-30162 hairpin change (the expensive one).** After the 1.16 → 1.20 upgrade, the
shared Gateway's L7 proxy re-checks a pod's egress policy against the destination's **real backend
identity**. So `toEntities: world` **no longer covers an in-cluster call to a `*.home.keithjackson.dev`
hostname** — the destination resolves to an in-cluster backend, not to `world`. Every app's SSO call
started returning **403 with `server: envoy`**. The fix is an explicit `toEndpoints` allow to the
Keycloak backend, which `catalog/render.py` now emits for every service with
`auth.mode ∈ {oidc, forward-auth}`.
**The durable rule that follows: any new app → app-by-hostname dependency needs its own explicit allow.**
If a newly added or newly SSO-enabled service is 403ing or timing out, this is the first thing to check.
A second recorded instance of the same class: Flux's `source-controller` was denied by
`gitea-allow-gateway-ingress`, whose ingress rule was `fromEntities: [ingress]` only — so an in-cluster
`wget` to Gitea simply timed out. And a matching diagnostic trap: **testing HTTPS from inside a pod is
misleading here**; test from outside with `curl --resolve`.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
See the actual drops — this is the tool, since the metric is not collected:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
hubble observe --verdict DROPPED --last 200 -o compact'
```
Narrow to a suspected pair:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
hubble observe --verdict DROPPED --from-namespace nextcloud --last 100 -o compact'
```
What policies apply to the source workload, and is the policy actually enforced?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get ciliumnetworkpolicies.cilium.io,networkpolicies.networking.k8s.io -A; echo; \
kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg endpoint list | head -30'
```
Cilium's own drop counters, as a cross-check:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg metrics list | grep -i drop'
```
Is the 403 an Envoy/L7 verdict rather than an L3/L4 drop? (The hairpin class above shows up here, not in
`hubble observe`.)
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n <app-ns> logs <pod> --tail=50 | grep -iE "403|envoy|oidc|well-known"'
```
## Likely causes, most likely first
### 1. A newly added or newly SSO-enabled service has no explicit allow to Keycloak
The CVE-2025-30162 hairpin class described above. Signature: HTTP **403 with `server: envoy`** on a call
to `keycloak.${BASE_DOMAIN}` or another `*.home` hostname from inside the cluster, while the same URL
works from outside. Remediation: the allow is **rendered**, not hand-written — set `auth.mode` in
`catalog/<service>.yaml` and re-render so `render.py` emits the `toEndpoints` allow to the keycloak
backend. Do not patch a CiliumNetworkPolicy by hand; it will be overwritten.
### 2. A platform component was moved or a namespace label changed
`fromEntities: [ingress]`-only ingress rules (the Gitea/source-controller case) break any in-cluster
client. Remediation: add the specific allow in the component's `networkpolicy.yaml`, in git.
### 3. A missing dependency declaration in the catalog
`dependencies:` in a ServiceDefinition is what generates app→app allows (R-NET-02). A service talking to
something it never declared is denied by design. The declaration is the fix.
### 4. A scrape target that nothing allows
Prometheus selects **every** ServiceMonitor in the cluster
(`serviceMonitorSelectorNilUsesHelmValues: false`), so a new workload's monitor is scraped immediately
and will be denied until the component allows ingress from `monitoring`. Cross-read
`/runbooks/targets-down`.
### 5. An actual probe or scan
Possible, and it is why the alert is worded as a security signal (docs/07). Distinguish it by the
**source**: a spike from outside the cluster or from an unexpected identity, rather than one named
workload repeatedly failing one destination. Escalate rather than tuning policy.
## When to escalate or stop
- **Do not widen a policy to make drops stop.** Default-deny is a safety-critical requirement here
(R-NET-01 is flagged `safety_critical: true` in the requirement registry). The correct fix is the
narrowest explicit allow, authored in git, usually via the catalog so it is rendered consistently.
- **Do not hand-edit a rendered CiliumNetworkPolicy** — it comes from `catalog/render.py` and will be
reverted on the next reconcile.
- **Escalate if the source is unexpected.** An unexplained spike from outside the cluster is a security
event, not a configuration bug.
- If drops coincide with a Cilium upgrade, assume the hairpin/identity class above before assuming a
policy authoring mistake — the upgrade changed the *evaluation*, not the policy.
- Note the diagnostic trap when verifying a fix: probe from **outside** with `curl --resolve`, because
in-pod HTTPS tests to the platform's own hostnames are misleading.
---
Source of truth: `runbooks/denied-flows.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` (the Cilium CVE-2025-30162 world-hairpin / envoy 403 finding and its
`render.py` fix; source-controller denied by `gitea-allow-gateway-ingress`).
runbooks/flux-reconcile.md
+164
View File
@@ -0,0 +1,164 @@
# flux-reconcile — A Flux resource has failed to reconcile for 15 minutes
**Alert:** `FluxReconciliationFailure` (warning, 15m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux`
**Expression:** `max by (namespace, name, kind) (gotk_reconcile_condition{type="Ready", status="False"}) == 1`
## What this means here
Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than
it sounds, because **`git push origin main` IS the deploy** — there is no other apply step. A stuck
Kustomization means git and the cluster have diverged and nothing will close the gap.
The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` →
`platform-config-cdi` → `apps-wave-0/1/2`, plus the non-gating leaves `apps-vms`, `oauth2-proxy`,
`headlamp`, `homepage`, `crafty`, `runbooks`, `backup`. Which tier is failing decides the blast radius:
- **`flux-system`, `cilium`, `platform`, `platform-config` are gating (`wait: true`).** One Failed
HelmRelease in the `platform` tier stalls **everything downstream** on "dependency not ready". That
is not theoretical: a crashlooping Headlamp HelmRelease once stalled `platform-config`,
`oauth2-proxy` and every apps wave (the "Headlamp gating stall"), and oauth2-proxy twice deadlocked
by **blocking its own fix** — it sat in the gating tier while needing Keycloak + trusted TLS +
split-horizon DNS, so `platform-config`, which applies the fix, could never reconcile. Both were
resolved by moving the leaf into its own `wait: false` tier.
- **The leaf tiers are `wait: false`** precisely so their unhealth can never gate anything.
Prod's source is `http://gitea.gitea.svc.cluster.local:3000` — the **in-cluster Service**, on purpose.
It used to be a public hostname, and the apex flip taught the expensive version of that lesson: the
HTTPRoute serving the source's own hostname lives in a downstream tier that cannot reconcile while the
source is down, so `srcReady=False` / *"unable to list remote"* became a **deadlock**. If you are
tempted to point the GitRepository at a public name, don't.
**This alert is currently INERT.** `count(gotk_reconcile_condition)` and `count(gotk_suspend_status)`
both returned **0 series** on prod on 2026-09-13: there is no scrape job for the Flux controllers in
Prometheus's 19 jobs, so every `gotk_*`-based rule (`FluxReconciliationFailure`,
`FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`,
`UpdateHealthGateFailed`) evaluates against nothing. **Flux health is therefore unmonitored today** and
must be read from the CRs directly, with the commands below. See `/runbooks/targets-down` for the full
list of inert rule groups. Adding the Flux controllers' ServiceMonitor is open work.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
The whole chain at a glance — this is the command to run first, every time:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
-o custom-columns=NAME:.metadata.name,PATH:.spec.path,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,REV:.status.lastAppliedRevision'
```
Is the SOURCE healthy? If not, nothing downstream can be:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system get gitrepositories.source.toolkit.fluxcd.io \
-o custom-columns=NAME:.metadata.name,URL:.spec.url,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
```
HelmReleases (the usual culprit inside `platform`):
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get helmreleases.helm.toolkit.fluxcd.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
```
The controller's own view, with the real error text:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system logs deploy/kustomize-controller --tail=80; echo ====; \
kubectl --context k3s-prod -n flux-system logs deploy/helm-controller --tail=80'
```
## Likely causes, most likely first
### 1. "no matches for kind X" — a CR applied before its CRD
The recurring structural failure here. Flux server-side-apply fails the **whole** apply, so a tier that
ships both a CRD-installing HelmRelease and CRs of that CRD can never converge — the CRD never gets
created because the apply that would create it also contains the CR. Symptoms:
`no matches for kind "ClusterIssuer"` / `"PrometheusRule"`. The standing rule: **any component shipping
CRs of a CRD it installs must split the CRs into `platform-config`** (`dependsOn: platform`), and
second-order CRs whose CRD only appears after a controller reconciles go in `platform-config-cdi`.
Remediation is a git change moving the resource; `bootstrap/standup.sh crds` is the belt-and-braces
repair for a cluster already wedged.
### 2. A HelmRelease is genuinely failing its install/upgrade
Read the message. Known instances: the `headlamp-0.40.1` `-session-ttl` chart/image desync; Grafana 13
OOM-looping after kps 65→88. Note `platform`'s `timeout: 5m` versus Longhorn's HelmRelease
`timeout: 15m` — **the tier can time out while a legitimate Longhorn upgrade is still running**. Give
it another interval before concluding it is broken.
### 3. A `${TOKEN}` problem in post-build substitution
Substitution **fails open**: an undefined variable becomes the empty string and the Kustomization still
reports Ready — which is how `k8sServiceHost: ""` broke Cilium on 2 of 3 dev nodes on 2026-09-01. A
*parse* error is the loud version and does fail the whole Kustomization
(`unable to parse variable name`). If the message mentions envsubst or a variable, run the L0 check:
`conformance/tests/l0-static/flux-postbuild-substitution/check.sh`. Note **strict substitution became
default-on in kustomize-controller v1.9.0**, so a Flux upgrade flips this behaviour whether or not you
opt in.
### 4. A remote kustomize base timed out
The kustomize-controller's git fetch has a 27s timeout and remote GitHub bases flake against it.
`external-snapshotter` was vendored into the repo for this reason; **`platform/kubevirt` still uses
remote bases**, so a `BuildFail` there is probably this. Remediation: vendor it the same way.
### 5. SOPS decryption
A tier that carries an encrypted Secret needs its own `decryption` stanza patched in by the overlay, by
name. A newly split-out tier that inherited a secret but not the patch fails here.
## Remediation notes
- Everything durable is a **git change**. `flux reconcile` only re-triggers; it does not fix a bad
manifest, and it is blocked for agents here.
- Flux is **level-triggered**: an interrupted reconcile is retried, so a restart of a controller is
safe and often the fastest way past a wedged in-flight apply (operator action).
- `dependsOn` is **revision-gated**, not merely Ready-gated: a dependent cannot reconcile revision N
until its dependency has applied N. That is what makes the chain safe and also what makes one stuck
tier so expensive.
## When to escalate or stop
- **If `flux-system`, `cilium`, `platform` or `platform-config` is the failing tier, treat it as a
platform outage in progress** — everything downstream is frozen, including the fix. Get the operator.
- **Do not `flux suspend` a tier to make the alert stop.** That trades a visible failure for
`/runbooks/flux-suspended` and silent drift, which is how prod's whole `platform` tier ended up
suspended and left that way after the 2026-09-04 external-dns incident.
- **Do not enable `driftDetection.mode: enabled` on a CNI.** During the 2026-09-01 Cilium incident the
Helm storage manifest *was* the bad render, so drift correction would have actively re-asserted the
broken value.
- If the GitRepository itself is not Ready, check Gitea before Flux — and remember prod deliberately
reads from the in-cluster Service so that public DNS can never be in this path.
---
Source of truth: `runbooks/flux-reconcile.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` "2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium" (substitution
fails open), the Headlamp gating stall, the oauth2-proxy cold-start deadlock, and the 2026-09-13
GitOps source deadlock.
runbooks/flux-suspended.md
+134
View File
@@ -0,0 +1,134 @@
# flux-suspended — A Flux resource has been suspended for an hour
**Alert:** `FluxResourceSuspended` (warning, 1h)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux`
**Expression:** `max by (namespace, name, kind) (gotk_suspend_status) == 1`
## What this means here
Someone turned reconciliation off for a resource and it is still off an hour later. While suspended,
Flux neither applies new commits nor corrects drift — **git stops being the truth** for that subtree,
silently, and everything downstream of a suspended gating tier is frozen with it.
Suspension is a legitimate emergency valve here and is used deliberately: `cilium` was split into its
own Kustomization partly so that `flux suspend kustomization cilium` becomes an option that does not
freeze the other eleven platform components. The problem is never the suspend; it is the **forgotten**
suspend. Two real cases:
- After the 2026-09-04 external-dns incident, prod's whole `platform` tier and everything downstream
were suspended **and left suspended** (`infra/PROD-PROVISIONING.md` §10). That suspension is also
the only reason a second bug never fired: `platform-config` would otherwise have requested a
`*.home.keithjackson.dev` wildcard from letsencrypt-prod for a certificate dev already held, and the
two clusters would have raced each other's `_acme-challenge` TXT records on one zone — neither would
reliably solve DNS-01.
- **Velero was found suspended live, with the suspension never recorded in git**, and resumed with
`flux resume`. A backup controller that is not reconciling is the definition of a silent DR gap.
The 1h `for` is the point: short suspensions during maintenance are expected and should not notify.
**This alert is currently INERT.** `count(gotk_reconcile_condition)` and `count(gotk_suspend_status)`
both returned **0 series** on prod on 2026-09-13: there is no scrape job for the Flux controllers in
Prometheus's 19 jobs, so every `gotk_*`-based rule (`FluxReconciliationFailure`,
`FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`,
`UpdateHealthGateFailed`) evaluates against nothing. **Flux health is therefore unmonitored today** and
must be read from the CRs directly, with the commands below. See `/runbooks/targets-down` for the full
list of inert rule groups. Adding the Flux controllers' ServiceMonitor is open work.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
What is suspended, across every Flux kind:
```
infra/devcontainer/dev.sh run bash -c \
'for k in kustomizations.kustomize.toolkit.fluxcd.io helmreleases.helm.toolkit.fluxcd.io \
gitrepositories.source.toolkit.fluxcd.io helmrepositories.source.toolkit.fluxcd.io \
imagerepositories.image.toolkit.fluxcd.io imagepolicies.image.toolkit.fluxcd.io \
imageupdateautomations.image.toolkit.fluxcd.io; do \
echo "== $k"; kubectl --context k3s-prod get "$k" -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SUSPENDED:.spec.suspend 2>/dev/null | grep -v "<none>"; done'
```
Is the suspension declared in git, or is it live-only drift? (This is the important question.)
```
grep -rn "suspend" clusters/ platform/ | grep -v "^Binary"
```
If it is not in git, it was applied by hand and nothing will restore it after a controller restart —
which means the resource may un-suspend itself at an unpredictable moment, or stay off forever.
How far behind is it?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
-o custom-columns=NAME:.metadata.name,SUSPENDED:.spec.suspend,APPLIED:.status.lastAppliedRevision; echo; \
kubectl --context k3s-prod -n flux-system get gitrepositories.source.toolkit.fluxcd.io \
-o jsonpath="{.items[0].status.artifact.revision}{\"\n\"}"'
```
## Likely causes, most likely first
### 1. A maintenance suspension nobody resumed
By far the most common, and the Velero case above is the archetype. Remediation (operator):
`flux resume kustomization <name>` — but **only after** reading the next paragraph.
### 2. It is suspended for a reason that still applies
Before resuming, find out what would be applied. A tier suspended during an incident may be holding
back a change that caused the incident. The 2026-09-04 case is the cautionary one: resuming
`platform-config` while a second cluster still held the same wildcard certificate would have started an
ACME DNS-01 race on a shared Cloudflare zone. **Diff first**: compare
`status.lastAppliedRevision` against the GitRepository's current revision and read the commits in
between.
### 3. A crashlooping leaf was suspended instead of being moved
The correct fix for "a leaf UI keeps failing and stalls the tier" is a `wait: false` tier of its own —
that is what happened to `oauth2-proxy`, `headlamp`, `homepage`, `crafty`, `velero` and `runbooks`.
Suspension as a permanent substitute leaves drift correction off for that component forever.
### 4. Image automation
`ImageUpdateAutomation`/`ImagePolicy`/`ImageRepository` are **authored and deliberately disabled** here
(pending a Gitea write credential and `flux install --components-extra`), so if those show up, this is
expected state, not an incident. See `/runbooks/image-automation`.
## When to escalate or stop
- **Do not resume a gating tier (`cilium`, `platform`, `platform-config`) without the operator.**
Resuming applies every commit since the suspension in one pass, to the tier everything depends on.
- **Do resume `velero` promptly** if it is the suspended resource — a suspended backup controller is a
DR gap, and it has happened before.
- If the suspension is not in git, record it in git (or resume it). A live-only suspension is
undocumented state, which is the failure class this repo keeps finding.
- If you cannot determine *why* something was suspended, stop and ask. That is a safer outcome than
either resuming blind or leaving it off.
---
Source of truth: `runbooks/flux-suspended.md`. Index: `/runbooks/`
Records: `infra/PROD-PROVISIONING.md` §10 (the 2026-09-04 suspension and the queued ACME race);
`DECISIONS-LOG.md` (Velero found suspended with the suspension never in git).
runbooks/image-automation.md
+129
View File
@@ -0,0 +1,129 @@
# image-automation — Flux's image-update-automation cannot write back to git
**Alert:** `ImageUpdateAutomationFailing` (warning, 30m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.automated-updates`
**Expression:** `max by (namespace, name) (gotk_reconcile_condition{kind="ImageUpdateAutomation", type="Ready", status="False"}) == 1`
## What this means here
`ImageUpdateAutomation` is the write-back half of hands-off updates (ADR-0009): it commits new image
tags into git so Flux then deploys them. When it is Ready=False, **stable image advances silently
stop** — nothing breaks, nothing degrades, the platform just quietly stops receiving updates. That is
the R-UPD-02 failure mode this alert exists for.
**Read this first: on this cluster, image automation is authored and DELIBERATELY DISABLED.** It has
been that way since 2026-08-24, pending (a) a Gitea write credential for the automation to push with
and (b) `flux install --components-extra=image-reflector-controller,image-automation-controller`. The
line is still commented out in `platform/kustomization.yaml`. So:
- If this alert fires, the most likely explanation is that **the feature was partially enabled** —
someone uncommented the component or a rendered app started emitting `ImageRepository`/`ImagePolicy`
CRs whose controllers are not installed.
- Rendered apps emit those CRs by default; `catalog/render.py --no-image-automation` is the documented
workaround for exactly this.
- There is consequently **no real incident history** for this alert. Everything below is mechanism, not
post-mortem, and it says so honestly rather than inventing precedent.
**This alert is currently INERT.** `count(gotk_reconcile_condition)` returned **0 series** on prod on
2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every `gotk_*` rule
evaluates against nothing. Read the CRs directly with the commands below, and see
`/runbooks/targets-down` for the full list of inert rule groups.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Are the controllers even installed?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system get deploy | sort; echo; \
kubectl --context k3s-prod api-resources --api-group=image.toolkit.fluxcd.io'
```
The automation objects and their messages:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get imageupdateautomations.image.toolkit.fluxcd.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,LASTRUN:.status.lastAutomationRunTime'
```
The controller log — the git error is always in here:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system logs deploy/image-automation-controller --tail=100 2>&1 | tail -40'
```
Is the repo configured with a write path at all?
```
grep -rn "image-automation\|ImageUpdateAutomation\|components-extra" platform/kustomization.yaml bootstrap/ clusters/ 2>/dev/null | head -20
```
## Likely causes, most likely first
### 1. The controllers are not installed (expected state here)
`api-resources --api-group=image.toolkit.fluxcd.io` returns nothing, or the deployments are absent.
Then the CRs are inert objects and the alert reflects a half-enabled feature. Remediation: either
finish enabling it (an operator runs `flux install --components-extra=...` — note that Flux's own
controllers were installed **imperatively** and carry no Flux ownership labels, so **nothing under
`clusters/**` can install a controller or change a controller flag**), or stop rendering the CRs with
`catalog/render.py --no-image-automation`.
### 2. No git write credential
The automation must push. Prod's source is `http://gitea.gitea.svc.cluster.local:3000`, and the
read-only source credential is not a write credential. The controller log will say authentication
failed or permission denied. Remediation: a Gitea token in a SOPS-encrypted Secret referenced by the
`GitRepository`/automation, added in git — and note the standing rule from the OIDC incident:
**non-secret environment-specific values must not live inside an encrypted Secret**, because
`${BASE_DOMAIN}` substitution can never reach inside one.
### 3. It cannot push because the branch moved
Automation commits to a branch; a force-push or a protected branch rejects it. The log names the ref.
### 4. The commit is being made but the Kustomization is not picking it up
Then this is not the right runbook — go to `/runbooks/flux-reconcile`.
## When to escalate or stop
- **This is a warning with a 30m `for` for a reason: nothing is broken.** Do not treat it as an
outage. The consequence is "no new image versions", which is measured in days.
- **Do not create a Gitea write credential ad hoc.** It is a new secret in the DR path and belongs to
the operator, SOPS-encrypted, in git.
- **Do not enable the extra Flux controllers to make the alert stop.** Enabling automation on a
cluster that has never run it will start proposing image bumps across every workload at once; that is
a planned change, with `/runbooks/update-rollback` as its companion.
- If the alert is firing because the feature is half-enabled, the honest fix is to finish or to revert
— both are git changes and both want the operator's decision.
---
Source of truth: `runbooks/image-automation.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` (image automation authored-and-disabled since 2026-08-24; the Flux
controllers are outside Flux's own inventory).
runbooks/image-policy-stalled.md
+137
View File
@@ -0,0 +1,137 @@
# image-policy-stalled — An ImagePolicy or ImageRepository cannot resolve its tag
**Alert:** `ImagePolicyStalled` (warning, 1h)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.automated-updates`
**Expression:** `max by (namespace, name, kind) (gotk_reconcile_condition{kind=~"ImagePolicy|ImageRepository", type="Ready", status="False"}) == 1`
## What this means here
An `ImageRepository` cannot scan a registry, or an `ImagePolicy` cannot pick a tag out of what was
scanned. The effect is narrower than `/runbooks/image-automation`: automation itself is healthy, but
**this one service stops receiving updates** (R-UPD-01), while everything else continues. It is
therefore easy to miss, which is what the 1h `for` and the per-object labels are for.
The same framing applies as for image-automation: **this feature is authored and deliberately disabled
on this cluster** and has been since 2026-08-24 (no Gitea write credential, and the
`image-reflector-controller` / `image-automation-controller` components are not installed — the line is
still commented out in `platform/kustomization.yaml`). Rendered apps emit `ImageRepository` and
`ImagePolicy` CRs anyway, so the objects can exist with no controller to reconcile them;
`catalog/render.py --no-image-automation` is the documented workaround. **There is no incident history
for this alert** — the section below is mechanism, honestly labelled.
One live constraint worth knowing while reading registry errors: the images in this platform come from
`docker.io`, `ghcr.io`, `lscr.io`, `quay.io`, `registry.k8s.io` and `registry.gitlab.com`. Docker
Hub's anonymous rate limit is the most likely real-world cause of a scan failure once automation is
switched on.
**This alert is currently INERT.** `count(gotk_reconcile_condition)` returned **0 series** on prod on
2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every `gotk_*` rule
evaluates against nothing. Read the CRs directly with the commands below, and see
`/runbooks/targets-down` for the full list of inert rule groups.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Which object, and the message (the message is nearly always the whole answer):
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get imagerepositories.image.toolkit.fluxcd.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,IMAGE:.spec.image,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,TAGS:.status.lastScanResult.tagCount; echo; \
kubectl --context k3s-prod get imagepolicies.image.toolkit.fluxcd.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,LATEST:.status.latestImage,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
```
Are the controllers installed at all?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system get deploy | grep -i image || echo "image controllers NOT installed (expected on this cluster)"'
```
The reflector's log:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system logs deploy/image-reflector-controller --tail=80 2>&1 | tail -30'
```
Can the cluster reach the registry at all? (Default-deny is the baseline; a new namespace may simply
not be allowed out.)
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get pods -A --field-selector status.phase!=Running | head; echo; \
kubectl --context k3s-prod get events -A --field-selector reason=Failed --sort-by=.lastTimestamp | grep -i "pull\|registry" | tail -10'
```
## Likely causes, most likely first
### 1. The image controllers are not installed (expected state here)
The CRs exist because `catalog/render.py` emits them; nothing reconciles them. Remediation: render with
`--no-image-automation`, or have the operator finish enabling automation. Note that the Flux
controllers were installed **imperatively** and are outside Flux's own inventory, so no git change can
install them.
### 2. Registry rate limit
Docker Hub anonymous pulls/scans are limited per IP, and all three nodes share one WAN address. The
message says `toomanyrequests`. Remediation: a registry credential (SOPS-encrypted, in git), or scan
less often via `spec.interval`.
### 3. Registry auth
A private repository with no `secretRef`, or an expired token. The message names it.
### 4. The policy's range matches nothing
`ImagePolicy` semver ranges come from each ServiceDefinition's `update.range` (for example
`">=2024.0.0"` for pihole, `">=24.0.0"` for keycloak). A range that no published tag satisfies — or a
repository that publishes only `latest` (this platform has `lscr.io/linuxserver/jackett:latest` and
`plexinc/pms-docker:latest`) — leaves the policy with nothing to select. Remediation: fix the range in
`catalog/<service>.yaml` and re-render; a `latest`-only image cannot participate in automation at all
and should not carry a policy.
### 5. Egress is blocked for that namespace
Default-deny everywhere (R-NET-01) means outbound registry access is an explicit allow. Cross-read
`/runbooks/denied-flows`, and in particular the post-Cilium-upgrade finding that `toEntities: world` no
longer covers a call whose destination resolves to an in-cluster backend.
## When to escalate or stop
- **Warning, 1h, single service: this does not page.** The consequence is one workload pinned at its
current version.
- **Do not widen an `update.range` to make the alert clear.** The range is the service's update policy
(R-UPD-01); loosening it to satisfy a scanner is how an unintended major version arrives.
- **Do not add registry credentials ad hoc** — new secret, DR path, operator's call.
- If the same object has been stalled for days, the honest outcome is to remove its automation CRs
(`--no-image-automation`) rather than leave a permanently-failing object training everyone to ignore
the alert. Alert fatigue is a recorded, expensive failure mode on this cluster.
---
Source of truth: `runbooks/image-policy-stalled.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` (image automation authored-and-disabled; `--no-image-automation`);
`catalog/*.yaml` (`update.channel` / `update.range` per service).
runbooks/keycloak-client-secrets.md
+201
View File
@@ -0,0 +1,201 @@
# keycloak-client-secrets — The OIDC client reconciler is failing or has gone silent
**Alerts:** `KeycloakClientSecretReconcilerFailing` (warning, 15m) and
`KeycloakClientSecretReconcilerStale` (warning, 2h)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
**Expressions:**
```
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"}) > 0
absent(kube_job_status_succeeded{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"} == 1)
```
## What this means here
These two rules watch the CronJob (every 15 minutes) that repairs OIDC client **secrets** and
environment-derived client **URLs** in the Keycloak realm. They exist because of the single worst
green-looking outage this cluster has had, and they are best understood through it.
### The 2026-09-06 lockout — why "everything is green" means nothing here
The Keycloak Operator resolves each OIDC client's secret from an `(env:NAME)` placeholder in the realm
import. **When it cannot resolve one, it stores the LITERAL placeholder text as the client secret and the
import Job still reports SUCCESS.** On prod, **all 12 clients** ended up holding the literal placeholder
and **every OIDC login on the cluster failed** with *"Invalid client or Invalid client credentials"*.
Meanwhile: the import Job succeeded, the CR rested at `Done=True / Started=False`,
`KeycloakDown` and `KeycloakDatabaseDown` were quiet, and **the conformance test that asserts
`Done=True` passed throughout the outage.** The only observable was a human failing to log in. Dev had
exactly one broken client (oauth2-proxy — its key was added *after* dev's import completed, and
**a completed import is never re-run**).
The realm import is a **ONE-SHOT that fails open**: the operator logs *"Job already executed - not
recreating"*, so a git change to `keycloak-realm-import.yaml` does **not** re-import.
### The second half: redirect URIs, same one-shot, one field over
The realm import carries **26 `${BASE_DOMAIN}` references** (39 files under `platform/` and `apps/`
reference the knob). Substitution happened **once**, at the old hostname, and then froze. After the apex
flip, *"every OIDC login fails with a `redirect_uri` mismatch while the CR still reads `Done=True` and
nothing alerts."* The reconciler was extended with a second pass over `redirectUris`, `webOrigins`,
`rootUrl`, `baseUrl` and `adminUrl`, with desired state **generated** by
`ci/render-keycloak-client-urls.py` (gated by an L0 `--check`).
Semantics worth knowing before you read a Job log:
- URLs are copied **verbatim, including `${BASE_DOMAIN}`**, and rendered by the *same* Flux
Kustomization as the import.
- **List fields compare order-insensitively (`sorted`, not `set`)** — measured, not defensive: Keycloak
genuinely reorders lists, and an order-sensitive compare would have rewritten `kubernetes` (8 URIs) and
`oauth2-proxy` (2) **every 15 minutes forever while reporting success**.
- A field the desired state omits is **UNMANAGED, never forced empty**.
- **Public clients' URLs are managed even though their secrets are not** (`kubernetes` is public, with
PKCE, so no kubeconfig holds a secret).
- The two passes **fail closed independently** — demonstrated live: pass 1 failed on an empty secrets
directory while pass 2 completed.
### Why there are exactly two rules, and what each catches
`...Failing` sees a run that *ran and failed* — unrepairable drift, or the admin API unreachable. It
fires alongside `KeycloakDown` during an IdP outage, deliberately: the reconciler genuinely is not
running, so treat `KeycloakDown` as the cause when both are lit. Self-clears when failed Jobs age out
(TTL 1h).
`...Stale` sees the **healer being quietly absent** — the CronJob suspended, never applied by Flux, or
unable to schedule a pod at all. That is the "looks applied and isn't" failure this whole change exists
to end. It is also the rule that would catch the healer missing during a domain migration.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Does the CronJob exist, is it suspended, and when did it last succeed?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n keycloak get cronjobs.batch \
-o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,SUSPEND:.spec.suspend,LAST:.status.lastScheduleTime,LASTSUCCESS:.status.lastSuccessfulTime'
```
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n keycloak get jobs.batch --sort-by=.metadata.creationTimestamp \
-o custom-columns=NAME:.metadata.name,SUCCEEDED:.status.succeeded,FAILED:.status.failed,START:.status.startTime | tail -15'
```
**Read the Job log — it names the pass, the client, and for a URL the exact before/after, and it never
prints a secret** (that is a design property, asserted by the L0 check):
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n keycloak logs job/$(kubectl --context k3s-prod -n keycloak get jobs.batch \
--sort-by=.metadata.creationTimestamp -o jsonpath="{.items[-1:].metadata.name}") --tail=200'
```
Is the IdP itself up? (If not, that is the cause; come back here after.)
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n keycloak get pods; echo; \
kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak'
```
Confirm the declaration is self-consistent **offline**, before touching the cluster — this is the same
check the merge gate runs unconditionally (Stage 3d):
```
bash conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh
python3 ci/render-keycloak-client-urls.py --check
```
Confirm what a user actually experiences:
```
curl -s -o /dev/null -w '%{http_code}\n' --resolve keycloak.home.keithjackson.dev:443:192.168.2.20 \
https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
```
```
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod auth whoami'
```
(Expected on a working OIDC path: `oidc:<user>` with `[oidc:platform-admins]`. From this workstation the
OIDC context hangs on a browser prompt — that is not evidence of breakage.)
## Likely causes, most likely first
### 1. Keycloak or its database is down, so the reconciler cannot reach the admin API
`...Failing` fires alongside `KeycloakDown` by design. Fix the IdP first: `/runbooks/keycloak-down`,
`/runbooks/keycloak-db`. Nothing here is actionable until the admin API answers.
### 2. Unresolvable drift on one client
The log names the client and the pass. Most often a secret key missing from the SOPS Secret, or a client
in the realm that git does not describe. The contract: keys are **`<clientId>-client-secret`**, and the
public `kubernetes` client declares **no** secret. Remediation is a git change to the declaration, then
let the CronJob heal it — the reconciler is compare-then-write and idempotent.
### 3. A hostname migration left the realm behind
The `redirect_uri` mismatch case. Regenerate desired state with `ci/render-keycloak-client-urls.py` and
push; the reconciler's second pass applies it. Historical scale: re-rendering at the post-flip apex
produced **24 drifted fields across 12 clients**, with `kubernetes` correctly `ok` because its localhost
callbacks are not domain-derived.
### 4. The CronJob is suspended or was never applied (`...Stale`)
Check `spec.suspend` and the Flux tier. The ordering requirement is a **real Flux `dependsOn` edge with
`wait: true`**, not a comment — the L0 check asserts that, because a comment is not an ordering guarantee.
### 5. The pod cannot read its own secrets
A specific, recorded trap: the reconciler needs **`fsGroup: 65532`** with the client-secrets volume at
**`defaultMode: 0440`**. With `0400` and no fsGroup the container dies with `PermissionError` on **every**
run. If the log shows that, this is it.
### 6. Everything is green and logins still fail
Then trust the human, not the dashboard — that is the entire lesson of 2026-09-06. Audit the live realm
directly with `kcadm.sh` inside `keycloak-0`.
## When to escalate or stop
- **Do not "fix" the realm by re-applying the import.** It is a one-shot: the operator logs *"Job already
executed - not recreating"* and nothing happens. The documented path is `kcadm.sh` inside `keycloak-0`
(admin credentials in Secret `keycloak-admin-credentials`; `kcadm` is present, `curl` and `python3` are
**not**) **and** fixing git so the change is reproducible. Doing only the live half creates drift; doing
only the git half changes nothing.
- **Never print or paste a client secret.** The reconciler deliberately logs URL values in full (they are
public hostnames) and never logs secrets. Keep that property.
- **Do not widen the reconciler's endpoint allow-list.** It is hard-scoped so that `/users`, `/groups`,
`/roles` and `/authentication` are unreachable — a blast-radius bound, not an oversight.
- **Do not disable or suspend the CronJob to stop the alert.** That is the healer, and `...Stale` exists
precisely because a suspended healer is invisible.
- Escalate for: rotating client secrets, editing the live realm, or anything touching MFA (a
conditional-OTP misconfiguration once 403'd every platform-admin login with
`"Invalid role name: null"`, fixed to `condUserRole: platform-admin`).
---
Source of truth: `runbooks/keycloak-client-secrets.md`. Index: `/runbooks/`
Records: `ci/l0.sh` Stage 3d and
`conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh` (the 2026-09-06 incident and the
reconciler's contract — 212 L0 assertions, an 84-assertion selftest, 12/12 injected regressions killed);
`DECISIONS-LOG.md` "Keycloak client REDIRECT URIs reconcile too — the same one-shot, one field over".
runbooks/keycloak-db.md
+169
View File
@@ -0,0 +1,169 @@
# keycloak-db — Keycloak's PostgreSQL is unavailable
**Alert:** `KeycloakDatabaseDown` (critical, 5m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
**Expression:**
```
max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*", condition="true"}) == 0
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*"})
```
## What this means here
No ready `keycloak-db-*` pod for five minutes. `keycloak-db` is a **CloudNativePG Cluster** with
**3 instances** on Longhorn (3 replicas each), and it is the database the IdP cannot live without. The
`pod=~"keycloak-db-.*"` scoping matters for the same reason as in `/runbooks/keycloak-down`: the
namespace also holds the Keycloak server and operator pods, and an unscoped `max()` would be masked by
those.
**Blast radius is the whole platform's identity:** Keycloak fails → kubectl OIDC, Grafana, Headlamp,
Gitea and forward-auth all fail. The certificate-based kubeconfig is an independent break-glass path and
still works. This page is deliberately served **without** forward-auth so it stays readable during
exactly this outage.
## The HA properties you are relying on, and why they are set the way they are
- **`instances: 3`.** All three CNPG Clusters used to run `instances: 1`, and all three primaries had
drifted onto one node — **on 2026-09-07 that node took SSO, Nextcloud and Semaphore down together.**
Scale-out to 3 is what gives you a standby to fail over to.
- **`podAntiAffinityType: required`**, not CNPG's default `preferred`. `preferred` is a scheduler
*hint* and on the path that matters — recreating an instance after node loss — it co-locates.
Recorded conclusion: *"`preferred` is how a 3-instance cluster decays back into the single-node
failure domain."* The accepted cost is that an instance stays **Pending** until a node returns.
- **`primaryUpdateMethod: switchover`.** The CRD default is `restart`, which is a delete-and-recreate of
the **primary** — tens of seconds with no Postgres, i.e. an SSO outage. Constraint to remember: under
`switchover` the operator **rejects an update that changes `imageName` and a postgresql parameter in
the same reconcile**.
- **Synchronous replication `method: any`, `number: 1`, `dataDurability: preferred` — never `required`.**
`required` is a strict quorum: with fewer than `number` healthy standbys, **writes BLOCK**. On this
database one degraded standby (a node reboot, a Longhorn rebuild, a Cilium restart) would become a
total write outage. `dataDurability` is only honoured while `standbyNamesPre/Post` are unset.
**And the thing that is easy to get wrong:** *"Standbys replicate corruption and deletions faithfully;
only the Barman archive is a backup."* Three ready replicas are not a recovery plan.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
The Cluster's own verdict first — fully-qualified, because `cluster` is an ambiguous short name:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase'
```
Where the instances are, and whether they are all on one node (the 2026-09-07 defect):
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n keycloak get pods -l cnpg.io/cluster=keycloak-db -o wide'
```
Why a pod is not ready:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n keycloak describe pod keycloak-db-2 | sed -n "/Events:/,\$p"; echo ====; \
kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=80 2>&1 | tail -40'
```
The conditions that matter for recoverability — check these **before** any recovery action:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak keycloak-db \
-o custom-columns=ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
```
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get backups.postgresql.cnpg.io -n keycloak'
```
`kubectl get backup -A` prints "No resources found" here even with 27 CNPG backups present — three CRDs
register the kind `Backup` and the short name resolves to Longhorn's, which has none. Never decide
anything from the short name.
Storage underneath:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n keycloak get pvc; \
kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy'
```
## Likely causes, most likely first
### 1. A node is down or was rebooted
The instances live on Longhorn with hard anti-affinity, so a node loss takes one instance. With
`instances: 3` the cluster should survive it and promote; with `podAntiAffinityType: required` the lost
instance stays **Pending** until a node returns, which is expected and accepted. Go to
`/runbooks/node-down`.
### 2. The data PVC filled up
The most dangerous cause, because the mechanism is a *backup* fault presenting as a *storage* failure.
With `archive_mode=on` and a failing `archive_command`, Postgres **refuses to recycle** unarchived WAL;
`pg_wal` shares the data PVC on every Cluster here (no `walStorage:` configured) and grows until the
volume is full and the postmaster **PANICs**. Measured on this very database: 17 unarchived segments /
~268 MiB on a 5 GiB PVC, growing ~6.8 MiB/h — about 29 days from 93% free to dead. Go to
`/runbooks/cnpg-archiving` and `/runbooks/pvc-capacity`. **Never delete WAL by hand.**
### 3. A Longhorn volume is degraded or faulted
`/runbooks/longhorn-degraded`, `/runbooks/longhorn-faulted`.
### 4. An upgrade is stuck mid-switchover
Signature: the operator refusing an update that changes `imageName` and a postgresql parameter together.
Split the change in git. See `/runbooks/update-rollback`.
### 5. The database is up and Keycloak still fails
Then read `/runbooks/keycloak-down` and, more likely, `/runbooks/keycloak-client-secrets` — the
2026-09-06 lockout had all 12 OIDC clients holding literal placeholder secrets behind an entirely green
cluster.
## When to escalate or stop
- **Escalate immediately.** This is SSO's foundation; a 5m `for` on a `critical` is already generous.
- **Never change `dataDurability` to `required`** while trying to "make replication safer" — it converts
one degraded standby into a total write outage on the SSO database.
- **Do not delete a CNPG pod or PVC to force a rebuild.** The operator manages instance lifecycle;
deleting a PVC destroys that instance's data and, if it is the primary, may lose transactions that the
archive does not have.
- **Do not attempt a restore before checking `firstRecoverabilityPoint` and `ContinuousArchiving`.** Prod
ran 38+ hours from bootstrap with `ContinuousArchiving=False` and `firstRecoverabilityPoint: null` on
all three Clusters and nothing said so. Know what you actually have before you destroy what you have.
- **Standbys are not backups.** Recovery is from the Barman archive in R2 (`CNPG_BACKUP_PATH =
s3://k3s-prod-backup/barman`, 30d retention), which is an operator operation.
---
Source of truth: `runbooks/keycloak-db.md`. Index: `/runbooks/`
Records: commits `f338e91` (instances 3), `e8d8601` (required anti-affinity + switchover), `cac413e`
(sync replication, `dataDurability: preferred`); `platform/cnpg/alerts.yaml` header (the 2026-09-06
archiving incident and the measured WAL fill rate).
runbooks/keycloak-down.md
+166
View File
@@ -0,0 +1,166 @@
# keycloak-down — The Keycloak IdP is down (SSO is unavailable)
**Alert:** `KeycloakDown` (critical, 5m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
**Expression:**
```
max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+", condition="true"}) == 0
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+"})
```
## What this means here
No ready Keycloak **server** pod for five minutes. The regex is deliberately scoped to
`keycloak-[0-9]+` (the StatefulSet pods `keycloak-0`, `keycloak-1`) because the `keycloak` namespace also
holds the operator (`keycloak-operator-*`) and the database (`keycloak-db-*`); an unscoped `max()` would
be satisfied by those being ready and would never fire for the IdP itself. That scoping was a fix — the
original matched `.*postgres.*`.
## Blast radius — read this before triaging anything else
Keycloak is the root of trust. With SSO down, all of these are down or degraded **at once**:
- **kubectl OIDC login** (the public `kubernetes` client with PKCE, groups → RBAC)
- **Grafana** (OIDC), **Headlamp** (forwards the user's id_token to the apiserver), **Gitea**
- **forward-auth** via oauth2-proxy, and the five per-app oauth2-proxy sidecars (pihole, jackett,
qbittorrent, radarr, sonarr) — these **fail closed**, 302ing to a Keycloak that is not there
**You can still get in.** The certificate-based kubeconfig is untouched by the OIDC work and remains a
fully independent break-glass path — that is a deliberate design property, not an accident. From this
workstation the working non-interactive path is the dev-container kubeconfig; the host `k3s-oidc`
context will hang on a browser prompt, which is not the cluster being down.
**And you can still read this page.** The runbook route is deliberately **public, with no
forward-auth**, precisely so that `keycloak-down` is readable by an operator who cannot log in. If that
ever changes, this runbook becomes unreachable exactly when it is needed.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Everything in the namespace, in one view — server, operator and database together:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n keycloak get pods -o wide; echo; \
kubectl --context k3s-prod -n keycloak get statefulsets,deployments'
```
Is the database the real cause? (It usually is. Fully-qualified: `clusters.postgresql.cnpg.io`.)
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak \
-o custom-columns=NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase'
```
The server's own complaint:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n keycloak logs keycloak-0 --tail=120 2>&1 | tail -50; echo ====; \
kubectl --context k3s-prod -n keycloak describe pod keycloak-0 | sed -n "/Events:/,\$p"'
```
The operator's view of the CR:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n keycloak get keycloaks.k8s.keycloak.org -o yaml | grep -A20 "status:"'
```
Is it reachable at all, from outside, with an explicit SNI (in-pod HTTPS tests to these hostnames are
misleading here):
```
curl -s -o /dev/null -w '%{http_code}\n' --resolve keycloak.home.keithjackson.dev:443:192.168.2.20 \
https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
```
## Likely causes, most likely first
### 1. The CNPG database is not available
Keycloak cannot start without Postgres. This is the highest-probability cause and it has its own runbook:
`/runbooks/keycloak-db`. Note that `keycloak-db` is a 3-instance CNPG Cluster with
`primaryUpdateMethod: switchover` and synchronous replication at `dataDurability: preferred` —
`preferred` specifically so that one degraded standby does **not** block writes, which on this database
would be a total SSO write outage.
### 2. A node that held the Keycloak pods went away
Both Keycloak pods and all three `keycloak-db` instances previously ran on **one node** — all three CNPG
primaries had drifted onto `k3s-prod-2` — and **on 2026-09-07 that node took SSO, Nextcloud and
Semaphore down together**. The fixes were `instances: 3` on every Cluster and
`podAntiAffinityType: required` (CNPG's default `preferred` is only a scheduler hint and co-locates on
the path that matters — recreating an instance after node loss). Check `-o wide` above: if everything is
on one node, that is a live re-occurrence of the same defect. Go to `/runbooks/node-down`.
### 3. A HelmRelease / operator upgrade went bad
Read `/runbooks/update-rollback`. Under `primaryUpdateMethod: switchover` the CNPG operator **rejects an
update that changes `imageName` and a postgresql parameter in the same reconcile** — a stuck reconcile
with that message is this.
### 4. TLS to the issuer is broken
Keycloak's clients validate the issuer's certificate. While the wildcard was LE-**staging**, in-cluster
clients rejected it with `x509: certificate signed by unknown authority`. If `/runbooks/cert-issuance` or
`/runbooks/cert-expiry` is also firing, fix that first.
### 5. Logins fail but Keycloak is UP
Then this is the **wrong alert and the wrong runbook**, and the distinction has cost this cluster dearly.
On 2026-09-06 every OIDC login on prod failed — *"Invalid client or Invalid client credentials"* — with
**all 12 clients holding the literal `(env:...)` placeholder as their secret**, while
`KeycloakDown`/`KeycloakDatabaseDown` were quiet, the import Job reported success, the CR read
`Done=True`, and the conformance test that asserts `Done=True` **passed throughout**. The only observable
was a human failing to log in. Go to `/runbooks/keycloak-client-secrets`.
Similarly, a `redirect_uri` mismatch after a hostname change is not an outage — same runbook.
And once, MFA broke every platform-admin login on its own: conditional-OTP used a non-existent config
key, giving `"Invalid role name: null"` → 403, fixed to `condUserRole: platform-admin`.
## When to escalate or stop
- **Escalate immediately.** SSO down is a platform-wide outage; treat it as such.
- **Use the certificate-based kubeconfig** (the dev-container path) rather than trying to fix OIDC login
first. Do not burn the outage on getting `kubectl` to authenticate through the thing that is down.
- **Do not edit the realm to "reset" anything.** The realm import is a **ONE-SHOT that fails open**: the
operator logs *"Job already executed - not recreating"*, so a git change to
`keycloak-realm-import.yaml` does **not** re-import. The documented path is `kcadm.sh` inside
`keycloak-0` (admin credentials in the Secret `keycloak-admin-credentials`; `kcadm` is present,
`curl`/`python3` are **not**) **and** fixing git for reproducibility. Doing only one of the two is how
drift is created.
- **Do not delete the Keycloak CR or the StatefulSet.** Recreating the IdP is a restore operation, not a
restart.
- **Do not touch `keycloak-db` volumes.** If the database is the cause, read `/runbooks/keycloak-db` and
`/runbooks/cnpg-archiving`; only the Barman archive is a backup, standbys replicate deletions
faithfully.
- Break-glass local admins exist for gitea, nextcloud and semaphore. Nextcloud has **no native
group→admin mapping**, so platform-admins there are promoted by hand.
---
Source of truth: `runbooks/keycloak-down.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` (the 2026-09-06 client-secret lockout; the 2026-09-07 single-node SSO
outage); commits `f338e91`, `e8d8601`, `cac413e` (CNPG instances/anti-affinity/switchover/sync).
runbooks/kubevirt-control-plane.md
+156
View File
@@ -0,0 +1,156 @@
# kubevirt-control-plane — The KubeVirt control plane is degraded
**Alert:** `KubeVirtControlPlaneDown` (critical, 10m)
**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt`
**Expression:**
```
(sum(kubevirt_virt_controller_ready_status) == 0)
or (sum(kubevirt_virt_api_ready_status) == 0)
or (absent(up{job="kubevirt-prometheus-metrics"} == 1))
```
## What this means here
No `virt-controller` or no `virt-api` replica reports ready, so **VM lifecycle operations — create, start,
stop, migrate — will fail** (docs/06). Note carefully what this does *not* mean: **already-running VMs
keep running.** `virt-launcher` pods and their qemu processes are independent of the control plane. So
this is an "I cannot manage VMs" alert, not "the VMs are down".
The `absent()` clause also fires if the whole KubeVirt metrics target disappears. The scrape job label is
**`kubevirt-prometheus-metrics`** (the Service name), **not** `kubevirt` — that precision is a fix: this
very alert once used a non-existent metric with the wrong job label and fired falsely (commit `b57b3e9`).
There is a sibling precedent too: a broken HTTP CDI ServiceMonitor caused a false `CDIOperatorDown`
(commit `6a7e9d1`). **So the first question with this alert is always whether it is telling the truth.**
Live state, measured 2026-09-13: `kubevirt-prometheus-metrics` **is** among Prometheus's 19 scrape jobs
and `kubevirt_virt_api_ready_status` returns **2 series**, so the first two clauses are evaluable and
quiet. (`kubevirt_vmi_phase_count` was absent at the same moment — that metric only exists when VMIs do,
which affects `/runbooks/kubevirt-launch-failed`, not this rule.)
Platform context: **KubeVirt v1.9.0 + CDI v1.66.0**, on real nested KVM (`/dev/kvm` present, vmx/svm
verified; the Proxmox nodes use `cpu_type=host`). The `kubevirt` and `cdi` namespaces deliberately carry
**no `app-namespace` label**, so Kyverno does not generate the default-deny baseline into them — it would
sever the virt control plane. Do not "fix" that by adding the label.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Is the control plane actually unhealthy, or is this a metrics artefact?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n kubevirt get pods -o wide; echo; \
kubectl --context k3s-prod get kubevirts.kubevirt.io -A \
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,READY:.status.conditions[?\(@.type==\"Available\"\)].status'
```
Are the VMs themselves still running? (This is the reassuring half.)
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get vmi -A -o wide; echo; \
kubectl --context k3s-prod get vm -A'
```
Is the scrape target up, and do the two metrics exist?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
for q in "up{job=\"kubevirt-prometheus-metrics\"}" "sum(kubevirt_virt_controller_ready_status)" "sum(kubevirt_virt_api_ready_status)"; do \
printf "%-55s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \
| python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done'
```
Why a component is unready:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n kubevirt logs deploy/virt-controller --tail=80 2>&1 | tail -30; echo ====; \
kubectl --context k3s-prod -n kubevirt logs deploy/virt-api --tail=80 2>&1 | tail -30; echo ====; \
kubectl --context k3s-prod -n kubevirt get events --sort-by=.lastTimestamp | tail -20'
```
Is the Flux tier that owns KubeVirt healthy?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
-o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
```
## Likely causes, most likely first
### 1. The metrics target, not the control plane
Check the pods first. If `virt-api` and `virt-controller` are Running and Ready, this is a monitoring
fault — and there is direct precedent for false positives on this exact alert (`b57b3e9`) and its CDI
sibling (`6a7e9d1`). Remediation: fix the ServiceMonitor/metric in git. Cross-read
`/runbooks/targets-down`.
### 2. A node that hosted the control-plane pods went away
`virt-controller` and `virt-api` are ordinary Deployments. Go to `/runbooks/node-down`.
### 3. The KubeVirt Flux tier failed to reconcile
Specific known hazard: **`platform/kubevirt` still uses remote kustomize bases**, and the
kustomize-controller's git fetch has a **27s timeout** that remote GitHub bases flake against — a
`BuildFail` there is probably this. `external-snapshotter` was vendored into the repo for exactly this
reason; vendoring KubeVirt the same way is the durable fix. See `/runbooks/flux-reconcile`.
### 4. Node pressure
Heavy sustained node load — TCG-emulated VMs are the recorded example — can degrade the Cilium datapath
into intermittent LB-VIP reachability problems, which looks like control-plane flapping. A restart
cleared it. Cross-read `/runbooks/node-io` and `/runbooks/node-cpu`.
### 5. A bad operator upgrade
`/runbooks/update-rollback`.
### 6. A NetworkPolicy severed the control plane
Only if someone added `homelab.keithjackson.dev/app-namespace=true` to `kubevirt` or `cdi`. Those
namespaces are deliberately unlabelled so Kyverno does not inject a default-deny baseline that would cut
virt-handler off. Reverting that label is the fix.
## When to escalate or stop
- **Prioritise honestly.** VM *lifecycle* is broken; running VMs are not. The UniFi VM keeps serving.
Treat this as urgent-but-not-catastrophic unless a VM also needs to start.
- **Do not drain a node to move virt pods.** This cluster does not drain, and a VM live-migrates on drain
**only** with per-VM `evictionStrategy: LiveMigrate` — a blanket setting would wedge drains for
RWO-disk VMs.
- **Do not delete a VMI or `virt-launcher` pod** while the control plane is down: nothing will bring the
VM back until `virt-controller` is ready again.
- **Do not add the `app-namespace` label to `kubevirt` or `cdi`** to "harden" them.
- Escalate for vendoring the remote bases or for any operator upgrade. If the alert is a false positive,
fixing the rule is itself worth a change — a permanently-firing critical is a recorded, expensive
failure mode here.
---
Source of truth: `runbooks/kubevirt-control-plane.md`. Index: `/runbooks/`
Records: commits `b57b3e9` (this alert's wrong metric/job label) and `6a7e9d1` (false `CDIOperatorDown`);
`DECISIONS-LOG.md` (KubeVirt v1.9.0 + CDI v1.66.0 on nested KVM; the remote-base 27s fetch timeout).
runbooks/kubevirt-crashloop.md
+163
View File
@@ -0,0 +1,163 @@
# kubevirt-crashloop — A VM's virt-launcher pod is restarting repeatedly
**Alert:** `KubeVirtVMCrashLooping` (warning, 10m)
**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt`
**Expression:** `increase(kube_pod_container_status_restarts_total{pod=~"virt-launcher-.*"}[15m]) > 3`
## What this means here
More than three restarts of a `virt-launcher` pod in 15 minutes. The threshold is chosen to distinguish a
crash-**loop** from a one-off reboot: a guest that restarts once is not interesting, a guest that restarts
four times in a quarter of an hour is either panicking, being OOM-killed, or crashing qemu.
Note what layer this watches: the **pod**, not the guest. `virt-launcher` wraps qemu, so a guest kernel
panic, an OOM kill of the launcher, and a qemu crash all present the same way here. The distinction comes
from the logs and the events, not from the alert.
This is a `warning`, correctly: the VM tier is deliberately non-gating (`apps-vms` is `wait: false`) and a
crash-looping VM does not affect the container platform. The one caveat is that on this network the UniFi
VM is the home network's controller, so *that* VM crash-looping is user-visible.
Two platform facts that shape the likely causes:
- These are **HP EliteDesk 800 G5 Minis with ~22 GiB usable each**, running the whole platform. A VM is
the largest discretionary memory consumer on a node, so OOM is a live possibility rather than a
theoretical one.
- **`useEmulation: false` is set deliberately** — on KubeVirt v1.9, `true` silently forces TCG. A VM
running under TCG burns a whole core per vCPU, and heavy sustained TCG load has been observed to degrade
the Cilium datapath into intermittent LB-VIP unreachability (cleared by a restart). So "the VM is
unstable **and** the network is flaky" is usually one cause, not two.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Which launcher, how many restarts, on which node:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get pods -A -l kubevirt.io=virt-launcher -o wide; echo; \
kubectl --context k3s-prod get vmi -A -o wide'
```
Why it died — the previous container's log is the one that matters:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n unifi logs -l kubevirt.io=virt-launcher --previous --tail=120 2>&1 | tail -60'
```
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n unifi describe pod -l kubevirt.io=virt-launcher 2>&1 | sed -n "/Last State/,\$p" | head -40'
```
Was it OOM-killed?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get events -A --field-selector reason=OOMKilling --sort-by=.lastTimestamp | tail -10; echo; \
kubectl --context k3s-prod top pods -A --sort-by=memory | head -10'
```
```
ssh k3s-prod-1 'dmesg -T | grep -iE "oom|killed process" | tail -10; free -h; swapon --show'
```
(An empty `swapon --show` is what you want. On 2026-09-07 all three prod nodes came back from a reboot with
977 MiB of swap **active** while the role reported success — `SwapTotal == 0` is not evidence swap is off.
See `/runbooks/node-memory`.)
Is it really TCG rather than KVM?
```
ssh k3s-prod-1 'ls -l /dev/kvm; grep -c -E "vmx|svm" /proc/cpuinfo; ps -eo pid,pcpu,comm | grep -i qemu'
```
Is the disk healthy underneath?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
-o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state | grep -v healthy'
```
## Likely causes, most likely first
### 1. The guest is panicking or the guest OS is broken
Read the serial console via the previous container's log. A guest that panics at boot after a change is
usually a bad cloud-init or a bad image. Two precise traps here: **KubeVirt caps inline `userData` at 2048
bytes** (the shipped payload is a deliberately minimal ~1.1 KB inline blob; `userDataSecretRef` was tried
and reverted), and **`apps-vms` must NOT carry `postBuild.substituteFrom`** — Flux's envsubst rewrote
`${...}` inside the cloud-init and corrupted it (commit `da1f833`). Do not restore substitution there for
consistency with the app waves.
### 2. The launcher is being OOM-killed
On ~22 GiB nodes running the whole platform, a VM with a generous memory request plus Prometheus (3Gi) and
Grafana (768Mi) is tight. Remediation: lower the VM's memory in git, or address node memory
(`/runbooks/node-memory`). Check swap is genuinely off first — active swap makes the kubelet's accounting
lie and eviction misbehave.
### 3. qemu is crashing
The launcher log will carry the qemu error. Most often a device or firmware mismatch: a UEFI image needs
`firmware.efi`, and `containerDisk` does not work here at all (it requires the ALPHA `ImageVolume` gate,
off on k3s — use a CDI DataVolume). See `/runbooks/kubevirt-launch-failed`.
### 4. The disk is unavailable or degraded mid-flight
A `longhorn-vm` volume that went degraded or faulted takes the guest with it. Cross-read
`/runbooks/longhorn-degraded` and `/runbooks/longhorn-faulted`. Remember `longhorn-vm` is **Block mode
only** and `migratable: true`; a VM on plain `longhorn` cannot live-migrate.
### 5. Node I/O pressure is starving the guest
A guest whose virtual disk cannot complete writes will wedge and be restarted. `/runbooks/node-io` — and
note the guest's writes land on the same partition as the embedded etcd WAL, so a thrashing VM is a
control-plane risk, not just its own problem.
### 6. It is running under TCG
`useEmulation: true` forces it on v1.9. Confirm with `/dev/kvm` presence and qemu CPU usage. Fix in git.
## When to escalate or stop
- **Bound the damage first.** If the VM is thrashing the disk, its restarts are loading the node that
etcd writes to. If `/runbooks/node-io` is also firing, stopping the VM (an operator action on the `VM`
object, `spec.running: false` in git) is the right move — protecting the control plane beats keeping one
VM trying.
- **Do not delete the DataVolume or the PVC.** A re-import is hours of I/O on the shared partition.
- **Do not `kubectl drain`** the node to relocate it. This cluster does not drain, and a VM live-migrates
on drain **only** with per-VM `evictionStrategy: LiveMigrate`; a blanket setting would wedge drains for
RWO-disk VMs.
- Escalate if the guest needs in-guest repair — that is console work, and for the UniFi VM it is also
home-network-visible.
- If the alert is firing for a `virt-launcher` in a leaked test namespace, deleting that namespace is the
fix; leaked TCG test namespaces have caused API blips here before.
---
Source of truth: `runbooks/kubevirt-crashloop.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` Session 7 (`useEmulation`, `firmware.efi`, `containerDisk`/ImageVolume);
commit `da1f833` (cloud-init corrupted by envsubst); the 2026-09-07 swap-persistence finding.
runbooks/kubevirt-launch-failed.md
+173
View File
@@ -0,0 +1,173 @@
# kubevirt-launch-failed — One or more VirtualMachineInstances are in the Failed phase
**Alert:** `KubeVirtVMLaunchFailing` (critical, 5m)
**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt`
**Expression:** `sum by (namespace) (kubevirt_vmi_phase_count{phase="Failed"}) > 0`
## What this means here
A VMI reached the `Failed` phase — a scheduling, launch or qemu failure. The IaaS path is broken for that
workload, which is why it is page-worthy even though the VM tier is deliberately **non-gating**
(`apps-vms` is `wait: false`, `dependsOn: platform-config`, because a VM's image import and in-guest
install are legitimately long-not-Ready).
What actually runs here: the `unifi` namespace holds the UniFi VM (rootdisk 40 GiB, Block mode on
`longhorn-vm`). **UniFi OS Server is not yet installed in the guest** — that is known, documented state,
not a launch failure.
**Live note, measured 2026-09-13:** `kubevirt_vmi_phase_count` returned **0 series** on prod. That metric
only exists while VMIs do, so this alert is currently evaluating against nothing. Confirm the real state
from `kubectl get vmi -A`, not from this alert's silence. (`kubevirt-prometheus-metrics` *is* a live scrape
job and `kubevirt_virt_api_ready_status` does publish, so the target is up — see
`/runbooks/kubevirt-control-plane`.)
### The three k3s-specific launch failures this platform already hit and fixed
Recorded from bring-up, and worth checking against any new VM definition because all three were
non-obvious:
1. **`containerDisk` requires the ALPHA `ImageVolume` feature gate**, which is off on k3s. The fix was to
switch to a **CDI DataVolume on Longhorn** instead. A VM authored with `containerDisk` will not launch
here.
2. **A UEFI image needs `firmware.efi`** set on the VM. Without it, boot fails.
3. **`useEmulation: true` silently forces TCG on KubeVirt v1.9** — it is set to `false` deliberately. A VM
that appears to launch but burns a whole core per vCPU is this; it also degrades the Cilium datapath
under sustained load.
Also recorded: a `virt-handler` certificate error seen during that work was **unrelated and cosmetic** —
do not chase it.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Which VMI, in which namespace, and what phase:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get vmi -A -o wide; echo; \
kubectl --context k3s-prod get vm -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,STATUS:.status.printableStatus,READY:.status.ready'
```
The reason — the VMI status conditions and the launcher pod's events are where it is stated:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n unifi describe vmi unifi 2>&1 | sed -n "/Status:/,\$p" | head -60'
```
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n unifi get pods -l kubevirt.io=virt-launcher -o wide; echo; \
kubectl --context k3s-prod -n unifi describe pod -l kubevirt.io=virt-launcher 2>&1 | sed -n "/Events:/,\$p"'
```
Is the disk there? (A missing or still-importing DataVolume is the most common cause.)
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get datavolumes.cdi.kubevirt.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PHASE:.status.phase,PROGRESS:.status.progress; echo; \
kubectl --context k3s-prod get pvc -A | grep -Ei "rootdisk|dv|unifi"'
```
Is nested virtualisation actually available on the node it was scheduled to?
```
ssh k3s-prod-1 'ls -l /dev/kvm; grep -c -E "vmx|svm" /proc/cpuinfo'
```
Is the storage class right for a live-migratable VM?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get sc; echo; \
kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy'
```
And whether the KubeVirt control plane can act at all:
```
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kubevirt get pods'
```
## Likely causes, most likely first
### 1. The disk is not ready — a CDI import that has not completed or has stalled
The VM cannot start without its DataVolume. Go to `/runbooks/cdi-import-stuck`. The specific recorded
failure: the **3-replica default class failed the UniFi 60Gi import with insufficient storage /
`ReplicaSchedulingFailure`**, which is why a dedicated single-replica **`longhorn-scratch`** class exists
and is referenced by the CDI CR's `scratchSpaceStorageClass`.
### 2. The VM is authored in a way k3s does not support
The three items above: `containerDisk` without the ALPHA `ImageVolume` gate (use a CDI DataVolume),
missing `firmware.efi` for a UEFI image, or `useEmulation: true` forcing TCG. All are git changes to the
VM definition.
### 3. The cloud-init payload is too large or corrupted
Two precise traps. **KubeVirt caps inline `userData` at 2048 bytes** — the original rich cloud-init
exceeded it; `userDataSecretRef` was tried and reverted, and the shipped state is a minimal ~1.1 KB inline
payload. And: **`apps-vms` must NOT carry `postBuild.substituteFrom`** — Flux's envsubst rewrote `${...}`
inside the cloud-init and corrupted it (commit `da1f833`). *Do not restore substitution there for
consistency with the app waves.* If a VM stopped booting right after someone "tidied up" the Flux
Kustomizations, this is it.
### 4. It cannot be scheduled
No node with room, or a storage constraint. A live-migratable VM needs **`longhorn-vm`**: RWX,
**Block mode only** (migratable RWX is a raw block device, not the share-manager/NFS path), annotated
`storageclass.kubevirt.io/is-default-virt-class: "true"` — which is *not* the cluster default class.
A VM pointed at plain `longhorn` will not live-migrate.
### 5. The CDI importer cannot open the raw Block device
`nonroot-devices: true` is set on the k3s servers because the non-root CDI importer must open a raw Block
device that Kubernetes never chowns for it (commit `df5979f`). A permission error in the importer on a
Block PVC is this.
### 6. The KubeVirt control plane is down
Then nothing can launch. `/runbooks/kubevirt-control-plane`.
## When to escalate or stop
- **Scope it before escalating.** The VM tier is non-gating and does not affect the container platform.
One failed VM is not a platform outage — but the UniFi VM is the home network's controller, so treat
that one as user-visible.
- **Do not delete the VM or its DataVolume to "start clean".** A 60 GiB re-import is hours of I/O on disks
that share a partition with the embedded etcd WAL, and that is the mechanism behind the 2026-09-07 node
outage. Fix the definition, not the data.
- **Do not set `evictionStrategy: LiveMigrate` cluster-wide.** A blanket setting would **wedge drains**
for RWO-disk VMs; it is per-VM on purpose.
- **Do not add `postBuild.substituteFrom` to `apps-vms`.** It corrupts cloud-init.
- Escalate for anything that needs a fresh image import during a storage incident, and for changes to the
k3s server flags (`nonroot-devices`) — those are Ansible/operator work.
---
Source of truth: `runbooks/kubevirt-launch-failed.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` Session 7 (the three k3s-specific launch failures); commits `da1f833`
(cloud-init corrupted by envsubst) and `df5979f` (`nonroot-devices`); the `longhorn-scratch` /
UniFi 60Gi import finding.
runbooks/kustomization.yaml
+86
View File
@@ -0,0 +1,86 @@
---
# Runbooks tier — reconciled by its OWN non-gating post-platform Flux Kustomization
# (clusters/base/runbooks.yaml, dependsOn platform-config, wait:false). Same placement and rationale as
# platform/homepage: a leaf web server must never sit in the gating `platform` tier, where a UI's
# unhealth once stalled the whole chain (DECISIONS-LOG: "Headlamp gating stall").
#
# WHY THE MANIFESTS LIVE HERE, ALONGSIDE THE DOCS, RATHER THAN IN platform/runbooks/.
# The markdown is the source of truth and the task is to serve it verbatim, which means the ConfigMap must
# be GENERATED from these files — there must not be a second hand-maintained copy of 34 documents that can
# drift from the originals. `configMapGenerator` does that, and its `files:` paths are resolved relative to
# THIS directory. A `platform/runbooks/kustomization.yaml` would therefore need `../../runbooks/<name>.md`,
# which depends on kustomize load restrictions being relaxed — true for Flux's kustomize-controller,
# refused by a plain `kustomize build`, and so a divergence between what the gate can verify and what the
# cluster does. Making this directory the Flux path removes the question entirely: no `../`, identical
# behaviour locally and in-cluster, one copy of every document. The serving manifests are tucked under
# `serving/` so `runbooks/*.md` stays exactly the flat set of pages the URLs name.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: runbooks
resources:
- serving/namespace.yaml
- serving/nginx-config.yaml
- serving/deployment.yaml
- serving/service.yaml
# default-deny baseline + Cilium gateway-ingress allow (DNS-only egress from the baseline).
- serving/networkpolicy.yaml
- serving/httproute.yaml # apex `/runbooks` prefix (binds the Gateway `https-apex` listener)
configMapGenerator:
# ONE ConfigMap holding every runbook, keyed by the EXACT name the alert annotations use.
#
# THE KEYS ARE EXTENSIONLESS ON PURPOSE. The annotations are `https://${BASE_DOMAIN}/runbooks/<name>`
# with NO `.md`, and this repo must not change them (they are spread across four alerts.yaml files that
# other work is editing). `<key>=<file>` maps `node-down.md` on disk to the key `node-down`, the volume
# mount turns each key into a file under the nginx root, and `/runbooks/node-down` becomes a plain static
# lookup — no rewrite rule, nothing to get subtly wrong at 3am. Drop the `<key>=` and every key would
# carry `.md`, so every alert link would 404 while this file still looked correct.
#
# The generated name carries kustomize's content hash, so editing any runbook rolls the server
# automatically and a stale projection is impossible.
#
# ADDING A RUNBOOK: add the file AND its line here. The L0 check
# conformance/tests/l0-static/alert-runbooks-resolvable asserts both halves, because a file in git with
# no generator entry is reviewed, merged, apparently done — and still a 404 in the cluster.
- name: runbooks-content
options:
labels:
app.kubernetes.io/name: runbooks
files:
# The index, served at /runbooks/ (nginx `try_files /runbooks/README`).
- README=README.md
- backup-failure=backup-failure.md
- backup-missed=backup-missed.md
- backup-partial=backup-partial.md
- cdi-import-stuck=cdi-import-stuck.md
- cert-expiry=cert-expiry.md
- cert-issuance=cert-issuance.md
- cnpg-archiving=cnpg-archiving.md
- cnpg-backup-missed=cnpg-backup-missed.md
- cnpg-exporter=cnpg-exporter.md
- cnpg-primary-spread=cnpg-primary-spread.md
- cnpg-wal-growth=cnpg-wal-growth.md
- denied-flows=denied-flows.md
- flux-reconcile=flux-reconcile.md
- flux-suspended=flux-suspended.md
- image-automation=image-automation.md
- image-policy-stalled=image-policy-stalled.md
- keycloak-client-secrets=keycloak-client-secrets.md
- keycloak-db=keycloak-db.md
- keycloak-down=keycloak-down.md
- kubevirt-control-plane=kubevirt-control-plane.md
- kubevirt-crashloop=kubevirt-crashloop.md
- kubevirt-launch-failed=kubevirt-launch-failed.md
- longhorn-capacity=longhorn-capacity.md
- longhorn-degraded=longhorn-degraded.md
- longhorn-faulted=longhorn-faulted.md
- monitoring-capacity=monitoring-capacity.md
- node-cpu=node-cpu.md
- node-disk=node-disk.md
- node-down=node-down.md
- node-io=node-io.md
- node-memory=node-memory.md
- pvc-capacity=pvc-capacity.md
- targets-down=targets-down.md
- update-rollback=update-rollback.md
- vpn-down=vpn-down.md
runbooks/longhorn-capacity.md
+158
View File
@@ -0,0 +1,158 @@
# longhorn-capacity — A Longhorn volume or node is near its storage capacity
**Alerts:** `LonghornVolumeCapacityHigh` (warning, 30m) and `LonghornNodeStorageHigh` (warning, 30m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn`
**Expressions:**
`(longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85`
`(longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85`
## What this means here
Either one volume is 85% full, or one node's Longhorn disk is 85% allocated. They are different
problems with different fixes, so read the labels: `volume` means a workload is filling its own PVC;
`node` means the cluster is running out of places to put replicas.
The node case is bounded by settings that are already conservative:
`storageReservedPercentageForDefaultDisk: 20`, `storageMinimalAvailablePercentage: 25`,
`storageOverProvisioningPercentage: 200`, `defaultDataLocality: disabled`, hard node anti-affinity.
Measured headroom on prod is roughly **269 GiB scheduled against an 829 GiB ceiling**, so the node
alert firing means something has changed materially, not that the platform has slowly grown.
The severe consequence of running out of node storage is that a degraded volume **cannot finish
rebuilding** — hard anti-affinity needs three distinct nodes with room, so one full node pins a
volume at 2 replicas indefinitely. Cross-read `/runbooks/longhorn-degraded`.
### Verify the alert can even fire (measured 2026-09-13)
Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`,
`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`,
`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`,
`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`,
`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`.
There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this
rule depends on before you trust its silence:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \
sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query'
```
**These alerts are currently INERT.** `count(longhorn_volume_capacity_bytes)` and
`count(longhorn_node_storage_capacity_bytes)` both returned **0 series** on prod on 2026-09-13 —
Longhorn ships no ServiceMonitor here. Volume-fill coverage actually comes from
`kubelet_volume_stats_*` instead, i.e. `/runbooks/pvc-capacity`, which is where the working alerts
are. Use this runbook for the Longhorn-side (node allocation) half, and expect to reach it by hand.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Node-level allocation (the half that has no substitute alert):
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o yaml | grep -E "name:|storageAvailable|storageMaximum|storageScheduled|allowScheduling"'
```
Volume sizes versus actual usage:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
-o custom-columns=NAME:.metadata.name,SPEC:.spec.size,ACTUAL:.status.actualSize,ROBUST:.status.robustness'
```
Which PVC each one is, so you know who is filling it:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SIZE:.spec.capacity.storage,SC:.spec.storageClassName'
```
The filesystem view, which is the alert that actually works:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
curl -s -G --data-urlencode "query=topk(10, 1 - kubelet_volume_stats_available_bytes{job=\"kubelet\"} / kubelet_volume_stats_capacity_bytes{job=\"kubelet\"})" localhost:19090/api/v1/query'
```
On the node itself:
```
ssh k3s-prod-1 'df -h /data; du -xh --max-depth=1 /data/longhorn/replicas 2>/dev/null | sort -h | tail'
```
## Likely causes, most likely first
### 1. A CNPG database is not recycling WAL (check this FIRST)
The only cause here that grows without bound. `archive_mode=on` plus a failing `archive_command` means
Postgres will not recycle unarchived WAL; measured on prod `keycloak-db` at ~6.8 MiB/h with `pg_wal`
sharing the data PVC (no `walStorage:` is configured on any Cluster) — about 29 days from 93% free to
disk-full, ending in a postmaster PANIC. Go to `/runbooks/cnpg-archiving`. **Never delete WAL by
hand.**
### 2. Snapshots and orphaned replicas accumulating
Longhorn snapshots are not free, and cleanup here is **manual**: `orphanAutoDeletion: true` in
`platform/longhorn/helmrelease.yaml` has been **silently inert since Longhorn 1.12** — it was renamed
`orphanResourceAutoDeletion` (now a list) and the chart drops unknown `defaultSettings` keys with no
error. Verify the key's current state in that file before citing it; then prune orphans and stale
snapshots from the Longhorn UI.
### 3. A workload genuinely grew
Plex/Sonarr/Radarr **media** does not live on Longhorn — it is static NFS from
`192.168.2.102:/mnt/Storage/plex`, via the `bulk` tier. If a media volume is on `longhorn`, that is a
misconfiguration worth fixing, not a capacity problem: the `bulk` StorageClass is a stub
(`provisioner: nfs.csi.k8s.io`, `server:/share: REPLACE-ME`) and provisions nothing, so real bulk
storage is static NFS PVs. Otherwise expand the PVC — all Longhorn classes have
`allowVolumeExpansion: true`; change the size in git and push.
### 4. Replicas piled onto one node
`replicaAutoBalance: best-effort` should prevent this, and hard anti-affinity bounds it to one replica
per volume per node. If one node is much fuller than the others, check whether a rebuild placed
replicas there while another node was down.
### 5. The cluster has outgrown its disks
Prod's ceiling is ~829 GiB across three nodes. `docs/plans/longhorn-ssd-tiering.md` is the planned
answer (a ~51 GB hot tier: Prometheus, CNPG, Loki, Gitea). Operator planning work.
## When to escalate or stop
- **A node over 85% while any volume is degraded is urgent**, because the rebuild cannot complete.
Free space or restore the other node before anything else.
- **Never delete files under `/data/longhorn`** or inside a Postgres `pg_wal`. Both are irreversible.
- **Do not expand a PVC as a reflex** if the cause is unrecycled WAL — you are buying days and hiding
the real fault.
- Escalate for adding disks, the SSD tiering plan, or anything that needs a node reboot (each reboot
degrades all ~18 volumes and triggers a rebuild storm).
---
Source of truth: `runbooks/longhorn-capacity.md`. Index: `/runbooks/`
Records: `platform/longhorn/helmrelease.yaml` (the settings above and the inert `orphanAutoDeletion`);
`platform/cnpg/alerts.yaml` header (the WAL disk-full clock); `docs/BULK-NFS-STORAGE.md`.
runbooks/longhorn-degraded.md
+170
View File
@@ -0,0 +1,170 @@
# longhorn-degraded — A Longhorn volume has been degraded for 10 minutes
**Alert:** `LonghornVolumeDegraded` (warning, 10m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn`
**Expression:** `longhorn_volume_robustness == 2` (0=unknown 1=healthy 2=degraded 3=faulted)
## What this means here
A volume has fewer healthy replicas than it should. It is **still serving I/O** — that is the
difference between degraded and faulted (`/runbooks/longhorn-faulted`). What you have lost is
redundancy, and on a 3-node cluster with hard node anti-affinity
(`replicaSoftAntiAffinity: false`, `numberOfReplicas: 3`) one degraded volume means one more node
loss away from data loss.
**Degraded is the NORMAL state after any node reboot on this cluster, and it is expensive.**
Recorded plainly in the stabilization notes: *"Longhorn on the HDD: every node reboot degrades all 18
volumes → full rebuild (slow; 30%+ IO-delay). Roll reboots one node at a time; wait
`robustness=healthy` before the next."* The rebuild is the thing that hurts, not the degradation:
rebuild writes land on the same partition as the embedded etcd WAL (`/data/k3s` and `/data/longhorn`
share one device), and that is the mechanism behind the 2026-09-07 `k3s-prod-1` NotReady outage. So
the question to answer is never just "why is it degraded" but "is the rebuild about to take a node
out". Cross-read `/runbooks/node-io`.
### Verify the alert can even fire (measured 2026-09-13)
Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`,
`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`,
`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`,
`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`,
`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`.
There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this
rule depends on before you trust its silence:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \
sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query'
```
**This alert is currently INERT.** `count(longhorn_volume_robustness)` returned **0 series** on prod
on 2026-09-13: the whole `platform-health.longhorn` rule group has no data, because Longhorn ships no
ServiceMonitor here. That is a separate open defect, and the practical consequence is that **you will
probably learn about a degraded volume from an application symptom or from `volumes.longhorn.io`, not
from this alert**. Do not infer "no alert, so volumes are fine".
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
The truth about robustness, straight from the CRs (this works whether or not the metric does):
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
-o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state,NODE:.status.currentNodeID,SIZE:.spec.size'
```
Which replicas are missing, and where:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n longhorn-system get replicas.longhorn.io \
-o custom-columns=NAME:.metadata.name,VOL:.spec.volumeName,NODE:.spec.nodeID,RUNNING:.status.currentState | sort -k2'
```
Which PVC/workload is affected:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SC:.spec.storageClassName'
```
Are the nodes and their disks schedulable, and is a rebuild in flight?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o wide; echo; \
kubectl --context k3s-prod -n longhorn-system get settings.longhorn.io concurrent-replica-rebuild-per-node-limit -o custom-columns=NAME:.metadata.name,VALUE:.value'
```
And the pressure that rebuild is causing — the reason this alert matters:
```
ssh k3s-prod-1 'cat /proc/pressure/io'
```
## Likely causes, most likely first
### 1. A node rebooted or was briefly NotReady (overwhelmingly the most common)
Expected, and it self-heals by rebuilding. Your job is to keep the rebuild from taking a node down.
Remediation (operator, mutating) — freeze rebuilds while replicas will **reattach** (an in-place
bounce):
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \
concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"'
```
Thaw to `"1"` — never higher — once every volume reads `healthy`. Use `1` rather than `0` when the
replicas are genuinely gone (an OS re-provision), so replenishment can actually proceed.
`concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and
held**; do not "fix" this by setting a knob that is already set. `replicaRebuildingBandwidthLimit` is
**v2/SPDK-only** and does nothing here; `rebuildConcurrentSyncLimit` is already at its floor of 1.
What was actually tightened after that incident was every *other* Longhorn I/O source:
`concurrentVolumeBackupRestorePerNodeLimit` and `snapshotHeavyTaskConcurrentLimit` 5 → 1.
### 2. A node is still down
Then the rebuild cannot complete and this will not clear. Go to `/runbooks/node-down` first; there is
nothing to fix in Longhorn.
### 3. A disk was replaced and Longhorn refuses it
Signature: `DiskFilesystemChanged` — *"record diskUUID doesn't match the one on the disk"*.
Remediation, in exactly this order (the validating webhook enforces it): **disable the disk → remove
the disk entry → re-add it**. Seen during the 2026-09-07 drive swap.
### 4. No node can hold the replica (anti-affinity or space)
Hard anti-affinity means a 3-replica volume needs three distinct schedulable nodes. With one node
cordoned or its disk over `storageMinimalAvailablePercentage: 25`, the third replica cannot be
placed and the volume stays degraded indefinitely. Check `nodes.longhorn.io` scheduling and
`/runbooks/longhorn-capacity`.
### 5. `replicaAutoBalance: best-effort` is shuffling replicas
Normal background behaviour, transient. If degradation follows a pattern of replicas moving and never
settling, that is worth an operator's attention.
## When to escalate or stop
- **If more than one volume is degraded and I/O-PSI on any node is climbing, freeze rebuilds first
and then get the operator.** That combination is the 2026-09-07 failure mode in progress.
- **Never reboot, cordon or re-provision a second node while any volume is degraded.** This is the
single most important rule in this runbook.
- **Do not `kubectl drain`.** This cluster deliberately does not drain: there is no Longhorn
`nodeDrainPolicy` configured anywhere in the repo, the upgrade plan rejects
system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD Longhorn", and the
node-loss conformance test evicts only its own writer pod.
- **Do not delete a degraded volume's replicas to "force a clean rebuild"** unless the operator agrees
— with 3 replicas and one already missing you are one mistake from `/runbooks/longhorn-faulted`.
- Escalate for: a node that will not come back, a disk replacement, or a volume degraded for more than
a few hours with all three nodes Ready (that is a Longhorn bug, not an operational state).
---
Source of truth: `runbooks/longhorn-degraded.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` "2026-08-31 — Post-upgrade stabilization" (the reboot tax) and
"2026-09-07 — k3s-prod-1 NotReady"; `docs/plans/cluster-upgrade-plan.md` §6.2/§7 (the freeze/thaw).
runbooks/longhorn-faulted.md
+164
View File
@@ -0,0 +1,164 @@
# longhorn-faulted — A Longhorn volume is FAULTED (data at risk, volume offline)
**Alert:** `LonghornVolumeFaulted` (critical, 1m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn`
**Expression:** `longhorn_volume_robustness == 3`
## What this means here
Faulted means Longhorn has **no usable replica**. The volume is offline; the workload's writes are
failing. This is the most serious storage state on the cluster and the `for: 1m` is deliberate — there
is no benign transient version of this.
Understand what is and is not recoverable before you touch anything:
- The volumes on this cluster are 3-replica with **hard** node anti-affinity, so faulted normally
means two or three replicas are gone at once — two nodes lost, or one node lost while another
volume's rebuild had not finished, or a disk that ate its replica.
- **Standbys and replicas are not backups.** Recorded verbatim for the databases and true for all of
this: *"Standbys replicate corruption and deletions faithfully; only the Barman archive is a
backup."* For a CNPG volume (`keycloak-db`, `nextcloud-db`, `semaphore-db`) the real recovery path
is the object-store archive, not Longhorn. For everything else it is Velero.
- If this is a **`longhorn-vm`** volume (RWX Block, `migratable: true`) the VM is down, not just the
disk. If it is **`longhorn-scratch`** (single replica, CDI scratch space) losing it is expected and
harmless — re-run the import.
### Verify the alert can even fire (measured 2026-09-13)
Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`,
`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`,
`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`,
`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`,
`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`.
There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this
rule depends on before you trust its silence:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \
sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query'
```
**This alert is currently INERT.** `count(longhorn_volume_robustness)` returned **0 series** on prod
on 2026-09-13 — Longhorn ships no ServiceMonitor here, so the whole `platform-health.longhorn` group
has no data. A faulted volume today announces itself as an application outage. That is a known open
defect; treat `volumes.longhorn.io` as the source of truth, not this alert's silence.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Stop and take inventory before changing anything. Which volume, which workload, which replicas exist:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
-o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state,NODE:.status.currentNodeID | grep -Ev "healthy"'
```
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n longhorn-system get replicas.longhorn.io \
-o custom-columns=NAME:.metadata.name,VOL:.spec.volumeName,NODE:.spec.nodeID,STATE:.status.currentState,FAILED:.spec.failedAt | sort -k2'
```
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SC:.spec.storageClassName | grep <volume-name>'
```
What recovery material exists — **fully-qualified names, or you will be told there is nothing**:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get backups.velero.io -A; echo; \
kubectl --context k3s-prod get backups.postgresql.cnpg.io -A; echo; \
kubectl --context k3s-prod -n longhorn-system get backups.longhorn.io'
```
`kubectl get backup -A` prints "No resources found" on this cluster even when six Velero and 27 CNPG
backups exist. Never make a recovery decision from the short name.
Node and disk state:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get nodes; kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o wide'
```
## Likely causes, most likely first
### 1. Two nodes were unavailable at once
Usually a second reboot or cordon while the first node's rebuilds were still running. Remediation:
bring the nodes back. Longhorn will reattach if the replica data survived on disk — which is why you
must not delete replicas before checking.
### 2. A disk failed and took its replica with it
`k3s-prod-1`'s original 2011 SSD had 61,440 reallocated sectors before it was replaced; the dev-era
Seagate had 448 current-pending plus 448 offline-uncorrectable. Check SMART on the node holding the
failed replica (`/runbooks/node-io` has the command). If the disk is bad, the replica is gone — the
path forward is restore, not repair.
### 3. All replicas were evicted or deleted
Check whether a disk was disabled, a node removed from Longhorn, or a `replicas.longhorn.io` object
deleted. `staleReplicaTimeout: 30` (minutes) means Longhorn eventually discards replicas it considers
stale.
### 4. It is a single-replica volume by design
`longhorn-scratch` is `numberOfReplicas: 1`. Losing it is not an incident — delete the DataVolume and
re-import.
## Remediation, in order
1. **Do not delete the volume, the PVC, or any replica.** A faulted volume with a stale replica is
recoverable; a deleted one is not.
2. If a node is down, restore the node (`/runbooks/node-down`). That is the highest-probability fix.
3. If Longhorn shows a replica with data but marked failed, an operator can salvage it from the
Longhorn UI (*Volume → Salvage*) — this is the documented path for "all replicas failed but the
data is still there".
4. If the volume backs a **CNPG database**, do not try to salvage Postgres files. Recover the Cluster
from the Barman archive (`/runbooks/cnpg-archiving` for how to confirm the archive is real first).
5. Otherwise restore from Velero. Note honestly: `dr-restore` is **authored and has never been
executed** (`conformance.home.k3s/never-executed`), and the R2 restore path has not been rehearsed.
Expect to be the first person to run it, and read `/runbooks/backup-failure` for the R2 plugin
caveats before you rely on a backup's metadata.
## When to escalate or stop
- **Get the operator now.** This is a critical, data-at-risk alert; it is not a runbook-and-carry-on
situation. Everything above is diagnosis, and steps 3–5 are operator actions.
- **Stop before any destructive command.** Specifically: no `delete pvc`, no `delete
volumes.longhorn.io`, no `delete replicas.longhorn.io`, no touching files under `/data/longhorn`.
- **Do not reboot anything** while a volume is faulted.
- If the affected volume is `keycloak-db`, expect a full SSO outage as a knock-on: kubectl OIDC,
Grafana, Headlamp, Gitea and forward-auth all depend on it. The certificate-based kubeconfig is an
independent break-glass path and still works.
---
Source of truth: `runbooks/longhorn-faulted.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` (drive failures 2026-08-22 and 2026-09-07; `dr-restore` never executed).
runbooks/monitoring-capacity.md
+133
View File
@@ -0,0 +1,133 @@
# monitoring-capacity — The monitoring stack's own PVC is filling up
**Alert:** `MonitoringVolumeFillingUp` (warning, 15m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.monitoring-self`
**Expression:** `kubelet_volume_stats_available_bytes{namespace="monitoring"} / kubelet_volume_stats_capacity_bytes{namespace="monitoring"} < 0.15`
## What this means here
"Don't let the monitor blind itself." Prometheus or Loki is running out of disk. If it fills, you lose
the ability to see every other alert in this set — which is why `monitoring` gets its own rule instead
of sharing `/runbooks/pvc-capacity` (the two files partition the namespace space between them; the
thresholds are deliberately identical, 15% free for 15m).
Live shape on prod:
- `prometheus-...-prometheus-0` PVC **50Gi**, `retention: 30d`, `retentionSize: 45GiB`
- `storage-loki-0` PVC **50Gi**
- both on the `longhorn` class, 3 replicas
`retentionSize: 45GiB` against a 50Gi PVC is intentional: Prometheus is *supposed* to plateau near the
high-water mark and then stay there. **So a Prometheus TSDB approaching 45 GiB is normal, not a leak.**
That is also why the predictive rule in `pvc-capacity-alerts.yaml` deliberately **excludes** the
`monitoring` namespace: a TSDB filling toward its retention limit looks exactly like an unbounded leak
to `predict_linear` until it plateaus.
So the real question is *which* PVC and *whether it should be plateauing*: read the
`persistentvolumeclaim` label first. Loki has no size cap analogous to `retentionSize`, so a log
volume growing past 85% is the more likely genuine problem of the two.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring get pvc; echo; \
kubectl --context k3s-prod -n monitoring get prometheus -o jsonpath="{.items[0].spec.retention} {.items[0].spec.retentionSize}{\"\n\"}"'
```
Actual on-disk usage from inside the pods:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- df -h /prometheus; \
kubectl --context k3s-prod -n monitoring exec loki-0 -- df -h /var/loki'
```
Is Prometheus at its retentionSize (normal) or past it (a problem)?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
for q in "prometheus_tsdb_storage_blocks_bytes" "prometheus_tsdb_head_series" "rate(prometheus_tsdb_compactions_failed_total[1h])"; do \
printf "%s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]])"; done'
```
Series growth is the usual driver — find the biggest contributors:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
curl -s "localhost:19090/api/v1/status/tsdb" | python3 -m json.tool | head -40'
```
## Likely causes, most likely first
### 1. Prometheus has reached its retention high-water mark (expected)
`retentionSize: 45GiB` on a 50Gi PVC leaves 10% headroom, which is close to the 15% trigger. If
`prometheus_tsdb_storage_blocks_bytes` is near 45 GiB and steady, **this is the design working** and
the correct response is either to accept it or to raise both the PVC and `retentionSize` together in
`platform/monitoring/kube-prometheus-stack.yaml`. Raising retention without raising the PVC is how you
get a genuinely full disk.
### 2. Loki log volume growth
Loki has no retentionSize equivalent configured. A noisy workload (a crashlooping pod, a chatty
reconciler) can grow it quickly. Remediation: find the noisy stream in Grafana → Explore, fix the
source, and set/lower Loki retention in git.
### 3. Series cardinality grew after a chart or workload change
`serviceMonitorSelectorNilUsesHelmValues` and `podMonitorSelectorNilUsesHelmValues` are both `false`
here, which means **Prometheus selects every ServiceMonitor/PodMonitor in the cluster regardless of
labels**. That is convenient (no `additionalLabels` needed anywhere) and it also means a new
component's monitor is picked up automatically, cardinality and all. Check
`/api/v1/status/tsdb` for the top label pairs.
### 4. Compaction is failing
`rate(prometheus_tsdb_compactions_failed_total[1h]) > 0` means blocks are not being merged or expired.
Usually an I/O problem underneath — cross-read `/runbooks/node-io`.
### 5. The volume is on a node that is itself short of space
`/runbooks/longhorn-capacity` and `/runbooks/node-disk`.
## When to escalate or stop
- **Do not delete Prometheus TSDB blocks or Loki chunks by hand.** Lower retention and let the
component expire them; hand-deleting blocks can leave the TSDB unopenable.
- **Do not raise `retention` or `retentionSize` without raising the PVC in the same change.**
- If Prometheus is already full and crashlooping, you are blind: check the alerts you care about
directly against the CRs (`clusters.postgresql.cnpg.io`, `volumes.longhorn.io`, `kubectl get nodes`)
rather than waiting for alerts that cannot be evaluated.
- Escalate for PVC expansion — it needs node headroom, and this cluster has been memory- and
disk-constrained before (Prometheus was already raised 2 → 3Gi of RAM after the kps 65→88 upgrade).
---
Source of truth: `runbooks/monitoring-capacity.md`. Index: `/runbooks/`
Records: `platform/monitoring/pvc-capacity-alerts.yaml` (why monitoring is excluded from the
predictive rule); `DECISIONS-LOG.md` (Grafana OOM / Prometheus 2→3Gi during kps 65→88).
runbooks/node-cpu.md
+125
View File
@@ -0,0 +1,125 @@
# node-cpu — CPU pressure or run-queue depth high on a node
**Alerts:** `NodeCPUPressureHigh` (warning, 15m) and `NodeCPULoadHigh` (warning, 15m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
**Expressions:**
`rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.15`
`node_load5 / count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1`
## What this means here
`NodeCPUPressureHigh` is PSI "some" for CPU: the fraction of wall-clock time during which at least
one task was stalled waiting for a CPU. Under 5% is healthy. Averaged CPU% — what `kubectl top`
shows — **hides this entirely**, which is why the alert uses PSI: it is the metric that surfaces the
"spikes cause blips of delay" symptom operators actually notice.
`NodeCPULoadHigh` is 5-minute load average divided by core count. Sustained above 1 means the run
queue is longer than the number of CPUs.
**Read them together.** CPU-PSI high *and* load high = genuine CPU contention. Load high *and* CPU-PSI
normal = the queue is full of tasks blocked on **I/O**, not CPU — go to `/runbooks/node-io`, which on
this cluster is the more dangerous of the two (etcd WAL and Longhorn replicas share one partition).
Measured history: steady CPU PSI ran **15–23%** with load at or above core count, which is what drove
the node resize from **6 to 8 vCPU**; afterwards PSI measured ~4.5–7.7%. So 0.15 is not an arbitrary
threshold, it is the level the cluster actually lived at before the resize.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Distinguish CPU contention from I/O contention — do this first:
```
ssh k3s-prod-1 'cat /proc/pressure/cpu; echo; cat /proc/pressure/io; echo; uptime; nproc'
```
Who is burning it:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod top nodes; echo; kubectl --context k3s-prod top pods -A --sort-by=cpu | head -20'
```
```
ssh k3s-prod-1 'top -b -n1 | head -20'
```
Are there VMs, and are any of them emulated (the worst case)?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get vmi -A -o wide'
```
```
ssh k3s-prod-1 'grep -c -E "vmx|svm" /proc/cpuinfo; ls -l /dev/kvm'
```
## Likely causes, most likely first
### 1. It is actually I/O, not CPU
Most common misread. A node blocked on disk has a long run queue and normal CPU-PSI. Confirm with
`/proc/pressure/io` and go to `/runbooks/node-io`. On 2026-09-07 the node that died had I/O-PSI full
at 11.36 while CPU looked unremarkable.
### 2. A KubeVirt VM running under TCG emulation
`useEmulation: true` **silently forces TCG on KubeVirt v1.9** — it is set to `false` on purpose. A VM
that lost `/dev/kvm` or was authored with emulation burns a whole core per vCPU. Heavy sustained TCG
load has also degraded the Cilium datapath into intermittent LB-VIP unreachability (cleared by a
restart), so an "intermittent networking" report alongside this alert is probably one cause, not two.
Check `grep -c -E "vmx|svm" /proc/cpuinfo` (nested KVM is verified working; Proxmox nodes use
`cpu_type=host`).
### 3. A Longhorn rebuild's checksumming
Rebuilds are I/O-heavy but not free on CPU. Same remediation as `/runbooks/longhorn-degraded`: freeze
with `concurrent-replica-rebuild-per-node-limit` `0`, thaw to `1`.
### 4. Prometheus rule evaluation or a Loki query
`PersistentVolumeFillingUpPredicted` runs `predict_linear` over a 24h window across every PVC; a
Grafana panel doing the same interactively is heavier still. Bursty, self-limiting, usually harmless.
### 5. Genuine capacity
If nothing is anomalous the node is simply undersized again. The 6 → 8 vCPU resize is the precedent.
A bpg cores change **reboots the VM**, so roll **one node at a time, waiting for Longhorn
`robustness=healthy` between each**.
## When to escalate or stop
- **If I/O-PSI is also elevated, stop treating this as a CPU problem** and switch to
`/runbooks/node-io`. Wrong runbook is the main risk here.
- Alone and steady, this alert is a capacity signal — schedule it, do not page on it. That is why it
is a warning with a 15m `for`.
- **Do not drain or cordon** to shed CPU load; this cluster does not drain.
- Escalate to the operator for a vCPU change (it reboots the node) or to kill a VM you did not create.
---
Source of truth: `runbooks/node-cpu.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` (node vCPU 6→8 and the post-resize PSI measurements; KubeVirt
`useEmulation` finding).
runbooks/node-disk.md
+131
View File
@@ -0,0 +1,131 @@
# node-disk — A node filesystem is filling up
**Alert:** `NodeDiskFillingUp` (warning, 15m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
**Expression:** `node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{...} < 0.10`
## What this means here
Less than 10% free on a real filesystem on a node, for 15 minutes. Read the `mountpoint` label
first, because on these nodes it tells you which of two very different problems you have.
**`/data` is the one that matters.** Both the embedded **etcd WAL** (`/data/k3s`) and **every
Longhorn replica** (`/data/longhorn`) live on that single partition —
`node_baseline_etcd_source: shared-data-partition`, not a dedicated device, and the exit condition
for splitting them is still open (`infra/PROD-PROVISIONING.md` §2). If `/data` fills:
- etcd cannot write its WAL. A control-plane member with a full WAL partition does not degrade
gracefully; it stops, and this is a 3-member cluster.
- Longhorn stops being able to hold replicas and volumes go degraded, which triggers rebuilds, which
need *more* space.
So `/data` at 10% free is not a warning, it is the last comfortable moment to act.
**`/` filling up** is usually logs, container images or a stuck journal. Lower stakes, but there is
history: in the dev era the host root filled to ~99% because VM disks sat under a snapper-snapshotted
subvolume, and the hypervisor **paused the k3s VMs on I/O error**. Recovery was deleting 11 snapper
timeline snapshots plus `virsh destroy` on the paused VMs (which were holding rsync-unlinked inodes):
99% → 30%.
Longhorn's own guard rails are already tight and are why this alert usually fires before anything
breaks: `storageReservedPercentageForDefaultDisk: 20`,
`storageMinimalAvailablePercentage: 25`, `storageOverProvisioningPercentage: 200`.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Which filesystem, on which node:
```
ssh k3s-prod-1 'df -h -x tmpfs -x overlay; echo; du -xh --max-depth=1 /data 2>/dev/null | sort -h | tail -10; echo; du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -10'
```
What Longhorn thinks it has:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,SCHED:.spec.allowScheduling; echo; \
kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,SIZE:.spec.size,ACTUAL:.status.actualSize,ROBUST:.status.robustness'
```
Journal and image bloat on `/`:
```
ssh k3s-prod-1 'sudo journalctl --disk-usage; sudo crictl images | wc -l; sudo du -sh /var/lib/rancher/k3s/agent/containerd 2>/dev/null'
```
## Likely causes, most likely first
### 1. Longhorn replicas grew, or replicas landed here after a rebuild
Hard node anti-affinity (`replicaSoftAntiAffinity: false`) means one replica per node per volume, but
`replicaAutoBalance: best-effort` moves them. Prod sits around 269 GiB scheduled against an 829 GiB
ceiling, so this is rarely tight — but a rebuild storm plus snapshots can spike it.
Remediation: check for orphaned replicas and stale snapshots in the Longhorn UI. Note
`orphanAutoDeletion: true` in `platform/longhorn/helmrelease.yaml` **has been silently inert since
Longhorn 1.12** — it was renamed `orphanResourceAutoDeletion` (a list) and the chart drops unknown
`defaultSettings` keys with no error. So orphan cleanup is manual. Verify the key's current state in
the file before you trust it.
### 2. A CNPG database is not recycling WAL
The single most dangerous cause, because it is unbounded. With `archive_mode=on` and a failing
`archive_command`, Postgres **refuses to recycle** unarchived WAL segments and `pg_wal` grows forever
— on prod's `keycloak-db` that was measured at ~6.8 MiB/h, 17 segments / ~268 MiB, about 29 days to
fill a 5 GiB PVC. `pg_wal` shares the data PVC (no `walStorage:` is configured on any Cluster).
Check `/runbooks/cnpg-archiving` **before** you free any space, and **never delete WAL by hand** —
that destroys the database.
### 3. Journal / container images on `/`
`journalctl --vacuum-size=200M` and pruning unused images are safe. Do not prune images for
workloads that may need to restart while the registry is unreachable.
### 4. A Velero or CDI staging write
Velero's kopia uploads and CDI's scratch space both stage locally. `longhorn-scratch`
(single-replica) exists precisely because the 3-replica default failed the UniFi 60Gi import on
insufficient storage.
### 5. The disk is smaller than the platform now needs
`docs/plans/longhorn-ssd-tiering.md` holds the planned hot tier (~51 GB: Prometheus, CNPG, Loki,
Gitea). Working set measured ~79 GB against ~323 GB free on SSD, so the "won't fit" blocker was a
measurement artifact. This is operator planning work, not an incident fix.
## When to escalate or stop
- **If the mountpoint is `/data` and free space is under 5%, treat it as an imminent control-plane
outage** and get the operator. etcd and Longhorn are both on it.
- **Never delete anything under `/data/longhorn` or a Postgres `pg_wal` directory by hand.** Both are
irreversible. Free space elsewhere, or fix the cause.
- **Do not `kubectl drain`** the node to move load off it — this cluster does not drain.
- If the cause is a CNPG archive failure, this runbook is the wrong one: go to
`/runbooks/cnpg-archiving`, which is the actual fix.
---
Source of truth: `runbooks/node-disk.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` "Disk/drive incident + k3s relocated to NVMe (2026-08-22)";
`platform/cnpg/alerts.yaml` header (the WAL disk-full clock).
runbooks/node-down.md
+156
View File
@@ -0,0 +1,156 @@
# node-down — A node's node-exporter is unscrapable (the node may be gone)
**Alert:** `NodeExporterDown` (critical, 5m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
**Expression:** `up{job="node-exporter"} == 0`
## What this means here
Prometheus has not scraped node-exporter on that instance for 5 minutes. The alert is worded
carefully — it says the *exporter* is down — because on this cluster the two cases look identical
from Prometheus and have opposite urgency:
- **The node is genuinely gone** (powered off, kernel panic, network loss).
- **The node is up and the kubelet stopped posting status.** This is what actually happened on
**2026-09-07 at 06:05:21Z**: `k3s-prod-1` went NotReady with "Kubelet stopped posting node status"
while the host was up, pingable and SSH-able. Root cause was disk-I/O starvation of the embedded
etcd, not a dead machine. If this alert is firing alongside `node-io`, read `/runbooks/node-io`
first — that is the cause and this is the symptom.
**Why it is critical rather than a warning.** All three nodes are control-plane + etcd members
(`k3s-prod-1` .10, `k3s-prod-2` .11, `k3s-prod-3` .12, kube-vip API VIP `192.168.2.9`). Losing one
leaves a 2-member etcd with no fault tolerance: the next loss ends quorum and the API server with it.
Losing a node also degrades every Longhorn volume that had a replica there, and every degraded volume
starts rebuilding — which loads the same disks etcd fsyncs to. One node down is not a routine event
on a 3-node cluster; it is one step from an outage.
Two node-specific roles matter when you read which node it is:
- `k3s-prod-3` is the **sole VPN egress gateway**. Cilium OSS has **no egress-gateway failover**, so
while that node is down, VPN-classed egress is an outage, not a degradation. See `/runbooks/vpn-down`.
- `k3s-prod-2` is the cluster **founder** (`[k3s_prod_primary]` in `inventory/hosts.ini`). Do not
re-provision it from the inventory without reading `/runbooks/node-io` — re-provisioning a node
still listed as primary founds a **second cluster**.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Is the node actually NotReady, or is only the exporter unscrapable?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get nodes -o wide; echo; kubectl --context k3s-prod -n monitoring get pods -l app.kubernetes.io/name=prometheus-node-exporter -o wide'
```
Is it the API server or the node? (If this hangs, you have a bigger problem than one node.)
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get --raw /readyz?verbose 2>&1 | tail -20'
```
Is the host alive at all? Prod is directly on `192.168.2.0/24` — no NAT hop:
```
ping -c3 192.168.2.10 ; ssh k3s-prod-1 'uptime; systemctl is-active k3s; cat /proc/pressure/io'
```
The three lines that distinguish "etcd starved" from "machine dead":
```
ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|Kubelet stopped|leader"'
```
Etcd membership and leader — do this before touching anything:
```
ssh k3s-prod-2 'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"'
```
What did the node take with it?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
-o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy; \
kubectl --context k3s-prod get pods -A --field-selector spec.nodeName=k3s-prod-1 -o wide | head -30'
```
## Likely causes, most likely first
### 1. Disk I/O starvation wedged the kubelet (host still up)
The signature is `slow fdatasync took=3.56s` / `apply request took too long took=8.30s` /
`etcdserver: too many requests` in `journalctl -u k3s`, with `/proc/pressure/io` full avg300 well
above 1. On 2026-09-07 node 1 read **11.36** against 0.75 on node 2 and 1.29 on node 3.
Remediation: follow `/runbooks/node-io`. Recovery that day required a manual
`systemctl restart k3s` on the node (operator, on-host), after which 18 Longhorn volumes rebuilt and
pressure only fell when the rebuilds finished.
### 2. The node rebooted (planned or otherwise)
Check `uptime`. A reboot is not harmless here: **every node reboot degrades all ~18 Longhorn volumes
and triggers a full rebuild**. If a reboot is intentional, roll one node at a time and wait for
`robustness=healthy` on every volume before the next, and freeze rebuilds first (see
`/runbooks/longhorn-degraded`). Note two hardware quirks recorded in `infra/PROD-PROVISIONING.md`
§0b: `k3s-prod-2` **hangs on a warm reboot while staying powered**, and all three nodes carry the EFI
memattr kernel bug that once presented `k3s-prod-1` as having 2.6 GiB usable of 16 GB (needs kernel
≥ 6.13).
### 3. node-exporter itself is unhealthy, node fine
If `kubectl get nodes` says Ready and the pod is CrashLoopBackOff or Pending, this is a monitoring
fault, not a node fault. Downgrade urgency, but do not ignore it — while that exporter is down you
are blind to `node-io`, `node-memory`, `node-disk` and `node-cpu` on that node, which is exactly the
blindness that let 2026-09-07 arrive as a surprise. Related: `/runbooks/targets-down`.
### 4. Network loss to the node
`ping` fails, SSH fails. Check the switch and the node's `eno1` (prod-3's
`/etc/network/interfaces` also holds a commented-out wireless stanza — inert, but worth scrubbing).
A node reachable by ping but not by the API is usually Cilium, not the network: heavy sustained node
load has been observed to degrade the Cilium datapath into intermittent LB-VIP unreachability,
cleared by a restart.
### 5. Hardware
`k3s-prod-1`'s original 2011 SSD had 61,440 reallocated sectors and was replaced 2026-09-07. Check
SMART (`/runbooks/node-io` has the command). Hardware is an operator job.
## When to escalate or stop
- **STOP if two nodes are affected.** Two of three etcd members down means no quorum, no API server,
and no GitOps. Nothing in this runbook applies; get the operator.
- **Do not reboot, drain or cordon a second node** while any Longhorn volume is not `healthy`. This
cluster deliberately does not `kubectl drain` — the auto-drain of system-upgrade-controller was
rejected precisely because it fights slow-disk Longhorn.
- **Do not delete the Node object** to "clean up". That triggers replica deletion in Longhorn.
- Escalate to the operator for: a power-cycle, a disk replacement, or re-provisioning a node (the
inventory's `[k3s_prod_primary]` trap above makes re-provisioning a two-person job).
- If the node returns but volumes stay degraded for more than an hour, go to
`/runbooks/longhorn-degraded`.
---
Source of truth: `runbooks/node-down.md` in `home-network-k3s-impl`. Index: `/runbooks/`
Incident record: `DECISIONS-LOG.md` "2026-09-07 — k3s-prod-1 NotReady"; `infra/PROD-PROVISIONING.md` §0b/§0c.
runbooks/node-io.md
+183
View File
@@ -0,0 +1,183 @@
# node-io — Disk I/O pressure / starvation on a node
**Alerts:** `NodeIOPressureHigh` (warning, 10m) and `NodeIOPressureCritical` (critical, 5m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
**Expression:** `rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05` (warning) / `> 0.10` (critical)
## What this means here
PSI **full** for I/O: the fraction of wall-clock time in which *every* runnable task on the node was
blocked on disk, i.e. the machine did no useful work. It is not "the disk is busy" — it is "the node
is stalled".
This alert exists because of a real outage it would have caught and could not. On **2026-09-07 at
06:05:21Z, `k3s-prod-1` went NotReady** ("Kubelet stopped posting node status") while the host stayed
up, pingable and SSH-able. The k3s journal showed embedded etcd starved of disk:
```
slow fdatasync took=3.56s / 6.71s / 4.95s / 3.40s (etcd expects 1s)
apply request took too long took=8.30s (expects 100ms)
etcdserver: too many requests
Failed to check local etcd status for learner management: context deadline exceeded
```
`/proc/pressure/io` **full avg300** at the time: `k3s-prod-1` **11.36**, `k3s-prod-2` (PM9A1 NVMe)
**0.75**, `k3s-prod-3` (WD Blue) **1.29**, against a ~1% steady state. The stack alerted on CPU-PSI
and memory and had **no I/O alert at all**, so the first signal the operator got was a node already
out of the cluster. The thresholds here are those measurements, not guesses: 0.05 is 5x steady
state, 0.10 is the level node 1 actually held while etcd stalled.
**Why this is self-reinforcing, and why it is urgent.** On every node the etcd WAL (`/data/k3s`) and
all Longhorn replicas (`/data/longhorn`) sit on **one partition** — `node_baseline_etcd_source:
shared-data-partition`, not a dedicated device, and §2's exit condition in
`infra/PROD-PROVISIONING.md` is still **open**. So: node stalls → drops out → its volumes go degraded
→ Longhorn rebuilds them → more writes to the same spindle etcd fsyncs to → worse. Recovery on
2026-09-07 needed a manual `systemctl restart k3s`, and I/O pressure only fell when the 18 rebuilds
finished.
**There is no etcd fsync metric to check.** `etcd_disk_wal_fsync_duration_seconds` is not collected —
`kubeEtcd` is deliberately disabled in `platform/monitoring/kube-prometheus-stack.yaml` for k3s. PSI
is the only signal that exists. Do not go looking for the etcd dashboard; it is not there.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Which node, and how bad:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get nodes -o wide'
```
Read PSI and per-device queue depth on the node itself (SSH direct — prod is on
`192.168.2.0/24` with no NAT hop; `.10` = prod-1, `.11` = prod-2, `.12` = prod-3):
```
ssh k3s-prod-1 'cat /proc/pressure/io; echo; iostat -x 1 3; echo; lsblk -o NAME,MOUNTPOINT,ROTA'
```
Is etcd already stalling? This is the line that decides urgency:
```
ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|leader"'
```
Is Longhorn rebuilding (the usual writer)?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
-o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,NODE:.status.currentNodeID | grep -v healthy'
```
Is the disk itself dying? (This is what it was in September.)
```
ssh k3s-prod-1 'sudo smartctl -a /dev/sda | grep -Ei "Reallocated|Pending|Uncorrectable|Wear|Power_On_Hours|Model"'
```
## Likely causes, most likely first
### 1. A Longhorn replica rebuild storm (most common by far)
Every node reboot degrades all ~18 volumes and triggers a full rebuild. That is the normal cost of
Longhorn on these disks, and it is why reboots are rolled **one node at a time, waiting for
`robustness=healthy` on every volume before the next**.
Remediation — freeze rebuilds while the node recovers (operator, mutating):
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \
concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"'
```
Thaw to `"1"` — never higher — once all volumes read `healthy`. Use `0` when replicas will
**reattach** (an in-place bounce); use `1` when they are genuinely gone (an OS re-provision).
`concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and
held** — do not "fix" this by setting a knob that is already set.
### 2. The node's disk is failing
`k3s-prod-1`'s original 2011 `C400-MTFDDAC512MAM` had **61,440 reallocated sectors**, 31% endurance
consumed and no power-loss protection. The operator replaced it with a Samsung 850 EVO on 2026-09-07
and the same 18-replica rebuild measured **3.79** instead of 11.36 — ~3x better, node stayed Ready.
The earlier dev-era drive (Seagate `ST4000DM000`: 29,408 reallocated, 448 pending, 448 offline
uncorrectable) drove etcd to 48% iowait. If SMART shows reallocated or pending sectors climbing,
**the fix is hardware**; nothing in software recovers it.
When replacing a disk, two traps recorded from that rebuild:
- Longhorn refuses the new disk with `DiskFilesystemChanged` ("record diskUUID doesn't match the one
on the disk"). Remedy: **disable → remove the disk entry → re-add, in that order** (the validating
webhook enforces it).
- `inventory/hosts.ini` decides `--cluster-init` via `[k3s_prod_primary]`. Re-provisioning a node
that is still listed there **founds a second cluster**. The founder now sits on `k3s-prod-2`.
### 3. A Velero backup or restore running against the same spindle
Velero's daily runs at `0 3 * * *` and reads Longhorn PVCs through kopia. Check
`backups.velero.io` (fully qualified — see above) for one in flight, and the Longhorn limits
`concurrentVolumeBackupRestorePerNodeLimit` / `snapshotHeavyTaskConcurrentLimit`, both pinned to 1
for exactly this reason.
### 4. A CDI image import or a TCG-emulated VM
A CDI import writes a whole disk image; heavy sustained node load from TCG-emulated VMs has also been
observed to degrade the Cilium datapath into intermittent LB-VIP unreachability (cleared by a
restart). Check `kubectl --context k3s-prod get vmi -A` and importer pods in `cdi`.
### 5. Structural: etcd and Longhorn share one partition
This is the standing cause underneath the other four, and it is open work. `roles/etcd_io_priority`
+ `playbooks/etcd-io-priority.yml` apply BFQ on the shared disk, `nr_requests` 64→32 and
`IOWeight=10000` on `system.slice` and `k3s.service`, reading every value back. **It applies nothing
until an operator runs it** — run `k3s-prod-3` first, with `--limit` and `--list-hosts`.
Four mechanisms are **proven inert on these nodes** — do not re-try them:
`IOSchedulingClass=realtime` on `k3s.service` is a footgun (ioprio is inherited across fork/exec and
containerd is a child of the k3s server process, so realtime for k3s means realtime for Longhorn
too); `IOSchedulingPriority` is a no-op under `mq-deadline`; `io.latency` is unavailable
(`CONFIG_BLK_CGROUP_IOLATENCY` is not built in Debian trixie); a bare `IOWeight=` drop-in is
decoration (blk-iocost compiled-but-disabled, BFQ not loaded). And no cgroup or scheduler control
arbitrates the **jbd2 journal** that etcd's `fdatasync` actually waits on.
## When to escalate or stop
- **Stop and get the operator** if `NodeIOPressureCritical` is firing *and* the journal shows
`slow fdatasync` or `apply request took too long`. The node is minutes from leaving the cluster.
With one of three etcd members already gone, a second loss ends quorum.
- **Do not reboot or drain a second node** while any volume is not `healthy`. That is how a
single-node problem becomes a data problem on a 3-replica cluster.
- **Do not `kubectl drain`.** This cluster does not drain: the posture is deliberate (the upgrade
plan rejects system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD
Longhorn"), and the node-loss conformance test evicts only its own writer pod.
- Hand over to the operator for anything physical (disk replacement) or for the Ansible role — both
are outside what can be done from git.
---
Source of truth: `runbooks/node-io.md` in `home-network-k3s-impl`. Index of all runbooks: `/runbooks/`
Incident record: `DECISIONS-LOG.md` "2026-09-07 — k3s-prod-1 NotReady: etcd starved of disk I/O by
Longhorn", and `infra/PROD-PROVISIONING.md` §0c / §2.
runbooks/node-memory.md
+123
View File
@@ -0,0 +1,123 @@
# node-memory — Memory pressure high on a node
**Alert:** `NodeMemoryPressureHigh` (warning, 10m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
**Expression:** `(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90`
## What this means here
Available memory has been under 10% for ten minutes. These are HP EliteDesk 800 G5 Minis with
~22 GiB usable, running the whole platform plus KubeVirt VMs on three nodes, so the headroom is
genuinely thin and this alert is a capacity signal rather than a bug signal most of the time.
Two cluster-specific facts change how you read it:
**`SwapTotal == 0` is not evidence swap is off.** On 2026-09-07 all three prod nodes came back from a
reboot with **977 MiB of swap ACTIVE** (`/dev/sda4` on prod-1 and prod-3, `/dev/nvme0n1p4` on prod-2)
while the Ansible role's commented-out fstab line reported success.
`systemd-gpt-auto-generator` claims any GPT partition typed
`0657fd6d-a4ab-43c4-84e5-0933c84b4f4f` straight from the partition table and synthesises a `.swap`
unit into `/run/systemd/generator.late/` — it never reads `/etc/fstab`. The fix masks `swap.target`
(the single choke point for gpt-auto, fstab-generator and cryptsetup-generator). If a node is
swapping, the kubelet's memory accounting is lying to you and eviction will misbehave. Check
explicitly with `swapon --show`, never `SwapTotal`.
**There is a kernel memory-reporting bug on all three nodes.** `infra/PROD-PROVISIONING.md` §0b
records an EFI memattr defect that once presented `k3s-prod-1` as having **2.6 GiB usable of 16 GB**.
It needs kernel ≥ 6.13 and all three nodes are exposed (prod-3 publishes `MEMATTR=0xbeece018` too).
If `MemTotal` looks absurd rather than merely tight, suspect this before suspecting a leak.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod top nodes; echo; kubectl --context k3s-prod get nodes -o custom-columns=NAME:.metadata.name,MEM:.status.capacity.memory,PRESSURE:.status.conditions[?\(@.type==\"MemoryPressure\"\)].status'
```
Who is using it — and is swap actually off?
```
ssh k3s-prod-1 'free -h; echo; swapon --show; echo "(empty output above = swap really off)"; echo; cat /proc/pressure/memory; echo; ps -eo pid,rss,comm --sort=-rss | head -15'
```
Biggest consumers in the cluster, and anything that has been OOM-killed:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod top pods -A --sort-by=memory | head -20; echo; \
kubectl --context k3s-prod get events -A --field-selector reason=OOMKilling --sort-by=.lastTimestamp | tail -20'
```
## Likely causes, most likely first
### 1. A monitoring component grew past its limit
This is the one with real history. When kube-prometheus-stack went 65 → 88 it shipped Grafana 13 and
**Grafana OOM-looped** — "the hidden blocker that stalled the whole kps upgrade and the platform
tier". Limits went 256 → 512 → **768Mi** for Grafana and Prometheus 2 → **3Gi** (commit `b2b1aa5`).
Prod keeps 30d retention / 45GiB retentionSize on a 50Gi PVC, so Prometheus is legitimately the
largest single consumer. Remediation: raise the limit in
`platform/monitoring/kube-prometheus-stack.yaml` and push; do not restart it in a loop.
### 2. A KubeVirt VM, especially a TCG-emulated one
VMs are the largest discretionary consumer. `useEmulation: false` is set deliberately (on v1.9
`true` silently forces TCG), but leaked test namespaces running emulated VMs have caused API blips
before and were deleted. Check `kubectl --context k3s-prod get vmi -A` and the `virt-launcher-*` pods'
memory.
### 3. Swap silently back on after a reboot
See above. Verify with `swapon --show` and `systemctl is-enabled swap.target` (should be `masked`).
`node_health` now reports `memory.swap_persistence` for this. The fix is the Ansible role; an
operator runs it.
### 4. Genuine capacity exhaustion
If nothing is anomalous, the node is simply full. The node vCPU count was already raised 6 → 8 for
CPU pressure; memory is next. A bpg/OpenTofu memory change **reboots the VM**, so roll **one node at
a time and wait for Longhorn `robustness=healthy` between each** — the same discipline as every other
reboot here.
### 5. A leak in an app-tier workload
Nextcloud, Plex and the *arr stack are the usual suspects. Compare `top pods` against the previous
day in Grafana rather than guessing.
## When to escalate or stop
- **Escalate immediately if this is firing together with `node-io` or `node-down`** on the same node.
Memory pressure plus I/O pressure on an etcd member is the shape of the 2026-09-07 outage.
- **Do not add swap** to relieve this. Kubernetes does not want it here and the platform has already
been bitten by swap re-enabling itself.
- **Do not evict or drain** to free memory: this cluster does not drain (Longhorn rebuild cost). Kill
the specific offending pod instead, and only if you know what it is.
- Hand to the operator for anything requiring a VM resize or an Ansible run.
---
Source of truth: `runbooks/node-memory.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` (Grafana OOM, kps 65→88); commits `0e92d25`/`9d00a3f` (swap persistence);
`infra/PROD-PROVISIONING.md` §0b (EFI memattr).
runbooks/pvc-capacity.md
+144
View File
@@ -0,0 +1,144 @@
# pvc-capacity — A PersistentVolumeClaim is filling up, nearly full, or predicted to fill
**Alerts:** `PersistentVolumeFillingUp` (warning, 15m), `PersistentVolumeAlmostFull` (critical, 15m),
`PersistentVolumeFillingUpPredicted` (warning, 1h)
**File:** `platform/monitoring/pvc-capacity-alerts.yaml`, group `platform-health.pvc-capacity`
**Expressions:** `< 0.15` free / `< 0.05` free / `< 0.50` free **and**
`predict_linear(kubelet_volume_stats_available_bytes[24h], 7*24*3600) < 0`, each `and
kubelet_volume_stats_inodes{job="kubelet"} > 0`
## What this means here
These are the PVC fill alerts for everything **outside** the `monitoring` namespace
(`/runbooks/monitoring-capacity` owns that half). Before this file existed, **no PVC outside
`monitoring` had any fill alert at all**. Unlike the Longhorn rule group, these work: they are built
on `kubelet_volume_stats_*`, which is live (56 series on prod).
Three design facts you need in order to read the labels correctly:
- **Every rule pins `job="kubelet"`.** `kubelet_volume_stats_*` is published by *two* jobs with
different label shapes: `job="kubelet"` carries the PVC's real namespace, while `job="apiserver"`
reports `namespace="default"` with the real one in `exported_namespace`. Without the pin,
`namespace!="monitoring"` double-fires on monitoring's own PVCs and names the wrong namespace.
- **Every rule carries `and kubelet_volume_stats_inodes > 0`.** Block-mode PVCs — KubeVirt VM disks
on `longhorn-vm` — report available=0, used=0, inodes=0 against a real capacity. Without the guard,
`unifi/unifi-rootdisk` (40 GiB Block) would fire both warning **and** critical permanently. If you
are ever tempted to remove that clause, this is why it is there.
- **The predicted rule is the point of the exercise.** A level threshold is useless for a slow leak on
a mostly-empty volume: prod's `keycloak-db` sat at **93% free** while leaking ~6.8 MiB/h of
unrecyclable WAL, so `<15%` was ~29 days away and would have arrived as a surprise. The `< 0.50`
gate keeps it quiet on volumes that are not plausibly about to fill, and the 24h fit / 7d horizon is
chosen so a daily backup or log rotation does not make it flap.
**If the PVC name matches `*-db-N`, read `/runbooks/cnpg-archiving` before doing anything else.**
That is by far the most likely cause on this cluster and the one where the obvious action is fatal.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Which PVC, how full, on which node:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
curl -s -G --data-urlencode "query=topk(10, 1 - kubelet_volume_stats_available_bytes{job=\"kubelet\"} / kubelet_volume_stats_capacity_bytes{job=\"kubelet\"})" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "persistentvolumeclaim|namespace|\"1\"" '
```
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get pvc -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SIZE:.spec.resources.requests.storage,SC:.spec.storageClassName,STATUS:.status.phase'
```
Is it a CNPG data volume? (Fully-qualified `clusters.postgresql.cnpg.io` — the short name `cluster` is
ambiguous on clusters carrying more than one `Cluster` kind.)
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A'
```
If yes, the one question that decides everything — is WAL archiving working?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
```
What is actually consuming the space, from inside the pod:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c "df -h /var/lib/postgresql/data; du -sh /var/lib/postgresql/data/pgdata/pg_wal; ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | wc -l"'
```
## Likely causes, most likely first
### 1. A CNPG database is not recycling WAL
The dominant cause. With `archive_mode=on` and a failing `archive_command`, Postgres **refuses to
recycle** unarchived WAL segments, so `pg_wal` grows until the data PVC is full and the postmaster
**PANICs** — taking the database, and for `keycloak-db` all of SSO, down. `pg_wal` shares the data
PVC on every Cluster here (no `walStorage:` is configured), which is why a *backup* fault presents as
a *capacity* alert. This actually happened: prod ran from bootstrap to 2026-09-07 with
`endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, all three Clusters at `ContinuousArchiving=False` with
`barman-cloud-wal-archive: exit status 4`, unnoticed for 38+ hours.
**Remediation: fix archiving (`/runbooks/cnpg-archiving`). Do NOT delete WAL by hand — that destroys
the database. Do NOT expand the PVC and call it done; that buys days.**
### 2. Log or cache growth in an app-tier workload
Nextcloud, the *arr apps, Gitea. Prune inside the app, or expand the PVC in git (every Longhorn class
has `allowVolumeExpansion: true`).
### 3. Media on the wrong storage class
Media is meant to be static NFS PVs from `192.168.2.102:/mnt/Storage/plex` (8.32 TiB measured), not
Longhorn. A media volume on `longhorn` is a misconfiguration; fix the class, do not grow the disk.
### 4. A Velero or CDI staging write
Transient. CDI uses `longhorn-scratch` deliberately (single replica) because the 3-replica default
failed the UniFi 60Gi import on insufficient storage.
### 5. The predicted rule caught a genuine slow leak
This is the rule working as designed and giving you a week. Find what is growing before the level
thresholds arrive.
## When to escalate or stop
- **`PersistentVolumeAlmostFull` on a `*-db-N` volume is an imminent database outage.** Get the
operator. A Postgres data volume at under 5% free will PANIC the postmaster.
- **Never delete WAL segments, and never delete anything under a Postgres data directory.**
- **Do not delete the PVC** to reclaim space.
- If `unifi-rootdisk` or another Block-mode volume appears in these alerts, the inodes guard has been
removed or broken — that is a monitoring bug, not a storage incident.
- Escalate for PVC expansion beyond what the node can hold (see `/runbooks/longhorn-capacity`).
---
Source of truth: `runbooks/pvc-capacity.md`. Index: `/runbooks/`
Records: `platform/monitoring/pvc-capacity-alerts.yaml` (the two-job and inodes findings);
`platform/cnpg/alerts.yaml` header (the 2026-09-06 archiving incident and the measured fill rate).
runbooks/serving/deployment.yaml
+151
View File
@@ -0,0 +1,151 @@
---
# The runbook static server. Deliberately the dumbest thing that can work.
#
# WHY A POD AT ALL, AND NOT A REDIRECT TO GITEA. The obvious cheaper option is an HTTPRoute whose
# only job is a RequestRedirect into the in-cluster Gitea web UI, which already renders these files.
# It was rejected for three reasons, in order of weight:
# 1. DEPENDENCY DIRECTION. A runbook is read when the platform is broken. Gitea is an app-tier
# workload on Longhorn; its own DR path and several of these runbooks' subjects sit UNDER it.
# `runbooks/longhorn-faulted` must be readable while Longhorn is faulted, and
# `runbooks/keycloak-db` while a CNPG database is down. Pointing the operator at Gitea makes
# the documentation fail in exactly the conditions that produce the alert.
# 2. NO SUFFIX REWRITE EXISTS. Gateway API's RequestRedirect path modifier offers ReplaceFullPath
# and ReplacePrefixMatch only. There is no "append .md", so mapping 34 extensionless names onto
# 34 `.md` blobs needs 34 hand-maintained ReplaceFullPath rules — MORE surface than this pod,
# and a silent 404 for every runbook someone forgets to add a rule for.
# 3. A 302 is not a document. The annotation's contract is that the URL returns the runbook; a
# redirect chain through a second app's auth and rendering is a worse contract.
# This pod reads only from a ConfigMap projected out of git. It depends on the CNI, the kubelet and
# the Gateway, and on nothing else in the platform.
#
# WHY NOT THE SERVICE CATALOG (catalog/*.yaml + render.py). The catalog is the idiomatic onboarding
# surface for a SERVICE, and it was the first thing checked — but its schema cannot express this
# route. `ingressRoutes[].host` is pattern-bound to `^[a-z0-9-]+\.(\$\{BASE_DOMAIN\}|home\.keithjackson\.dev)$`:
# a single DNS label plus the apex, so the BARE apex is unrepresentable, and there is no `path` field
# at all, so a path-prefix route cannot be authored. Bending the schema for one docs server would be
# the tail wagging the dog. The repo's OTHER established pattern is exactly right instead: a small
# hand-authored platform component reconciled by its own non-gating Flux Kustomization — what
# platform/homepage does for the apex `/` prefix, which this mirrors file for file.
apiVersion: apps/v1
kind: Deployment
metadata:
name: runbooks
namespace: runbooks
labels:
app.kubernetes.io/name: runbooks
spec:
# TWO replicas with anti-affinity, unlike homepage's one. This is the only workload in the
# cluster whose whole purpose is to be available during an incident, and a single replica is
# unavailable for the minutes a node reboot or a drain takes — which is when it is wanted. It
# costs 2 x 16Mi.
replicas: 2
revisionHistoryLimit: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
selector:
matchLabels:
app.kubernetes.io/name: runbooks
template:
metadata:
labels:
app.kubernetes.io/name: runbooks
spec:
automountServiceAccountToken: false
# Spread across nodes so losing one node never takes the runbooks with it. `preferred`, not
# `required`: on a 3-node cluster with one node cordoned for the very incident being debugged,
# a hard rule would leave a replica Pending instead of running somewhere.
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
app.kubernetes.io/name: runbooks
# Tolerate a node that is under memory/disk pressure or unreachable for a while. An operator
# reading `runbooks/node-disk` is by definition dealing with a node under pressure, and the
# default 300s unreachable toleration is long enough; these exist so eviction pressure does
# not silently remove the last replica while the cluster is sick.
tolerations:
- key: node.kubernetes.io/unreachable
operator: Exists
effect: NoExecute
tolerationSeconds: 900
securityContext:
runAsNonRoot: true
runAsUser: 101 # `nginx` in nginxinc/nginx-unprivileged
runAsGroup: 101
fsGroup: 101
seccompProfile:
type: RuntimeDefault
containers:
- name: nginx
# Renovate maintains this pin. nginx-unprivileged (not nginx) so the container can honour
# runAsNonRoot: the stock nginx image's entrypoint needs root to drop privileges and to
# bind :80. `stable-alpine` branch, pinned exactly.
image: nginxinc/nginx-unprivileged:1.30.4-alpine
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8080
protocol: TCP
resources:
requests:
cpu: 10m
memory: 16Mi
limits:
memory: 64Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
# Achievable here, unlike homepage/headlamp: nginx only needs /tmp (its pid file, on the
# unprivileged image) and /var/cache/nginx writable, and both are emptyDirs below.
readOnlyRootFilesystem: true
livenessProbe:
httpGet: { path: /healthz, port: http }
initialDelaySeconds: 3
periodSeconds: 15
readinessProbe:
# Probe a REAL runbook, not just /healthz. nginx answers /healthz from its own config
# even if the ConfigMap volume failed to project, so /healthz alone would report a pod
# that serves 404 for every runbook as Ready. `node-down` is the canonical alert link
# (NodeExporterDown, platform/monitoring/alerts.yaml), so if this path 404s the pod is
# useless and must not take traffic.
httpGet: { path: /runbooks/node-down, port: http }
initialDelaySeconds: 3
periodSeconds: 15
volumeMounts:
- name: nginx-config
mountPath: /etc/nginx/conf.d/default.conf
subPath: default.conf
readOnly: true
# The runbooks themselves, one extensionless file per key. Whole-volume mount (NOT
# subPath): subPath mounts do not update when the ConfigMap changes, and here the
# ConfigMap name carries kustomize's content hash so a content change rolls the pod
# anyway — the whole-volume mount just means a stale projection is impossible.
- name: runbooks
mountPath: /usr/share/nginx/html/runbooks
readOnly: true
- name: tmp
mountPath: /tmp
- name: cache
mountPath: /var/cache/nginx
volumes:
- name: nginx-config
configMap:
name: runbooks-nginx
- name: runbooks
configMap:
# Generated by ../kustomization.yaml's configMapGenerator from runbooks/*.md. kustomize
# rewrites this reference to the hashed name, so the pods roll whenever a runbook changes.
name: runbooks-content
- name: tmp
emptyDir: {}
- name: cache
emptyDir: {}
runbooks/serving/httproute.yaml
+87
View File
@@ -0,0 +1,87 @@
---
# North-south ingress for the runbooks at https://<apex>/runbooks/ — the exact prefix the alert
# annotations already point at (`runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"`, 42 of them
# across platform/monitoring/alerts.yaml, platform/monitoring/pvc-capacity-alerts.yaml,
# platform/cnpg/alerts.yaml and platform/kubevirt/alerts.yaml). Nothing about those annotations
# changes; this route is what finally makes them resolve.
#
# NO `hostnames:` — DELIBERATE, and the reason this manifest needs no Flux variable substitution.
# An HTTPRoute with no hostnames inherits the hostname of every listener it attaches to, and this
# route attaches to exactly one: `https-apex`, whose hostname is `${BASE_DOMAIN}`
# (platform/gateway/gateway.yaml). So the route is correct on dev and on prod with no
# `postBuild.substituteFrom` anywhere in this tier — which matters a great deal here, because the
# SAME Flux Kustomization renders the runbook markdown into a ConfigMap. With substitution enabled,
# Flux's envsubst would run over the runbook prose and silently EMPTY every `${...}` in every shell
# example a runbook contains (it fails open — DECISIONS-LOG 2026-09-01, the Cilium k8sServiceHost
# incident). Declaring no hostname removes the last reason to substitute, so the docs ship verbatim.
# See clusters/base/runbooks.yaml.
#
# PATH PRECEDENCE AGAINST HOMEPAGE. platform/homepage/httproute.yaml already binds `https-apex` with
# PathPrefix `/`. Gateway API resolves a conflict between rules on the same hostname by preferring
# the LONGEST path match, so `/runbooks` wins over `/` for these requests and homepage keeps
# everything else. The two routes coexist by specification, not by luck — and the live 200s recorded
# in the commit prove it on this Gateway implementation (Cilium).
#
# ============================ AUTH: THIS ROUTE IS PUBLIC ==============================
# auth-mode=none, deliberately, and it is the one route in this repo where that is the SAFE choice.
#
# THE CIRCULARITY. Every other exposed UI here declares forward-auth (oauth2-proxy -> Keycloak).
# Put these pages behind that and `runbooks/keycloak-down` becomes unreadable precisely when
# Keycloak is down; `runbooks/keycloak-db` unreadable when its CNPG database is down;
# `runbooks/keycloak-client-secrets` unreadable during the exact failure mode that broke all 12
# OIDC clients on 2026-09-06 while every health signal stayed green. An on-call who cannot log in
# is the primary reader of this route. Gating documentation on the identity provider it documents
# is a loop with no exit.
#
# WHAT IS ACTUALLY EXPOSED. Operational procedure: metric names, thresholds, kubectl invocations,
# component topology. No credentials, no secrets, no tokens, no internal IPs beyond the LAN ranges
# already written in the public spec repo. The content is the same text any reader of the git repo
# can see. The real cost is reconnaissance value — it names what runs here and how it fails — and
# that is accepted knowingly, against the certainty of a locked-out operator staring at a login page
# during an outage. If that trade is ever revisited, the correct fix is NOT forward-auth on this
# route; it is to keep it public and move sensitive specifics out of the runbooks.
#
# HOW forward-auth IS ENFORCED HERE — and why this route is genuinely different. Enforcement is NOT
# a Gateway ext-authz attachment; it is PER-POD. Each forward-auth service runs an oauth2-proxy
# sidecar, its Service publishes BOTH the app port and 4180, and the service's HTTPRoute targets
# :4180 — verified on prod for all five that declare `auth: forward-auth` in the catalog (pihole,
# jackett, qbittorrent, radarr, sonarr). Ingress traffic therefore always traverses oauth2-proxy,
# which performs the Keycloak exchange and the group check. The app port stays on the Service so
# in-cluster clients can talk service-to-service without SSO (that is how sonarr/radarr reach
# qBittorrent's API), which is deliberate, not a leak: Cilium default-deny decides who may use it.
#
# So this route is NOT "no more open than its neighbours" — its neighbours are closed and this one is
# open, on purpose, for the lockout reason argued above. It has no oauth2-proxy sidecar and targets
# its app port directly. If a Gateway-level ext-authz attachment is ever introduced, this route must
# be left OUT of it ON PURPOSE, and this comment is the record of that decision so nobody "fixes" it
# later and re-closes the loop. (Headlamp is also excluded from forward-auth, for a different and
# equally deliberate reason: it runs native Keycloak OIDC and forwards the logged-in user's id_token
# to the apiserver for per-user RBAC, with clusterRoleBinding.create=false so its ServiceAccount can
# do nothing on its own.)
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: runbooks
namespace: runbooks
labels:
app.kubernetes.io/name: runbooks
annotations:
homelab.keithjackson.dev/auth-mode: none
homelab.keithjackson.dev/auth-rationale: >-
Public on purpose. An operator locked out of Keycloak is this route's primary reader;
gating it behind Keycloak would make runbooks/keycloak-down unreadable exactly when it is
needed. Content is operational procedure only — no secrets. Do NOT add this route to the
Gateway ext-authz attachment when forward-auth enforcement lands.
spec:
parentRefs:
- name: shared
namespace: gateway
sectionName: https-apex # the apex listener; the wildcard `https` listener won't match it
rules:
- matches:
- path:
type: PathPrefix
value: /runbooks
backendRefs:
- name: runbooks
port: 80
runbooks/serving/namespace.yaml
+16
View File
@@ -0,0 +1,16 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: runbooks
labels:
# Serve the runbooks at https://<apex>/runbooks/<name> — the APEX host, on a `/runbooks` path
# prefix. The apex has its own `https-apex` listener on the shared Gateway
# (platform/gateway/gateway.yaml); this label opts the namespace in to attach HTTPRoutes to it
# (serving/httproute.yaml). Same opt-in as homepage, which owns the apex `/` prefix.
gateway-access: "shared"
# NOT homelab.keithjackson.dev/app-namespace=true. This is a PLATFORM component (like homepage and
# headlamp), so Kyverno does not generate the app default-deny baseline here and the app-scoped
# auth-mode / egress-class policies do not apply. The default-deny NetworkPolicy and the
# Gateway-only CiliumNetworkPolicy are shipped explicitly in serving/networkpolicy.yaml instead.
app.kubernetes.io/name: runbooks
runbooks/serving/networkpolicy.yaml
+57
View File
@@ -0,0 +1,57 @@
---
# Default-deny baseline (R-NET-01): deny all ingress + egress, allow only DNS. `runbooks` is a
# PLATFORM namespace (no app-namespace label) so Kyverno does not auto-generate this — it is shipped
# explicitly, defence in depth, same shape as the Kyverno-generated app baseline and as
# platform/homepage/networkpolicy.yaml. Cilium enforces the UNION of this NetworkPolicy and the
# CiliumNetworkPolicy below (allow if ANY policy allows).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: runbooks
labels:
homelab.keithjackson.dev/baseline: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress: []
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
# Gateway ingress allow (R-NET-01/02): only the shared Gateway (the Cilium `ingress` entity) may
# reach the runbook server, on container port 8080. The server makes NO outbound calls whatsoever —
# every byte it serves comes from a ConfigMap projected by the kubelet — so the baseline's DNS-only
# egress is already more than it needs and no egress CNP is added.
#
# CONSEQUENCE WORTH STATING: because egress is DNS-only, these pages cannot pull a stylesheet, a
# font or a markdown renderer from anywhere. That is why the content is served as text/plain
# (serving/nginx-config.yaml) rather than as HTML with a CDN dependency, and it is a feature: the
# runbooks render with zero network beyond the request that fetched them.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: runbooks-allow-gateway-ingress
namespace: runbooks
labels:
app.kubernetes.io/name: runbooks
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: runbooks
ingress:
- fromEntities:
- ingress
toPorts:
- ports:
- port: "8080"
protocol: TCP
runbooks/serving/nginx-config.yaml
+81
View File
@@ -0,0 +1,81 @@
---
# nginx server config for the runbook static server. Hand-authored ConfigMap (same plain-manifest style
# as platform/homepage), mounted over /etc/nginx/conf.d/default.conf.
#
# WHY THE URI NEEDS NO REWRITE. The alert annotations are `.../runbooks/<name>` with NO `.md` suffix, and
# this repo must not change them (they are spread across four alerts.yaml files that other work is
# editing). Rather than rewriting the URI, the ConfigMap generated from runbooks/*.md maps each file to an
# EXTENSIONLESS key (`node-down=node-down.md` in ../kustomization.yaml) and is mounted at <root>/runbooks/.
# So `/runbooks/node-down` is a plain static file lookup: no rewrite, no fallback chain, nothing to get
# subtly wrong at 3am.
#
# CONTENT TYPE. An extensionless file has no MIME mapping, so nginx would serve it as
# application/octet-stream and a browser would DOWNLOAD it instead of showing it. The empty `types {}`
# block removes the mapping table entirely inside these locations and `default_type` forces
# text/plain;charset=utf-8 — markdown read as plain text, which is what markdown is for, and which works
# identically in a browser, in `curl`, and in a terminal with no JavaScript, no CDN and no egress. This
# namespace has DNS-only egress, so a client-side markdown renderer fetched from a CDN could not load
# anyway; plain text is the honest choice here, not a shortcut.
apiVersion: v1
kind: ConfigMap
metadata:
name: runbooks-nginx
namespace: runbooks
labels:
app.kubernetes.io/name: runbooks
data:
default.conf: |
# Replaces the image's stock conf.d/default.conf. Port 8080 because the base image is
# nginx-unprivileged (runs as uid 101 and cannot bind 80).
server {
listen 8080;
server_name _;
root /usr/share/nginx/html;
# Never leak the server version on an error page.
server_tokens off;
# Keep the 301 below relative so it works through the Gateway without knowing the public host.
absolute_redirect off;
# Probes. Kept OUT of the access log so a 15s kubelet probe does not bury the one request an
# operator actually made.
location = /healthz {
access_log off;
default_type text/plain;
return 200 "ok\n";
}
# /runbooks and /runbooks/ both land on the index (runbooks/README.md, mounted as the
# extensionless key `README`).
location = /runbooks {
return 301 /runbooks/;
}
location = /runbooks/ {
types { }
default_type "text/plain; charset=utf-8";
try_files /runbooks/README =404;
}
# The runbooks themselves. One static file per alert runbook_url.
location /runbooks/ {
types { }
default_type "text/plain; charset=utf-8";
# A ConfigMap volume is a tree of symlinks into ..data/; nginx must follow them.
disable_symlinks off;
try_files $uri =404;
add_header Cache-Control "public, max-age=300";
# An unknown name gets a 404 with a POINTER, not a blank page: the likeliest cause is an
# alert whose runbook was never written, and the reader needs to know where to look next.
error_page 404 = @no_such_runbook;
}
location @no_such_runbook {
default_type "text/plain; charset=utf-8";
return 404 "No runbook by that name.\n\nIndex of all runbooks: /runbooks/\nSource of truth: runbooks/<name>.md in the home-network-k3s-impl repo.\nIf an alert linked you here, its runbook_url has no matching runbooks/<name>.md, and the\nconformance check alert-runbooks-resolvable should have caught it before merge.\n";
}
# Nothing else is served from this pod. The apex `/` prefix belongs to homepage.
location / {
default_type "text/plain; charset=utf-8";
return 404 "This host serves /runbooks/ only.\n";
}
}
runbooks/serving/service.yaml
+19
View File
@@ -0,0 +1,19 @@
---
# ClusterIP fronting the runbook server. The shared Gateway routes the apex `/runbooks` prefix here
# (serving/httproute.yaml); Service port 80 -> container port 8080.
apiVersion: v1
kind: Service
metadata:
name: runbooks
namespace: runbooks
labels:
app.kubernetes.io/name: runbooks
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: runbooks
ports:
- name: http
port: 80
targetPort: http
protocol: TCP
runbooks/targets-down.md
+138
View File
@@ -0,0 +1,138 @@
# targets-down — More than 10% of Prometheus scrape targets are down
**Alert:** `PrometheusTargetsMissing` (warning, 15m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.monitoring-self`
**Expression:** `(count(up == 0) / count(up)) > 0.10`
## What this means here
A tenth of everything Prometheus is supposed to scrape is unreachable. This is a **blind-spot** alert:
its purpose is to tell you that other alerts may be silent because their data is missing, not because
the thing they watch is healthy.
Live baseline on prod, measured 2026-09-13: **55 targets across 19 jobs** —
`apiserver`, `cdi-prometheus-metrics`, `coredns`, `keycloak-discovery`, `keycloak-service`,
`keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`, `kube-prometheus-stack-grafana`,
`kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`, `kube-state-metrics`, `kubelet`,
`kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`, `node-exporter`, `oauth2-proxy`,
`semaphore-db/semaphore-db`, `velero`, `velero/node-agent`. So 10% is roughly **6 targets** — one
node's worth of exporters, or one CNPG cluster's three pods.
**The far more dangerous failure this alert CANNOT see: a target that was never configured.** `up`
only counts targets Prometheus knows about. A missing ServiceMonitor produces no `up == 0` series and
no alert at all. This cluster has been bitten by that repeatedly, and these are the known live gaps as
of 2026-09-13 — every one of them means the corresponding alerts are **inert**:
| Missing scrape job | Alerts that cannot fire |
|---|---|
| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornVolumeCapacityHigh`, `LonghornNodeStorageHigh` |
| cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` |
| Flux controllers (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
| Hubble / Cilium (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
| etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
Precedent for how costly this is: Velero shipped with three backup alerts and
`metrics.serviceMonitor.enabled: false`, so *"a nightly backup could fail — or silently stop running —
for weeks with nothing to say so."* That ServiceMonitor now exists. The rest of the table does not.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Which targets are down, grouped by job:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
curl -s -G --data-urlencode "query=up == 0" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "\"job\"|\"instance\"|\"namespace\"" '
```
The scrape-error text, which usually names the cause outright:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
curl -s "localhost:19090/api/v1/targets?state=active" | python3 -c "
import json,sys
for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]:
if t[\"health\"]!=\"up\": print(t[\"labels\"].get(\"job\"), t[\"scrapeUrl\"], t[\"lastError\"][:120])"'
```
Which monitors exist at all (the gap this alert cannot see):
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get servicemonitors,podmonitors -A'
```
## Likely causes, most likely first
### 1. A node is down or NotReady
One node takes out its `node-exporter` and `kubelet` targets together — roughly the 10% threshold on
its own. Go to `/runbooks/node-down`; this alert is the echo.
### 2. A workload was rolled, scaled to zero, or crashlooped
Targets follow pods. A CNPG cluster restarting takes three targets with it. Transient; if it persists,
the workload is the problem, not the monitoring.
### 3. A NetworkPolicy blocks the scrape
Default-deny is the baseline everywhere (R-NET-01) and a new component's monitor will be scraped the
moment it exists — `serviceMonitorSelectorNilUsesHelmValues: false` means Prometheus selects every
monitor in the cluster — so a newly added workload can produce a permanently-down target purely
because nothing allows ingress from `monitoring`. The `lastError` will be a timeout. Fix is an
explicit allow in the component's CiliumNetworkPolicy, in git.
### 4. TLS or auth on the scrape endpoint
`lastError` will say so (certificate or 401/403). Note a related trap from this cluster: after the
Cilium upgrade, an in-cluster call to a `*.home` hostname is re-checked against the destination's real
backend identity, so `toEntities: world` no longer covers it — see `/runbooks/denied-flows`.
### 5. The exporter is genuinely broken
CDI once had a broken HTTP ServiceMonitor that caused a false `CDIOperatorDown`, and
`KubeVirtControlPlaneDown` once used a non-existent metric with the wrong job label. Both were fixed.
An exporter that is up but publishing the wrong series is the hardest case — see
`/runbooks/cnpg-exporter` for the canonical example (the CNPG exporter can be `up=1` while its whole
`cnpg_pg_stat_*` family is missing).
## When to escalate or stop
- **Treat this alert as "trust nothing else for now".** Before acting on the *silence* of any other
alert, confirm its series exists (`count(<metric>)`), because the table above shows five whole rule
groups are currently evaluating against nothing.
- Escalate if the down targets are `apiserver` or `kubelet` — that is a control-plane problem, not a
monitoring one.
- Adding a missing ServiceMonitor is a git change (a new `ServiceMonitor` next to the component, or a
chart value). Do not "fix" a blind spot by silencing the alert.
- The Longhorn, cert-manager and Flux monitor gaps are known open work. Note them; do not try to
hand-apply monitors into the cluster — every durable change goes through git.
---
Source of truth: `runbooks/targets-down.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` / commit `2d5171e` (Velero had no ServiceMonitor, so three critical backup
alerts could never fire); commits `6a7e9d1`, `b57b3e9` (the false CDI/KubeVirt alerts).
runbooks/update-rollback.md
+150
View File
@@ -0,0 +1,150 @@
# update-rollback — A HelmRelease failed its health gate and is rolling back
**Alert:** `UpdateHealthGateFailed` (critical, 10m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.automated-updates`
**Expression:** `max by (namespace, name) (gotk_reconcile_condition{kind="HelmRelease", type="Ready", status="False"}) == 1`
## What this means here
helm-controller holds a HelmRelease at Ready=False while a failed upgrade remediates and rolls back to
the last-good revision. This is the **R-UPD-03 safety rail firing** — the bad-update auto-rollback
working — so the alert is a report that the platform protected itself, not necessarily that something
is broken right now.
Read the expression honestly, though: it matches **any** HelmRelease at Ready=False, not only one that
is mid-rollback. The rule's own comment flags this as a `TODO(bring-up)`: tighten it with the `reason`
label (`UpgradeFailed` / `RolledBack`) once confirmed against the deployed helm-controller. So a
HelmRelease that simply cannot install — a bad value, a missing CRD, a crashlooping pod — lights this
`critical` too. Your first job is to tell those two cases apart, because they need opposite responses:
a rollback is self-healing and wants a git revert; a failed *install* is a stall and may be gating an
entire tier (see `/runbooks/flux-reconcile`).
Real precedent that rollback does work here: during the 2026-09-01 Cilium incident the Helm release
history shows `v8 e83bf695 → v9 eebd6c88 (the bad render) → v10 e83bf695 (identical to v8)`, and the
release secret for v9 reads `failed`. Helm's wait genuinely failed and the release was remediated — a
claim that remediation never engaged was **measured wrong**. That is why the cluster self-healed in
about six minutes.
**This alert is currently INERT.** `count(gotk_reconcile_condition)` returned **0 series** on prod on
2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every `gotk_*` rule
evaluates against nothing. Read the HelmRelease CRs directly with the commands below, and see
`/runbooks/targets-down` for the full list of inert rule groups.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Which release, and what happened:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get helmreleases.helm.toolkit.fluxcd.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,REASON:.status.conditions[?\(@.type==\"Ready\"\)].reason,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,LASTSUCCESS:.status.lastAppliedRevision'
```
The Helm revision history — this is what distinguishes "rolled back" from "never installed":
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n <namespace> get secrets -l owner=helm --sort-by=.metadata.creationTimestamp \
-o custom-columns=NAME:.metadata.name,STATUS:.metadata.labels.status,VERSION:.metadata.labels.version'
```
The controller's reasoning:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system logs deploy/helm-controller --tail=120 2>&1 | grep -iE "<release>|rollback|remediat|upgrade failed" | tail -30'
```
What the workload itself is doing:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n <namespace> get pods -o wide; \
kubectl --context k3s-prod -n <namespace> get events --sort-by=.lastTimestamp | tail -20'
```
If the release touches backups or databases, verify them with **fully-qualified names** before deciding
anything is fine — `kubectl get backup -A` prints "No resources found" on this cluster even with six
Velero and 27 CNPG backups present:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get backups.velero.io -A; echo; \
kubectl --context k3s-prod get backups.postgresql.cnpg.io -A; echo; \
kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A'
```
## Likely causes, most likely first
### 1. A chart upgrade shipped an incompatible image or value
The recorded instances: **kube-prometheus-stack 65 → 88 brought Grafana 13, which OOM-looped** and
became "the hidden blocker that stalled the whole kps upgrade and the platform tier" (fixed by raising
Grafana 256 → 512 → 768Mi and Prometheus 2 → 3Gi, commit `b2b1aa5`); and **headlamp-0.40.1** shipped a
`-session-ttl` chart/image desync that crashlooped. Remediation: pin the previous version in git
(`update.pin` is the documented escape hatch, docs/11) and push. The rollback has already restored
service; the git change stops it happening again next interval.
### 2. Not a rollback at all — a first install that cannot succeed
`REASON` will be an install/dependency failure and there will be no earlier `deployed` Helm secret. If
the release is inside the gating `platform` tier, everything downstream is frozen behind it — go to
`/runbooks/flux-reconcile`, which covers the "no matches for kind" / CRD-ordering class.
### 3. A timeout, not a failure
`platform` has `timeout: 5m` while Longhorn's HelmRelease has `timeout: 15m` — **the tier can time out
while a legitimate Longhorn upgrade is still running.** Give it another interval before declaring a
failure, and check whether pods are still progressing.
### 4. The rollback itself is stuck
Helm cannot roll back cleanly if the upgrade already mutated something immutable (a StatefulSet
selector, a PVC). Then Ready=False is permanent until an operator intervenes.
### 5. A post-upgrade health check is failing for an unrelated reason
A node under I/O pressure makes pods slow to become Ready and the upgrade's wait expires. Cross-read
`/runbooks/node-io`.
## When to escalate or stop
- **Get the operator if the release is `cilium` or anything in the gating `platform` tier.** A CNI
rollback affects every node's datapath.
- **Do NOT enable `driftDetection.mode: enabled` on a CNI.** During the 2026-09-01 incident the Helm
storage manifest *was* the bad render, so drift correction would have actively re-asserted the broken
value. This is a recorded conclusion, not a preference.
- **Do not `helm rollback` by hand.** helm-controller owns the release; a manual rollback creates drift
it will fight. Use git.
- **Do not suspend the HelmRelease to silence the alert** — that trades a visible failure for
`/runbooks/flux-suspended` and silent drift.
- If the rolled-back release is CNPG or Keycloak, expect knock-on SSO effects and read
`/runbooks/keycloak-down`; note also that a CNPG update under `primaryUpdateMethod: switchover`
cannot change `imageName` and a postgresql parameter in the same reconcile — the operator rejects it.
---
Source of truth: `runbooks/update-rollback.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` "2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium" (the measured
Helm revision history and the drift-detection conclusion); the Grafana OOM and headlamp desync.
runbooks/vpn-down.md
+187
View File
@@ -0,0 +1,187 @@
# vpn-down — The egress VPN tunnel is down
**Alert:** `VPNTunnelDown` (critical, 5m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.networking`
**Expression:**
```
absent(node_network_receive_bytes_total{device="tun0"})
and on() (
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
or (count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0)
)
```
## What this means here
There is **no in-cluster egress-gateway pod** on this platform (ADR-0015). The tunnel is a **host
systemd service** — `vpn-egress-tunnel`, OpenVPN to PIA — running on one node, and steering is Cilium
eBPF via a `CiliumEgressGatewayPolicy` (`vpn-egress-steering`). OpenVPN creates and destroys `tun0` with
the connection, so the `tun0` byte-counter series exists **if and only if** the tunnel is up. The rule
keys on that presence rather than `node_network_up`, because `tun0` reports `operstate=unknown` and
`node_network_up` is therefore 0 even when the tunnel is fine.
Live state, measured 2026-09-13: **the tunnel is up** (`node_network_receive_bytes_total{device="tun0"}`
= 1 series) and the gateway node is labelled (`kube_node_labels{...egress_gateway="true"}` = 1 series),
so this alert is armed and quiet. The gateway is **`k3s-prod-3`** (192.168.2.12, `eno1`).
**The two facts that decide your response:**
1. **Cilium OSS has no egress-gateway failover.** Gateway-node loss is an outage, not a degradation.
The ADR-0016 supersede drafted on 2026-09-06 is still open. Worse, Cilium OSS picks *"the first node
in lexical ordering"* among nodeSelector matches — so labelling `k3s-prod-1` would **silently steal**
the gateway role and the public egress IP from `k3s-prod-3`. `[vpn_egress_gateway_prod]` is a
separate inventory group holding only `k3s-prod-3` for this reason; run the role as
`-e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3`, with `--list-hosts` first,
every time.
2. **The `absent()` was originally evaluated cluster-wide and became a permanent false alarm.** On a
cluster with no gateway node it could never clear, and from 2026-09-05T21:23Z it fired continuously
into `email-critical` — unbounded noise on the one channel that must stay trustworthy. The two
guards OR'd above are the fix: the alert arms itself the moment a node is labelled, and stays armed
through a kube-state-metrics outage. Before that, the very first version keyed on
`absent(gluetun_vpn_up)`, a metric gluetun's control API never exported at all — a pre-existing
perma-false-alarm. **If this alert is firing, verify the tunnel by hand before believing it.**
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Confirm on the host, not from Prometheus:
```
ssh k3s-prod-3 'systemctl status vpn-egress-tunnel --no-pager | head -20; echo; ip -br addr show tun0; echo; ip link show tun0 | head -2'
```
Which node is the gateway, and is it labelled?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink; echo; \
kubectl --context k3s-prod get ciliumegressgatewaypolicies.cilium.io -o yaml | grep -A6 "nodeSelector\|egressIP\|interface"'
```
**The check that catches the silent failure** (see cause 1 below) — does Cilium's datapath point at the
interface that actually exists?
```
ssh k3s-prod-3 'ip -o link | awk -F: "{print \$1, \$2}" | grep tun0'
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg bpf egress list'
```
If the `Egress Ifindex` in that map is not the live `tun0` ifindex, egress is broken **even though this
alert is quiet**.
Is traffic actually leaving through the tunnel?
```
ssh k3s-prod-3 'curl -s --interface tun0 --max-time 10 https://ipinfo.io/ip; echo; nft list table inet vpn_killswitch 2>/dev/null | grep -E "counter|killswitch_" | head'
```
## Likely causes, most likely first
### 1. The tunnel is UP but Cilium is steering at a dead interface (the silent one)
This is the richest failure on the platform and it does **not** set off this alert.
**Cilium resolves a `CiliumEgressGatewayPolicy`'s `interface: tun0` exactly once**, caches
(egress IP, ifindex), and never re-resolves. On `k3s-dev-3`, `cilium-dbg bpf egress list` showed
`Egress Ifindex 3` — **there is no ifindex 3 on that node**; the live `tun0` was **1780**. Egress was
dead for about 80 minutes with nothing alerting and the conformance ledger still reading green.
`egress-gateway-reconciliation-trigger-interval: 1s` does **not** cover it (netlink link events are not
Kubernetes object events).
Measured behaviour, so you can pick a lever that works:
| action | result |
|---|---|
| annotate the CEGP | map → 1780 in 8s, PIA IP returns |
| `systemctl restart vpn-egress-tunnel` | tun0 1780→1985, **map stays 1780**, still broken |
| the exact `vpn-killswitch` stop/30s/start sequence | still broken 20s+ later |
| set **any** unrelated label on the gateway **node** | recovers within 10s |
The durable fix is in `roles/vpn_egress_gateway`: the OpenVPN up-script publishes a node label
`homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on every connect (via
`systemctl start --no-block`, so it can never block the tunnel), plus a 2-minute convergence timer for
boot. A node label, not a CEGP annotation — a node writing to GitOps state is a drift fight.
**Nothing monitors this coupling yet**; that gap is exactly why dev sat dead for 80 minutes.
### 2. PIA handed two gateways the same tunnel address
Standing rule: **one PIA server per gateway.** The vaulted `.ovpn` pins one server, so when prod came up
green, dev went down: both gateways dialled `158.173.241.112:1198` and PIA gave both the tunnel address
`10.72.192.7`. Each reconnect steals it — a measured ~2-minute mutual-eviction cycle (dev restarting at
22:57:05, 22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC). Fix: the role var
`vpn_egress_gateway_pia_remote` ("`<host> <port>`") rewrites only the `.ovpn` `remote` line; prod is
pinned to **`158.173.241.135`** (pool `158.173.241.{34,55,112,135,239}`). *"A single green probe is not
a stability check."*
### 3. PIA "unreachable" — suspect a stale server list first
The 2026-08-22/23 chain went port collision → UDP/53 → 8080 → "MTU" (Cilium VXLAN pod MTU 1450 is real
but irrelevant to a **14-byte** handshake) → "the network blackholes PIA" (**wrong**) before the actual
cause: **gluetun's stale bundled server list**, dialling retired `46.246.x`/`212.112.x` addresses when
live SE Stockholm is `158.173.241.x`. Lesson recorded verbatim: when a client "can't reach" the
provider, suspect the stale server list **first**.
### 4. The gateway node is down
`absent()` also fires when the gateway node or its node-exporter is gone, so this alert covers
gateway-node-down by design. Go to `/runbooks/node-down`. Remember: **there is no failover** — while
`k3s-prod-3` is down, VPN-classed egress is an outage.
### 5. Nothing is wrong and the alert is lying
Given the two false-alarm post-mortems above, always confirm `ip -br addr show tun0` on the host before
acting.
## What the kill-switch does and does not do
Proven with counters on `k3s-prod-3`, because the intuitive reading is wrong: with
**`bpf.masquerade: true`** (which Egress Gateway requires) SNAT and egress happen in eBPF and
`bpf_redirect()` **skips netfilter POSTROUTING**. Measured: `forward` saddr=pod-CIDR **0**,
`postrouting` oif WAN saddr=pod-CIDR **0**. **What actually fails closed is Cilium's own drop** —
tcpdump during tunnel stop and restart showed **0 packets leaving either device**. The nft table is
kept as defence-in-depth for node-originated tunnel-sourced traffic and was proven both ways:
`ping -I 10.72.192.7` with the ip-rule → 3/3 at 144 ms with `killswitch_accept` advancing; rule removed
→ `killswitch_drop` 0→3 packets / 252 bytes, 100% loss. Every rule carries a `counter`, so:
**if `killswitch_drop` ever counts in normal operation, the datapath has changed.** A reported "leak
window" was **measured as no WAN leak** — a stale egress IP follows the cached ifindex, not a routing
lookup. Note the `.nft` file uses `table` + `flush table`, which clears rules but leaves named-set
elements intact.
## When to escalate or stop
- **Escalate immediately if the gateway node is down.** No failover exists; this is an outage with no
in-cluster remedy.
- **Never label a second node `egress-gateway=true`.** Lexical ordering means `k3s-prod-1` would
silently take the role and the public egress IP.
- **Do not restart the tunnel expecting Cilium to follow** — measured above, it does not. If the map is
stale, touching a node label is the lever that works (operator action).
- **Do not remove or "simplify" the nft kill-switch** because the counters read zero. Zero on the pod
path is the expected, measured result of eBPF masquerade; the table guards a different path.
- A leak concern is a security matter (docs/07) — escalate rather than experimenting.
---
Source of truth: `runbooks/vpn-down.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` "2026-09-12 — VPN egress on PROD (k3s-prod-3), and three things the repo
believed that are not true"; the 2026-08-22/23 gluetun chain; commit `155df8e` (R-NET-06 re-proven on
prod, recovery ≤3s).