#!/usr/bin/env bash # ============================================================================= # L0 static check (offline, no cluster) — alert-runbooks-resolvable # Serves: R-OBS-04 ("The platform alerts on its own health (nodes, storage, GitOps, certs, # VPN)." — spec docs/10-observability-and-alerting.md) # # WHY THIS EXISTS — A DEAD LINK IN AN ALERT IS WORSE THAN NO LINK # -------------------------------------------------------------- # Every platform-health alert rule in this repo carries # runbook_url: "https://${BASE_DOMAIN}/runbooks/" # 42 such annotations across four files, over 34 distinct names. For the whole of phases 4-6 # there was NO runbooks/ directory and NO HTTPRoute serving that path: a live in-cluster probe # of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. So an alert reaching a # human at 3am handed them a link that fails, at the one moment the documentation is read. The # annotation LOOKED like the problem was solved, which is the specific failure this repo keeps # finding (an import Job that reports success while storing placeholder secrets; a guard whose # glob silently stopped matching) — a green-looking surface over an absent mechanism. # # The link is now served (runbooks/, clusters/base/runbooks.yaml). This check is the guard that # keeps it true, because the regression is SILENT IN BOTH DIRECTIONS and nothing else catches it: # * add an alert with a new runbook_url and forget the runbook -> that alert links to a 404, # and no test, lint, render or policy notices; # * write runbooks/.md but forget to add it to the ConfigMap generator -> the file is in # git, reviewable, apparently done, and the URL still 404s in the cluster. "The file exists" # and "the URL resolves" are different claims and only the second one helps an operator. # Both halves are asserted below, which is why this check reaches past the markdown into the # kustomization, the HTTPRoute, the nginx config and the Flux tier. # # GIVEN the repository as authored, # WHEN every `runbook_url` annotation on every PrometheusRule is parsed out of git, # THEN (1) each URL has EXACTLY the shape the server resolves — # https://${BASE_DOMAIN}/runbooks/, `${BASE_DOMAIN}` as the token (never a literal # hostname: a literal is how prod was authored to serve dev's hostnames, see the # ServiceDefinition schema's note), no `.md` suffix, no trailing slash, no query; # (2) every referenced has a runbooks/.md in git; # (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the # dev-container kubectl invocation — so a one-line stub cannot satisfy (2) and make the # link look fixed while still helping nobody (the failure mode this check is most likely # to be defeated by is a placeholder file, not a missing one); # (4) every runbook markdown file is actually PROJECTED INTO THE SERVER by # runbooks/kustomization.yaml's configMapGenerator, under the EXTENSIONLESS key the URL # asks for — the "exists in git but 404s in the cluster" half; # (5) the route and the server agree with the annotations: the HTTPRoute's PathPrefix is the # prefix the URLs use, it attaches to a Gateway listener that actually exists, its # namespace is opted in to that Gateway, and the nginx config answers that same prefix; # (6) the Flux tier that reconciles all of it is wired into clusters/base AND declares NO # postBuild.substituteFrom — substitution here would run envsubst over the runbook prose # and silently empty every `${...}` shell example in it (it fails open: DECISIONS-LOG # 2026-09-01, the emptied Cilium k8sServiceHost); # AND (7) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and ci/gate.sh sets the # merge block when the l0 stage fails. # # WHY (7) IS AN UNCONDITIONAL ci/l0.sh STAGE and not left to the requirement-scoped run: Stage 3b # scopes `conformance/run.sh --layer l0` to the TOUCHED requirements (R-TEST-03), and # ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" — NOT R-OBS-04 — and # maps platform/kubevirt/* and runbooks/* to nothing at all. So the single most likely breaking # change (adding an alert with a runbook_url to platform/monitoring/alerts.yaml, or adding a # runbook and forgetting the generator entry) attributes a requirement set that would FILTER THIS # CHECK OUT. Declaring `conformance.home.k3s/scope: whole-repo` makes ci/l0.sh Stage 3b union it into # EVERY gate invocation instead; a SKIP of a scope-declaring check is UNWIRED, never green. # # WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check): # - It does not fetch the URL. That it returns HTTP 200 with real content was proven by an # in-cluster probe (recorded in the commit message and DECISIONS-LOG); a live probe needs a # cluster, which L0 by definition does not have. What is asserted here is every static # precondition for that 200: the file, its projection, the key name, the route, the prefix. # - It does not judge whether a runbook is CORRECT or current. (3) is a floor, not a review: it # can tell a stub from a runbook, not a good runbook from a stale one. Nothing automatable can. # - It does not prove Alertmanager actually delivers the annotation to a human. That is the L3 # `alerting` test. # - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06) and is not consulted. # ============================================================================= set -euo pipefail DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root PYTHON="${PYTHON:-python3}" # Honest SKIP (never green) if there are no alert rules at all — nothing to audit yet. if ! grep -rl 'kind: PrometheusRule' "${ROOT}/platform" >/dev/null 2>&1; then echo "check SKIPPED: no PrometheusRule found under platform/ — no alert rules to audit" >&2 exit 3 fi cd "${ROOT}" "${PYTHON}" - "${ROOT}" <<'PY' """Static audit: every alert runbook_url resolves to a runbook that is actually served. Pure standard library, no PyYAML — the same constraint and the same style as the neighbouring L0 checks (lib/coverage.py, lib/status.py). Every parse contract is documented at its parser; all of them are line-constrained shapes in hand-authored manifests. """ from __future__ import annotations import re import sys from pathlib import Path ROOT = Path(sys.argv[1]) sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree FAILURES: list[str] = [] ADVISORIES: list[str] = [] CHECKS = 0 def check(cond, msg): global CHECKS CHECKS += 1 if not cond: FAILURES.append(msg) def read(p: Path) -> str: try: return p.read_text() except (OSError, UnicodeDecodeError): return "" # =============================================================== (1) parse the annotations # Parse contract: `runbook_url:` is an annotation on a Prometheus alerting rule, authored one per # line as `runbook_url: ""` (single or double quoted, or bare). Only files that declare # `kind: PrometheusRule` are scanned, so a runbook_url quoted inside this check or inside prose # elsewhere in the repo is not mistaken for an alert annotation. Commented-out lines are skipped: # platform/monitoring/alerts.yaml carries a `# runbook_url points at a docs base` header comment. SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo" RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""") # NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such # as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so # a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs # and, worse, can satisfy an "something declares X" check from a copy rather than real source. # Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43). rule_files = sorted( p for p in ROOT.rglob("*.yaml") if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and "kind: PrometheusRule" in read(p) ) check(len(rule_files) > 0, "no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; " "this check would silently audit nothing") # url -> [ ":", ... ] refs: dict[str, list[str]] = {} for f in rule_files: rel = f.relative_to(ROOT) for lineno, line in enumerate(read(f).splitlines(), 1): if line.lstrip().startswith("#"): continue m = RUNBOOK_URL_RE.match(line) if m: refs.setdefault(m.group(1), []).append(f"{rel}:{lineno}") n_annotations = sum(len(v) for v in refs.values()) check(n_annotations > 0, "no runbook_url annotation parsed out of any PrometheusRule — the parser contract broke; " "this check would pass vacuously while every alert link rots") # ---- URL SHAPE. The server resolves exactly one shape. Anything else is a link that 404s even # though a file with a plausible name exists, so shape is a failure, not a style note. URL_RE = re.compile(r"^https://\$\{BASE_DOMAIN\}/(runbooks)/([a-z0-9][a-z0-9-]*)$") names: dict[str, list[str]] = {} # runbook name -> annotation sites prefixes: set[str] = set() for url, sites in sorted(refs.items()): m = URL_RE.match(url) check(m is not None, f"runbook_url '{url}' ({', '.join(sites)}) is not the shape this platform serves. " f"Required: https://${{BASE_DOMAIN}}/runbooks/ — the ${{BASE_DOMAIN}} TOKEN (never a " f"literal hostname: a literal is how prod got authored to serve dev's public names), a " f"lowercase-hyphen , and NO '.md' suffix, trailing slash or query. The runbook " f"server maps the URL path straight onto an extensionless ConfigMap key, so any other " f"shape is a 404 no matter which files exist") if m: prefixes.add(m.group(1)) names.setdefault(m.group(2), []).extend(sites) check(len(prefixes) <= 1, f"alert annotations use more than one runbook path prefix ({sorted(prefixes)}); the single " f"HTTPRoute + nginx location can only serve one, so the others are dead links") PREFIX = (sorted(prefixes) or ["runbooks"])[0] # =================================================== (2)(3) the runbooks exist and are real RUNBOOK_DIR = ROOT / PREFIX check(RUNBOOK_DIR.is_dir(), f"{PREFIX}/ is not a directory in this repo, so every one of the {n_annotations} " f"runbook_url annotations is a dead link") # The quality floor. (2) alone is satisfiable by `touch runbooks/node-down.md`, which makes the # link resolve and still abandons the operator — a worse outcome than the 404, because the 404 at # least told the truth. These four sections are the operator's path through an incident: what the # alert means HERE, what to type first, what is probably wrong, and when to stop. `dev.sh` + # `--context k3s-prod` are required because kubectl does not work on the workstation at all # (the toolchain lives only in the pinned dev container) and because dev.sh defaults its # current-context to k3s-DEV — a runbook that omits the context teaches the reader to answer # questions about the wrong cluster during an incident on this one. REQUIRED_SECTIONS = ( "## What this means here", "## First commands", "## Likely causes, most likely first", "## When to escalate or stop", ) MIN_BYTES = 1500 for name, sites in sorted(names.items()): md = RUNBOOK_DIR / f"{name}.md" if not md.is_file(): check(False, f"runbook '{name}' does not exist: {md.relative_to(ROOT)} is missing, but " f"{len(sites)} alert annotation(s) link to it ({', '.join(sites[:3])}" f"{', ...' if len(sites) > 3 else ''}). An alert firing now hands the operator a 404. " f"Write {PREFIX}/{name}.md (see {PREFIX}/README.md for the required sections)") continue body = read(md) check(len(body) >= MIN_BYTES, f"{md.relative_to(ROOT)} is {len(body)} bytes, under the {MIN_BYTES}-byte floor — a stub " f"that makes the link resolve while still helping nobody is worse than the 404 it " f"replaced, because it looks fixed") for sec in REQUIRED_SECTIONS: check(sec in body, f"{md.relative_to(ROOT)} has no '{sec}' section — the four operator sections are what " f"make this a runbook rather than a restatement of the alert name") check("dev.sh" in body, f"{md.relative_to(ROOT)} never mentions dev.sh — kubectl does not exist on the " f"workstation; every command in a runbook here must be runnable as " f"`infra/devcontainer/dev.sh run bash -c '...'` and the runbook must say so") check("--context k3s-prod" in body, f"{md.relative_to(ROOT)} never passes --context k3s-prod — dev.sh merges both clusters " f"and picks a current-context PER INVOCATION — a default that has already changed once " f"(k3s-dev until dev was decommissioned on 2026-09-13, k3s-prod now). A context-less command " f"silently answers about whichever cluster the default happens to be, in the middle of an " f"incident") # ADVISORY, not a failure: a runbook nothing links to yet. Writing the runbook BEFORE the alert is # the right order, and the repo's precedent for "authored but unreferenced" is advisory (see the # dead-knob advisory in flux-postbuild-substitution/check.sh). served_extra = {"README"} if RUNBOOK_DIR.is_dir(): for md in sorted(RUNBOOK_DIR.glob("*.md")): stem = md.stem if stem in served_extra or stem in names: continue ADVISORIES.append(f"{md.relative_to(ROOT)} is a runbook no alert links to (written ahead of " f"its alert, or the alert was removed — advisory, not a failure)") # ================================= (4) every runbook is PROJECTED INTO THE SERVER, not just in git # THIS IS THE HALF A FILE-EXISTENCE CHECK WOULD MISS. The pod serves a ConfigMap, not the git tree. # Parse contract: runbooks/kustomization.yaml declares one configMapGenerator whose `files:` entries # are `- =.md`, one per line. The key is EXTENSIONLESS on purpose (the URL carries no # `.md`, and the ConfigMap volume mount turns each key into a file the nginx root resolves directly). KUST = RUNBOOK_DIR / "kustomization.yaml" check(KUST.is_file(), f"{PREFIX}/kustomization.yaml is missing — the Flux tier renders nothing and " f"every runbook 404s no matter what is in git") GEN_FILE_RE = re.compile(r"^\s*-\s*([A-Za-z0-9._-]+)=([A-Za-z0-9._/-]+\.md)\s*(?:#.*)?$") generated: dict[str, str] = {} kust_text = read(KUST) if kust_text: check("configMapGenerator:" in kust_text, f"{PREFIX}/kustomization.yaml has no configMapGenerator — the runbook markdown is not " f"rendered into the ConfigMap the server mounts") for line in kust_text.splitlines(): if line.lstrip().startswith("#"): continue m = GEN_FILE_RE.match(line) if m: generated[m.group(1)] = m.group(2) check(len(generated) > 0, f"{PREFIX}/kustomization.yaml declares no `=.md` generator entries — the " f"parser contract broke, or the markdown is being added with implicit keys (which would " f"carry the '.md' suffix the alert URLs do NOT have, so every link would 404)") for name in sorted(names): check(generated.get(name) == f"{name}.md", f"runbook '{name}' is not projected into the server: {PREFIX}/kustomization.yaml has no " f"configMapGenerator entry `- {name}={name}.md`. The file can be in git, reviewed and " f"merged and the URL still returns 404 — 'the file exists' and 'the URL resolves' are " f"different claims") # And the reverse, so a generator entry cannot point at a file that is not there. for key, fname in sorted(generated.items()): check((RUNBOOK_DIR / fname).is_file(), f"{PREFIX}/kustomization.yaml generates key '{key}' from {PREFIX}/{fname}, which does not " f"exist — `kustomize build` fails, so the WHOLE tier stops reconciling and every runbook " f"goes stale at once") # ============================== (5) the route, the listener and the server agree with the URLs ROUTE = RUNBOOK_DIR / "serving" / "httproute.yaml" NGINX = RUNBOOK_DIR / "serving" / "nginx-config.yaml" NS = RUNBOOK_DIR / "serving" / "namespace.yaml" GATEWAY = ROOT / "platform" / "gateway" / "gateway.yaml" route_text = read(ROUTE) check(bool(route_text), f"{PREFIX}/serving/httproute.yaml is missing — nothing routes the " f"/{PREFIX} prefix to the server") # Parse contract: a single HTTPRoute with one PathPrefix match and one parentRefs entry naming a # listener via sectionName; hand-authored, one key per line. m_path = re.search(r"^\s*type:\s*PathPrefix\s*$\n\s*value:\s*(\S+)\s*$", route_text, re.M) check(m_path is not None and m_path.group(1) == f"/{PREFIX}", f"{PREFIX}/serving/httproute.yaml does not match PathPrefix /{PREFIX} " f"(found {m_path.group(1) if m_path else 'no PathPrefix match'}). The alert annotations point " f"at /{PREFIX}/; if the route's prefix drifts from the annotations the links 404 while " f"both files look individually correct") m_sec = re.search(r"^\s*sectionName:\s*(\S+)", route_text, re.M) check(m_sec is not None, f"{PREFIX}/serving/httproute.yaml declares no parentRefs sectionName. The apex host needs the " f"dedicated apex listener: the wildcard `*.${{BASE_DOMAIN}}` listener does NOT match the bare " f"apex, so a route without sectionName binds nothing and serves nothing") if m_sec: listener = m_sec.group(1) gw = read(GATEWAY) check(re.search(rf"^\s*-\s*name:\s*{re.escape(listener)}\s*$", gw, re.M) is not None, f"{PREFIX}/serving/httproute.yaml attaches to Gateway listener '{listener}', which " f"platform/gateway/gateway.yaml does not define — the route would report NotAllowed and " f"never serve") # An HTTPRoute with no `hostnames:` inherits the listener's hostname, which is how this tier avoids # needing Flux substitution (see (6)). Either form is routable; a LITERAL hostname is not acceptable, # for the same reason the schema forbids it in a ServiceDefinition. m_host = re.search(r"^\s*hostnames:\s*$\n\s*-\s*(\S+)", route_text, re.M) check(m_host is None or "${BASE_DOMAIN}" in m_host.group(1), f"{PREFIX}/serving/httproute.yaml hardcodes hostname '{m_host.group(1) if m_host else ''}' — " f"use the ${{BASE_DOMAIN}} token, or omit `hostnames:` entirely and inherit the listener's") ns_text = read(NS) check('gateway-access: "shared"' in ns_text or "gateway-access: shared" in ns_text, f"{PREFIX}/serving/namespace.yaml does not carry gateway-access=shared — the shared Gateway " f"selects namespaces by that label, so without it the HTTPRoute is refused attachment") nginx_text = read(NGINX) check(f"location /{PREFIX}/" in nginx_text, f"{PREFIX}/serving/nginx-config.yaml has no `location /{PREFIX}/` block — the route would " f"deliver the request and the server would answer 404 for every runbook") check("default_type" in nginx_text and "text/plain" in nginx_text, f"{PREFIX}/serving/nginx-config.yaml does not force a text/plain default_type. The ConfigMap " f"keys are EXTENSIONLESS, so with the stock MIME table nginx serves them as " f"application/octet-stream and a browser DOWNLOADS the runbook instead of showing it") # ====================================== (6) the Flux tier is wired, and is NOT substituted FLUX = ROOT / "clusters" / "base" / f"{PREFIX}.yaml" BASE_KUST = ROOT / "clusters" / "base" / "kustomization.yaml" flux_text = read(FLUX) check(bool(flux_text), f"clusters/base/{PREFIX}.yaml is missing — nothing reconciles {PREFIX}/ into the cluster, so " f"the runbooks exist in git and the URL still 404s") check(re.search(rf"^\s*path:\s*\./{re.escape(PREFIX)}\s*$", flux_text, re.M) is not None, f"clusters/base/{PREFIX}.yaml does not set `path: ./{PREFIX}`") check(re.search(r"^\s*wait:\s*false\s*$", flux_text, re.M) is not None, f"clusters/base/{PREFIX}.yaml is not wait:false — a leaf web server that can be legitimately " f"unhealthy must never gate the reconcile chain (DECISIONS-LOG: 'Headlamp gating stall')") # Strip comments before testing: this file's own comment block EXPLAINS at length why # substituteFrom must never be added here, and a naive substring test would fire on the warning. flux_code = "\n".join(l for l in flux_text.splitlines() if not l.lstrip().startswith("#")) check("substituteFrom" not in flux_code, f"clusters/base/{PREFIX}.yaml declares postBuild.substituteFrom. It MUST NOT: this tier " f"renders documentation into a ConfigMap, and Flux's envsubst would run over the prose and " f"silently EMPTY every `${{...}}` shell example in every runbook — it fails open, so the " f"Kustomization would still report Ready (DECISIONS-LOG 2026-09-01, the emptied Cilium " f"k8sServiceHost). The flux-postbuild-substitution check cannot see this: it scans files " f"reachable through `resources:`, not configMapGenerator inputs") check(re.search(rf"^\s*-\s*{re.escape(PREFIX)}\.yaml\b", read(BASE_KUST), re.M) is not None, f"clusters/base/kustomization.yaml does not list {PREFIX}.yaml — the Flux Kustomization is " f"authored and never applied, which is the 'looks done, is absent' class this repo has been " f"bitten by repeatedly") # ============================================ (7) this check is load-bearing in the merge gate L0 = read(ROOT / "ci" / "l0.sh") GATE = read(ROOT / "ci" / "gate.sh") # THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism, in two halves. # It is not enough that check.sh exists: the merge gate must SELECT it on every run. The repo used to # get that with a hand-written "UNCONDITIONAL" stage per whole-repo check in ci/l0.sh, hardcoding the # path; those stages are gone, because the next whole-repo invariant only got one if somebody # REMEMBERED to write it. The property is now DECLARED by the test and HONOURED by the gate, so both # halves are asserted — neither alone is sufficient. The mechanism itself (that the union really # selects, and that a scope-declared SKIP exits 3 rather than passing green) is proven end-to-end and # falsifiably by conformance/tests/l0-static/whole-repo-invariants-wired. SELF_YAML = (ROOT / "conformance" / "tests" / "l0-static" / "alert-runbooks-resolvable" / "chainsaw-test.yaml") SELF = read(SELF_YAML) check(re.search(SCOPE_DECL_RE, SELF, re.M) is not None, "alert-runbooks-resolvable/chainsaw-test.yaml does not declare " "conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a " "change happens to touch a requirement it is labelled with. ci/touched-requirements.sh maps " "platform/monitoring/* to 'R-OBS-01 R-OBS-02' (NOT R-OBS-04) and maps platform/kubevirt/* and " "runbooks/* to nothing at all — so adding an alert with a new runbook_url, or adding a runbook " "without its configMapGenerator entry, would be filtered out of exactly the change that " "introduces the dead link") check("--include-scope" in L0, "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a " "purely requirement-scoped run again and the scope annotation above is inert decoration — this " "check would then run only by coincidence") check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE), "ci/gate.sh does not set the merge block when the l0 stage fails") # ======================================================================== report print(f"Alert runbook inventory: {n_annotations} runbook_url annotation(s) across " f"{len(rule_files)} PrometheusRule file(s) -> {len(names)} distinct runbook(s), " f"prefix /{PREFIX}") missing = sorted(n for n in names if not (RUNBOOK_DIR / f"{n}.md").is_file()) print(f" runbooks present: {len(names) - len(missing)}/{len(names)} " f"projected into the ConfigMap: {sum(1 for n in names if generated.get(n) == f'{n}.md')}/{len(names)}") if missing: print(f" MISSING: {', '.join(missing)}", file=sys.stderr) if ADVISORIES: print("\nADVISORIES (not failures):") for a in ADVISORIES: print(f" - {a}") print() if not FAILURES: print(f"check OK: alert-runbooks-resolvable — {CHECKS} assertions (R-OBS-04)") raise SystemExit(0) print(f"CHECK FAIL (alert-runbooks-resolvable): {len(FAILURES)} failed, " f"{CHECKS - len(FAILURES)} passed", file=sys.stderr) for f in FAILURES: print(f" FAIL {f}", file=sys.stderr) raise SystemExit(1) PY