Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

merge: 35 runbooks, served, with a guard that they resolve
gate / gate (push) Has been cancelled
Details
gate / conformance (push) Has been cancelled
Details

Browse Source 
All 46 runbook_url annotations pointed at https://<apex>/runbooks/<name>
and every one returned 404 — there was no runbooks/ directory and no route.
Every alert that reached a human handed them a dead link at the moment they
needed it.

Runbooks are now served from their own non-gating tier at the apex prefix,
PUBLIC on purpose: gating them behind Keycloak would make keycloak-down
unreadable exactly when Keycloak is down, which is a loop with no exit. The
new L0 guard checks both directions — a runbook_url with no file, and a file
not wired into the generator — and it caught a live one on contact with
main: CNPGPrimariesCoLocated had shipped hours earlier pointing at a runbook
that did not exist.
This commit is contained in:
jakman85 committed 2026-09-13 00:11:03 -05:00
2 parents d616fd4025 bce8778096
commit e49b86eb41
47 files changed
+6768
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

conformance/tests/l0-static/alert-runbooks-resolvable/chainsaw-test.yaml
+89
View File
@@ -0,0 +1,89 @@
# =============================================================================
# L0 STATIC — every alert's runbook_url resolves to a runbook that is actually served
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# Every platform-health alert rule in this repo carries
# `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations across four files, over 34
# distinct names. For the whole of phases 4-6 there was NO runbooks/ directory and NO HTTPRoute serving
# that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404.
# So an alert reaching a human handed them a dead link at the one moment documentation is read, while the
# annotation made the problem look solved — the same green-surface-over-absent-mechanism class as the
# import Job that reported success while storing placeholder secrets, and the guard whose glob silently
# stopped matching.
#
# The link is served now (runbooks/, clusters/base/runbooks.yaml). This test is the guard that keeps it
# true, and it asserts BOTH halves of the claim because the regression is silent in both directions: an
# alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its
# configMapGenerator entry is in git, reviewed, merged — and still 404s in the cluster. "The file exists"
# and "the URL resolves" are different claims; only the second one helps an operator.
#
# GIVEN every `runbook_url` annotation on every PrometheusRule in the repo,
# WHEN each is parsed out of git and resolved against runbooks/,
# THEN (1) each URL has EXACTLY the shape the server resolves — https://${BASE_DOMAIN}/runbooks/<name>,
# the ${BASE_DOMAIN} token (never a literal hostname), no `.md`, no trailing slash, no query;
# (2) every referenced <name> has a runbooks/<name>.md in git;
# (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the
# dev-container kubectl invocation with --context k3s-prod — so a stub cannot satisfy (2) and
# make the link look fixed while still helping nobody;
# (4) every runbook is PROJECTED INTO THE SERVER by runbooks/kustomization.yaml's
# configMapGenerator under the extensionless key the URL asks for;
# (5) route, listener and server agree with the annotations (PathPrefix == the annotations' prefix,
# a sectionName that platform/gateway/gateway.yaml actually defines, the namespace opted in to
# the Gateway, and an nginx location for that same prefix with a text/plain default_type);
# (6) the Flux tier is wired into clusters/base AND declares NO postBuild.substituteFrom —
# substitution would run envsubst over the runbook prose and silently empty every `${...}`
# shell example in it (it fails open: DECISIONS-LOG 2026-09-01, the emptied Cilium
# k8sServiceHost), and the flux-postbuild-substitution check cannot see it because that check
# scans files reachable through `resources:`, not configMapGenerator inputs;
# AND (7) the check is LOAD-BEARING: this Test DECLARES
# `conformance.home.k3s/scope: whole-repo`, so ci/l0.sh Stage 3b's
# `--include-scope whole-repo` unions it into EVERY gate invocation rather than only the runs
# whose touched requirements happen to include R-OBS-04; and ci/gate.sh sets the merge block
# when the l0 stage fails. A SKIP of a scope-declaring check is UNWIRED, never green.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library only);
# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test wraps the SAME
# script.
#
# NOT proven (named in check.sh's header): this does not FETCH the URL. That it returns HTTP 200 with
# real content was proven by an in-cluster probe recorded in the commit message; a live probe needs a
# cluster, which L0 by definition does not have. What is asserted here is every static precondition for
# that 200. It also does not judge whether a runbook is correct or current — (3) is a floor, not a
# review — and it does not prove Alertmanager delivers the annotation to a human (that is the L3
# `alerting` test). R-OBS-04 is tagged for the narrow slice only: the platform's own health alerts carry
# runbook links that resolve to real, served procedure rather than to a 404.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: alert-runbooks-resolvable
labels:
conformance.home.k3s/R-OBS-04: "true"
annotations:
conformance.home.k3s/requirements: "R-OBS-04"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE — this is what makes the check unconditional. Stage 3b of ci/l0.sh runs
# `conformance/run.sh --layer l0 --include-scope whole-repo`, which UNIONS every test carrying this
# annotation into the selection regardless of which requirements the change touched. That union is
# load-bearing here: ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" —
# NOT R-OBS-04 — and maps platform/kubevirt/* and runbooks/* to nothing at all, so the single most
# likely breaking change (adding an alert with a runbook_url, or adding a runbook without its
# configMapGenerator entry) attributes a requirement set that would otherwise FILTER THIS CHECK OUT
# of exactly the change that introduces the bug. Declaring the scope replaces the hand-written
# per-test ci/l0.sh stage that this repo used to need (and that nobody remembered to add).
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every runbook_url on every alert rule resolves to a runbook that exists in git, clears a quality
floor, and is actually projected into the server that answers that path — so an alert can never
hand an operator a dead link.
spec:
steps:
- name: audit-every-alert-runbook-link
description: >-
Parse every runbook_url out of every PrometheusRule and assert the runbook exists, is real, is
served at that exact path, and that the serving tier is wired and unsubstituted.
try:
- script:
content: bash ./check.sh
conformance/tests/l0-static/alert-runbooks-resolvable/check.sh
Executable
+410
View File
@@ -0,0 +1,410 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — alert-runbooks-resolvable
# Serves: R-OBS-04 ("The platform alerts on its own health (nodes, storage, GitOps, certs,
# VPN)." — spec docs/10-observability-and-alerting.md)
#
# WHY THIS EXISTS — A DEAD LINK IN AN ALERT IS WORSE THAN NO LINK
# --------------------------------------------------------------
# Every platform-health alert rule in this repo carries
# runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
# 42 such annotations across four files, over 34 distinct names. For the whole of phases 4-6
# there was NO runbooks/ directory and NO HTTPRoute serving that path: a live in-cluster probe
# of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. So an alert reaching a
# human at 3am handed them a link that fails, at the one moment the documentation is read. The
# annotation LOOKED like the problem was solved, which is the specific failure this repo keeps
# finding (an import Job that reports success while storing placeholder secrets; a guard whose
# glob silently stopped matching) — a green-looking surface over an absent mechanism.
#
# The link is now served (runbooks/, clusters/base/runbooks.yaml). This check is the guard that
# keeps it true, because the regression is SILENT IN BOTH DIRECTIONS and nothing else catches it:
# * add an alert with a new runbook_url and forget the runbook -> that alert links to a 404,
# and no test, lint, render or policy notices;
# * write runbooks/<name>.md but forget to add it to the ConfigMap generator -> the file is in
# git, reviewable, apparently done, and the URL still 404s in the cluster. "The file exists"
# and "the URL resolves" are different claims and only the second one helps an operator.
# Both halves are asserted below, which is why this check reaches past the markdown into the
# kustomization, the HTTPRoute, the nginx config and the Flux tier.
#
# GIVEN the repository as authored,
# WHEN every `runbook_url` annotation on every PrometheusRule is parsed out of git,
# THEN (1) each URL has EXACTLY the shape the server resolves —
# https://${BASE_DOMAIN}/runbooks/<name>, `${BASE_DOMAIN}` as the token (never a literal
# hostname: a literal is how prod was authored to serve dev's hostnames, see the
# ServiceDefinition schema's note), no `.md` suffix, no trailing slash, no query;
# (2) every referenced <name> has a runbooks/<name>.md in git;
# (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the
# dev-container kubectl invocation — so a one-line stub cannot satisfy (2) and make the
# link look fixed while still helping nobody (the failure mode this check is most likely
# to be defeated by is a placeholder file, not a missing one);
# (4) every runbook markdown file is actually PROJECTED INTO THE SERVER by
# runbooks/kustomization.yaml's configMapGenerator, under the EXTENSIONLESS key the URL
# asks for — the "exists in git but 404s in the cluster" half;
# (5) the route and the server agree with the annotations: the HTTPRoute's PathPrefix is the
# prefix the URLs use, it attaches to a Gateway listener that actually exists, its
# namespace is opted in to that Gateway, and the nginx config answers that same prefix;
# (6) the Flux tier that reconciles all of it is wired into clusters/base AND declares NO
# postBuild.substituteFrom — substitution here would run envsubst over the runbook prose
# and silently empty every `${...}` shell example in it (it fails open: DECISIONS-LOG
# 2026-09-01, the emptied Cilium k8sServiceHost);
# AND (7) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and ci/gate.sh sets the
# merge block when the l0 stage fails.
#
# WHY (7) IS AN UNCONDITIONAL ci/l0.sh STAGE and not left to the requirement-scoped run: Stage 3b
# scopes `conformance/run.sh --layer l0` to the TOUCHED requirements (R-TEST-03), and
# ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" — NOT R-OBS-04 — and
# maps platform/kubevirt/* and runbooks/* to nothing at all. So the single most likely breaking
# change (adding an alert with a runbook_url to platform/monitoring/alerts.yaml, or adding a
# runbook and forgetting the generator entry) attributes a requirement set that would FILTER THIS
# CHECK OUT. Declaring `conformance.home.k3s/scope: whole-repo` makes ci/l0.sh Stage 3b union it into
# EVERY gate invocation instead; a SKIP of a scope-declaring check is UNWIRED, never green.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
# - It does not fetch the URL. That it returns HTTP 200 with real content was proven by an
# in-cluster probe (recorded in the commit message and DECISIONS-LOG); a live probe needs a
# cluster, which L0 by definition does not have. What is asserted here is every static
# precondition for that 200: the file, its projection, the key name, the route, the prefix.
# - It does not judge whether a runbook is CORRECT or current. (3) is a floor, not a review: it
# can tell a stub from a runbook, not a good runbook from a stale one. Nothing automatable can.
# - It does not prove Alertmanager actually delivers the annotation to a human. That is the L3
# `alerting` test.
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06) and is not consulted.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
# Honest SKIP (never green) if there are no alert rules at all — nothing to audit yet.
if ! grep -rl 'kind: PrometheusRule' "${ROOT}/platform" >/dev/null 2>&1; then
echo "check SKIPPED: no PrometheusRule found under platform/ — no alert rules to audit" >&2
exit 3
fi
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
"""Static audit: every alert runbook_url resolves to a runbook that is actually served.
Pure standard library, no PyYAML — the same constraint and the same style as the neighbouring
L0 checks (lib/coverage.py, lib/status.py). Every parse contract is documented at its parser;
all of them are line-constrained shapes in hand-authored manifests.
"""
from __future__ import annotations
import re
import sys
from pathlib import Path
ROOT = Path(sys.argv[1])
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
FAILURES: list[str] = []
ADVISORIES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def read(p: Path) -> str:
try:
return p.read_text()
except (OSError, UnicodeDecodeError):
return ""
# =============================================================== (1) parse the annotations
# Parse contract: `runbook_url:` is an annotation on a Prometheus alerting rule, authored one per
# line as `runbook_url: "<url>"` (single or double quoted, or bare). Only files that declare
# `kind: PrometheusRule` are scanned, so a runbook_url quoted inside this check or inside prose
# elsewhere in the repo is not mistaken for an alert annotation. Commented-out lines are skipped:
# platform/monitoring/alerts.yaml carries a `# runbook_url points at a docs base` header comment.
SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo"
RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""")
rule_files = sorted(
p for p in ROOT.rglob("*.yaml")
if ".git" not in p.parts and "kind: PrometheusRule" in read(p)
)
check(len(rule_files) > 0,
"no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
"this check would silently audit nothing")
# url -> [ "<relpath>:<lineno>", ... ]
refs: dict[str, list[str]] = {}
for f in rule_files:
rel = f.relative_to(ROOT)
for lineno, line in enumerate(read(f).splitlines(), 1):
if line.lstrip().startswith("#"):
continue
m = RUNBOOK_URL_RE.match(line)
if m:
refs.setdefault(m.group(1), []).append(f"{rel}:{lineno}")
n_annotations = sum(len(v) for v in refs.values())
check(n_annotations > 0,
"no runbook_url annotation parsed out of any PrometheusRule — the parser contract broke; "
"this check would pass vacuously while every alert link rots")
# ---- URL SHAPE. The server resolves exactly one shape. Anything else is a link that 404s even
# though a file with a plausible name exists, so shape is a failure, not a style note.
URL_RE = re.compile(r"^https://\$\{BASE_DOMAIN\}/(runbooks)/([a-z0-9][a-z0-9-]*)$")
names: dict[str, list[str]] = {} # runbook name -> annotation sites
prefixes: set[str] = set()
for url, sites in sorted(refs.items()):
m = URL_RE.match(url)
check(m is not None,
f"runbook_url '{url}' ({', '.join(sites)}) is not the shape this platform serves. "
f"Required: https://${{BASE_DOMAIN}}/runbooks/<name> — the ${{BASE_DOMAIN}} TOKEN (never a "
f"literal hostname: a literal is how prod got authored to serve dev's public names), a "
f"lowercase-hyphen <name>, and NO '.md' suffix, trailing slash or query. The runbook "
f"server maps the URL path straight onto an extensionless ConfigMap key, so any other "
f"shape is a 404 no matter which files exist")
if m:
prefixes.add(m.group(1))
names.setdefault(m.group(2), []).extend(sites)
check(len(prefixes) <= 1,
f"alert annotations use more than one runbook path prefix ({sorted(prefixes)}); the single "
f"HTTPRoute + nginx location can only serve one, so the others are dead links")
PREFIX = (sorted(prefixes) or ["runbooks"])[0]
# =================================================== (2)(3) the runbooks exist and are real
RUNBOOK_DIR = ROOT / PREFIX
check(RUNBOOK_DIR.is_dir(),
f"{PREFIX}/ is not a directory in this repo, so every one of the {n_annotations} "
f"runbook_url annotations is a dead link")
# The quality floor. (2) alone is satisfiable by `touch runbooks/node-down.md`, which makes the
# link resolve and still abandons the operator — a worse outcome than the 404, because the 404 at
# least told the truth. These four sections are the operator's path through an incident: what the
# alert means HERE, what to type first, what is probably wrong, and when to stop. `dev.sh` +
# `--context k3s-prod` are required because kubectl does not work on the workstation at all
# (the toolchain lives only in the pinned dev container) and because dev.sh defaults its
# current-context to k3s-DEV — a runbook that omits the context teaches the reader to answer
# questions about the wrong cluster during an incident on this one.
REQUIRED_SECTIONS = (
"## What this means here",
"## First commands",
"## Likely causes, most likely first",
"## When to escalate or stop",
)
MIN_BYTES = 1500
for name, sites in sorted(names.items()):
md = RUNBOOK_DIR / f"{name}.md"
if not md.is_file():
check(False,
f"runbook '{name}' does not exist: {md.relative_to(ROOT)} is missing, but "
f"{len(sites)} alert annotation(s) link to it ({', '.join(sites[:3])}"
f"{', ...' if len(sites) > 3 else ''}). An alert firing now hands the operator a 404. "
f"Write {PREFIX}/{name}.md (see {PREFIX}/README.md for the required sections)")
continue
body = read(md)
check(len(body) >= MIN_BYTES,
f"{md.relative_to(ROOT)} is {len(body)} bytes, under the {MIN_BYTES}-byte floor — a stub "
f"that makes the link resolve while still helping nobody is worse than the 404 it "
f"replaced, because it looks fixed")
for sec in REQUIRED_SECTIONS:
check(sec in body,
f"{md.relative_to(ROOT)} has no '{sec}' section — the four operator sections are what "
f"make this a runbook rather than a restatement of the alert name")
check("dev.sh" in body,
f"{md.relative_to(ROOT)} never mentions dev.sh — kubectl does not exist on the "
f"workstation; every command in a runbook here must be runnable as "
f"`infra/devcontainer/dev.sh run bash -c '...'` and the runbook must say so")
check("--context k3s-prod" in body,
f"{md.relative_to(ROOT)} never passes --context k3s-prod — dev.sh merges both clusters "
f"and picks a current-context PER INVOCATION — a default that has already changed once "
f"(k3s-dev until dev was decommissioned on 2026-09-13, k3s-prod now). A context-less command "
f"silently answers about whichever cluster the default happens to be, in the middle of an "
f"incident")
# ADVISORY, not a failure: a runbook nothing links to yet. Writing the runbook BEFORE the alert is
# the right order, and the repo's precedent for "authored but unreferenced" is advisory (see the
# dead-knob advisory in flux-postbuild-substitution/check.sh).
served_extra = {"README"}
if RUNBOOK_DIR.is_dir():
for md in sorted(RUNBOOK_DIR.glob("*.md")):
stem = md.stem
if stem in served_extra or stem in names:
continue
ADVISORIES.append(f"{md.relative_to(ROOT)} is a runbook no alert links to (written ahead of "
f"its alert, or the alert was removed — advisory, not a failure)")
# ================================= (4) every runbook is PROJECTED INTO THE SERVER, not just in git
# THIS IS THE HALF A FILE-EXISTENCE CHECK WOULD MISS. The pod serves a ConfigMap, not the git tree.
# Parse contract: runbooks/kustomization.yaml declares one configMapGenerator whose `files:` entries
# are `- <key>=<file>.md`, one per line. The key is EXTENSIONLESS on purpose (the URL carries no
# `.md`, and the ConfigMap volume mount turns each key into a file the nginx root resolves directly).
KUST = RUNBOOK_DIR / "kustomization.yaml"
check(KUST.is_file(), f"{PREFIX}/kustomization.yaml is missing — the Flux tier renders nothing and "
f"every runbook 404s no matter what is in git")
GEN_FILE_RE = re.compile(r"^\s*-\s*([A-Za-z0-9._-]+)=([A-Za-z0-9._/-]+\.md)\s*(?:#.*)?$")
generated: dict[str, str] = {}
kust_text = read(KUST)
if kust_text:
check("configMapGenerator:" in kust_text,
f"{PREFIX}/kustomization.yaml has no configMapGenerator — the runbook markdown is not "
f"rendered into the ConfigMap the server mounts")
for line in kust_text.splitlines():
if line.lstrip().startswith("#"):
continue
m = GEN_FILE_RE.match(line)
if m:
generated[m.group(1)] = m.group(2)
check(len(generated) > 0,
f"{PREFIX}/kustomization.yaml declares no `<key>=<file>.md` generator entries — the "
f"parser contract broke, or the markdown is being added with implicit keys (which would "
f"carry the '.md' suffix the alert URLs do NOT have, so every link would 404)")
for name in sorted(names):
check(generated.get(name) == f"{name}.md",
f"runbook '{name}' is not projected into the server: {PREFIX}/kustomization.yaml has no "
f"configMapGenerator entry `- {name}={name}.md`. The file can be in git, reviewed and "
f"merged and the URL still returns 404 — 'the file exists' and 'the URL resolves' are "
f"different claims")
# And the reverse, so a generator entry cannot point at a file that is not there.
for key, fname in sorted(generated.items()):
check((RUNBOOK_DIR / fname).is_file(),
f"{PREFIX}/kustomization.yaml generates key '{key}' from {PREFIX}/{fname}, which does not "
f"exist — `kustomize build` fails, so the WHOLE tier stops reconciling and every runbook "
f"goes stale at once")
# ============================== (5) the route, the listener and the server agree with the URLs
ROUTE = RUNBOOK_DIR / "serving" / "httproute.yaml"
NGINX = RUNBOOK_DIR / "serving" / "nginx-config.yaml"
NS = RUNBOOK_DIR / "serving" / "namespace.yaml"
GATEWAY = ROOT / "platform" / "gateway" / "gateway.yaml"
route_text = read(ROUTE)
check(bool(route_text), f"{PREFIX}/serving/httproute.yaml is missing — nothing routes the "
f"/{PREFIX} prefix to the server")
# Parse contract: a single HTTPRoute with one PathPrefix match and one parentRefs entry naming a
# listener via sectionName; hand-authored, one key per line.
m_path = re.search(r"^\s*type:\s*PathPrefix\s*$\n\s*value:\s*(\S+)\s*$", route_text, re.M)
check(m_path is not None and m_path.group(1) == f"/{PREFIX}",
f"{PREFIX}/serving/httproute.yaml does not match PathPrefix /{PREFIX} "
f"(found {m_path.group(1) if m_path else 'no PathPrefix match'}). The alert annotations point "
f"at /{PREFIX}/<name>; if the route's prefix drifts from the annotations the links 404 while "
f"both files look individually correct")
m_sec = re.search(r"^\s*sectionName:\s*(\S+)", route_text, re.M)
check(m_sec is not None,
f"{PREFIX}/serving/httproute.yaml declares no parentRefs sectionName. The apex host needs the "
f"dedicated apex listener: the wildcard `*.${{BASE_DOMAIN}}` listener does NOT match the bare "
f"apex, so a route without sectionName binds nothing and serves nothing")
if m_sec:
listener = m_sec.group(1)
gw = read(GATEWAY)
check(re.search(rf"^\s*-\s*name:\s*{re.escape(listener)}\s*$", gw, re.M) is not None,
f"{PREFIX}/serving/httproute.yaml attaches to Gateway listener '{listener}', which "
f"platform/gateway/gateway.yaml does not define — the route would report NotAllowed and "
f"never serve")
# An HTTPRoute with no `hostnames:` inherits the listener's hostname, which is how this tier avoids
# needing Flux substitution (see (6)). Either form is routable; a LITERAL hostname is not acceptable,
# for the same reason the schema forbids it in a ServiceDefinition.
m_host = re.search(r"^\s*hostnames:\s*$\n\s*-\s*(\S+)", route_text, re.M)
check(m_host is None or "${BASE_DOMAIN}" in m_host.group(1),
f"{PREFIX}/serving/httproute.yaml hardcodes hostname '{m_host.group(1) if m_host else ''}' — "
f"use the ${{BASE_DOMAIN}} token, or omit `hostnames:` entirely and inherit the listener's")
ns_text = read(NS)
check('gateway-access: "shared"' in ns_text or "gateway-access: shared" in ns_text,
f"{PREFIX}/serving/namespace.yaml does not carry gateway-access=shared — the shared Gateway "
f"selects namespaces by that label, so without it the HTTPRoute is refused attachment")
nginx_text = read(NGINX)
check(f"location /{PREFIX}/" in nginx_text,
f"{PREFIX}/serving/nginx-config.yaml has no `location /{PREFIX}/` block — the route would "
f"deliver the request and the server would answer 404 for every runbook")
check("default_type" in nginx_text and "text/plain" in nginx_text,
f"{PREFIX}/serving/nginx-config.yaml does not force a text/plain default_type. The ConfigMap "
f"keys are EXTENSIONLESS, so with the stock MIME table nginx serves them as "
f"application/octet-stream and a browser DOWNLOADS the runbook instead of showing it")
# ====================================== (6) the Flux tier is wired, and is NOT substituted
FLUX = ROOT / "clusters" / "base" / f"{PREFIX}.yaml"
BASE_KUST = ROOT / "clusters" / "base" / "kustomization.yaml"
flux_text = read(FLUX)
check(bool(flux_text),
f"clusters/base/{PREFIX}.yaml is missing — nothing reconciles {PREFIX}/ into the cluster, so "
f"the runbooks exist in git and the URL still 404s")
check(re.search(rf"^\s*path:\s*\./{re.escape(PREFIX)}\s*$", flux_text, re.M) is not None,
f"clusters/base/{PREFIX}.yaml does not set `path: ./{PREFIX}`")
check(re.search(r"^\s*wait:\s*false\s*$", flux_text, re.M) is not None,
f"clusters/base/{PREFIX}.yaml is not wait:false — a leaf web server that can be legitimately "
f"unhealthy must never gate the reconcile chain (DECISIONS-LOG: 'Headlamp gating stall')")
# Strip comments before testing: this file's own comment block EXPLAINS at length why
# substituteFrom must never be added here, and a naive substring test would fire on the warning.
flux_code = "\n".join(l for l in flux_text.splitlines() if not l.lstrip().startswith("#"))
check("substituteFrom" not in flux_code,
f"clusters/base/{PREFIX}.yaml declares postBuild.substituteFrom. It MUST NOT: this tier "
f"renders documentation into a ConfigMap, and Flux's envsubst would run over the prose and "
f"silently EMPTY every `${{...}}` shell example in every runbook — it fails open, so the "
f"Kustomization would still report Ready (DECISIONS-LOG 2026-09-01, the emptied Cilium "
f"k8sServiceHost). The flux-postbuild-substitution check cannot see this: it scans files "
f"reachable through `resources:`, not configMapGenerator inputs")
check(re.search(rf"^\s*-\s*{re.escape(PREFIX)}\.yaml\b", read(BASE_KUST), re.M) is not None,
f"clusters/base/kustomization.yaml does not list {PREFIX}.yaml — the Flux Kustomization is "
f"authored and never applied, which is the 'looks done, is absent' class this repo has been "
f"bitten by repeatedly")
# ============================================ (7) this check is load-bearing in the merge gate
L0 = read(ROOT / "ci" / "l0.sh")
GATE = read(ROOT / "ci" / "gate.sh")
# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism, in two halves.
# It is not enough that check.sh exists: the merge gate must SELECT it on every run. The repo used to
# get that with a hand-written "UNCONDITIONAL" stage per whole-repo check in ci/l0.sh, hardcoding the
# path; those stages are gone, because the next whole-repo invariant only got one if somebody
# REMEMBERED to write it. The property is now DECLARED by the test and HONOURED by the gate, so both
# halves are asserted — neither alone is sufficient. The mechanism itself (that the union really
# selects, and that a scope-declared SKIP exits 3 rather than passing green) is proven end-to-end and
# falsifiably by conformance/tests/l0-static/whole-repo-invariants-wired.
SELF_YAML = (ROOT / "conformance" / "tests" / "l0-static" / "alert-runbooks-resolvable"
/ "chainsaw-test.yaml")
SELF = read(SELF_YAML)
check(re.search(SCOPE_DECL_RE, SELF, re.M) is not None,
"alert-runbooks-resolvable/chainsaw-test.yaml does not declare "
"conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
"change happens to touch a requirement it is labelled with. ci/touched-requirements.sh maps "
"platform/monitoring/* to 'R-OBS-01 R-OBS-02' (NOT R-OBS-04) and maps platform/kubevirt/* and "
"runbooks/* to nothing at all — so adding an alert with a new runbook_url, or adding a runbook "
"without its configMapGenerator entry, would be filtered out of exactly the change that "
"introduces the dead link")
check("--include-scope" in L0,
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
"purely requirement-scoped run again and the scope annotation above is inert decoration — this "
"check would then run only by coincidence")
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
"ci/gate.sh does not set the merge block when the l0 stage fails")
# ======================================================================== report
print(f"Alert runbook inventory: {n_annotations} runbook_url annotation(s) across "
f"{len(rule_files)} PrometheusRule file(s) -> {len(names)} distinct runbook(s), "
f"prefix /{PREFIX}")
missing = sorted(n for n in names if not (RUNBOOK_DIR / f"{n}.md").is_file())
print(f" runbooks present: {len(names) - len(missing)}/{len(names)} "
f"projected into the ConfigMap: {sum(1 for n in names if generated.get(n) == f'{n}.md')}/{len(names)}")
if missing:
print(f" MISSING: {', '.join(missing)}", file=sys.stderr)
if ADVISORIES:
print("\nADVISORIES (not failures):")
for a in ADVISORIES:
print(f" - {a}")
print()
if not FAILURES:
print(f"check OK: alert-runbooks-resolvable — {CHECKS} assertions (R-OBS-04)")
raise SystemExit(0)
print(f"CHECK FAIL (alert-runbooks-resolvable): {len(FAILURES)} failed, "
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
for f in FAILURES:
print(f" FAIL {f}", file=sys.stderr)
raise SystemExit(1)
PY