fix(runbooks): declare whole-repo scope, drop the bespoke l0 stage, +cnpg-primary-spread
Three corrections after rebasing onto main. 1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what co-located primaries actually cost here (blast radius, not an outage, because instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet during a real node outage, and the ~9s-per-database switchover rebalance, one cluster at a time. The 2026-09-07 outage in which all three primaries sat on one node is the reason the alert exists at all. 2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's per-check "UNCONDITIONAL" stages with a declaration: `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to main) and the annotation is declared instead. Assertion (7) of the check was rewritten to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half alone makes the check load-bearing. That matters here because ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all. 3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to prod` made that false, and a runbook that misstates which cluster a command talks to is worse than one that says nothing. The pages now say what is durably true: the default has already changed once, so naming `--context k3s-prod` is what makes a command mean the same thing next month, and the per-invocation banner is there to be read. The check's own failure message was corrected the same way. Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving, longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`, confirming Gateway API longest-path precedence. Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green; prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster). Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
1 parent
44983095e1
commit
b7341b329b
40 files changed
+514
-191
No files matched your search
@@ -136,44 +136,6 @@ else
|
||||
emit_offline_summary
|
||||
fi
|
||||
|
||||
# ---- Stage 3f: every alert runbook_url resolves to a served runbook (UNCONDITIONAL) ----
|
||||
# Every platform-health alert rule carries `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` —
|
||||
# 42 annotations over 34 names. For three roadmap phases there was no runbooks/ directory and no
|
||||
# HTTPRoute serving that path, so a live in-cluster probe of .../runbooks/node-down returned HTTP 404:
|
||||
# every alert that reached a human handed them a dead link at the one moment the documentation is read,
|
||||
# while the annotation made it look solved. The check asserts the whole chain — the URL's shape, the
|
||||
# runbook's existence, a quality floor that a stub cannot clear, the configMapGenerator entry that
|
||||
# actually projects it into the server, and the route/listener/nginx prefix agreeing with the
|
||||
# annotations — because "the file exists in git" and "the URL resolves in the cluster" are different
|
||||
# claims and only the second one helps an operator.
|
||||
#
|
||||
# WHY ITS OWN UNCONDITIONAL STAGE, same reasoning as 3c/3d/3e: Stage 3b scopes the conformance run to
|
||||
# the TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh maps platform/monitoring/* to
|
||||
# "R-OBS-01 R-OBS-02" — NOT R-OBS-04 — and maps platform/kubevirt/* and runbooks/* to nothing at all.
|
||||
# So the single most likely breaking change (adding an alert with a runbook_url, or adding a runbook
|
||||
# without its generator entry) attributes a requirement set that would FILTER THIS CHECK OUT of exactly
|
||||
# the change that introduces the bug. The invariant spans platform/, runbooks/ and clusters/, so it runs
|
||||
# on EVERY gate invocation.
|
||||
RUNBOOK_CHECK="${RUNBOOK_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/alert-runbooks-resolvable/check.sh}"
|
||||
log "Stage: alert runbook links resolve (${RUNBOOK_CHECK})"
|
||||
if [ ! -f "$RUNBOOK_CHECK" ]; then
|
||||
skip "alert runbook check not present at '${RUNBOOK_CHECK}' — UNWIRED, not green"
|
||||
summary "- alert-runbooks: **SKIPPED** (check missing)"
|
||||
if [ "$rc" -eq 0 ]; then rc=3; fi
|
||||
else
|
||||
rb_rc=0
|
||||
bash "$RUNBOOK_CHECK" || rb_rc=$?
|
||||
case "$rb_rc" in
|
||||
0) ok "Alert runbook links resolve"
|
||||
summary "- alert-runbooks: **passed**" ;;
|
||||
3) skip "alert runbook check SKIPPED (no PrometheusRule present) — not a pass"
|
||||
summary "- alert-runbooks: **SKIPPED** (nothing to audit)"
|
||||
if [ "$rc" -eq 0 ]; then rc=3; fi ;;
|
||||
*) err "Alert runbook links FAILED — an alert links to a runbook that does not exist or is not served"
|
||||
summary "- alert-runbooks: **FAILED**"; rc=1 ;;
|
||||
esac
|
||||
fi
|
||||
|
||||
case "$rc" in
|
||||
0) ok "L0 stage complete" ;;
|
||||
3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;
|
||||
|
||||
Reference in new issue
Block a user