jakman85andClaude Opus 5
b7341b329b
Three corrections after rebasing onto main. 1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what co-located primaries actually cost here (blast radius, not an outage, because instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet during a real node outage, and the ~9s-per-database switchover rebalance, one cluster at a time. The 2026-09-07 outage in which all three primaries sat on one node is the reason the alert exists at all. 2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's per-check "UNCONDITIONAL" stages with a declaration: `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to main) and the annotation is declared instead. Assertion (7) of the check was rewritten to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half alone makes the check load-bearing. That matters here because ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all. 3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to prod` made that false, and a runbook that misstates which cluster a command talks to is worse than one that says nothing. The pages now say what is durably true: the default has already changed once, so naming `--context k3s-prod` is what makes a command mean the same thing next month, and the per-invocation banner is there to be read. The check's own failure message was corrected the same way. Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving, longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`, confirming Gateway API longest-path precedence. Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green; prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster). Co-Authored-By: Claude Opus 5 <[email protected]>
90 lines
6.3 KiB
YAML
90 lines
6.3 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — every alert's runbook_url resolves to a runbook that is actually served
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
#
|
|
# Every platform-health alert rule in this repo carries
|
|
# `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations across four files, over 34
|
|
# distinct names. For the whole of phases 4-6 there was NO runbooks/ directory and NO HTTPRoute serving
|
|
# that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404.
|
|
# So an alert reaching a human handed them a dead link at the one moment documentation is read, while the
|
|
# annotation made the problem look solved — the same green-surface-over-absent-mechanism class as the
|
|
# import Job that reported success while storing placeholder secrets, and the guard whose glob silently
|
|
# stopped matching.
|
|
#
|
|
# The link is served now (runbooks/, clusters/base/runbooks.yaml). This test is the guard that keeps it
|
|
# true, and it asserts BOTH halves of the claim because the regression is silent in both directions: an
|
|
# alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its
|
|
# configMapGenerator entry is in git, reviewed, merged — and still 404s in the cluster. "The file exists"
|
|
# and "the URL resolves" are different claims; only the second one helps an operator.
|
|
#
|
|
# GIVEN every `runbook_url` annotation on every PrometheusRule in the repo,
|
|
# WHEN each is parsed out of git and resolved against runbooks/,
|
|
# THEN (1) each URL has EXACTLY the shape the server resolves — https://${BASE_DOMAIN}/runbooks/<name>,
|
|
# the ${BASE_DOMAIN} token (never a literal hostname), no `.md`, no trailing slash, no query;
|
|
# (2) every referenced <name> has a runbooks/<name>.md in git;
|
|
# (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the
|
|
# dev-container kubectl invocation with --context k3s-prod — so a stub cannot satisfy (2) and
|
|
# make the link look fixed while still helping nobody;
|
|
# (4) every runbook is PROJECTED INTO THE SERVER by runbooks/kustomization.yaml's
|
|
# configMapGenerator under the extensionless key the URL asks for;
|
|
# (5) route, listener and server agree with the annotations (PathPrefix == the annotations' prefix,
|
|
# a sectionName that platform/gateway/gateway.yaml actually defines, the namespace opted in to
|
|
# the Gateway, and an nginx location for that same prefix with a text/plain default_type);
|
|
# (6) the Flux tier is wired into clusters/base AND declares NO postBuild.substituteFrom —
|
|
# substitution would run envsubst over the runbook prose and silently empty every `${...}`
|
|
# shell example in it (it fails open: DECISIONS-LOG 2026-09-01, the emptied Cilium
|
|
# k8sServiceHost), and the flux-postbuild-substitution check cannot see it because that check
|
|
# scans files reachable through `resources:`, not configMapGenerator inputs;
|
|
# AND (7) the check is LOAD-BEARING: this Test DECLARES
|
|
# `conformance.home.k3s/scope: whole-repo`, so ci/l0.sh Stage 3b's
|
|
# `--include-scope whole-repo` unions it into EVERY gate invocation rather than only the runs
|
|
# whose touched requirements happen to include R-OBS-04; and ci/gate.sh sets the merge block
|
|
# when the l0 stage fails. A SKIP of a scope-declaring check is UNWIRED, never green.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library only);
|
|
# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test wraps the SAME
|
|
# script.
|
|
#
|
|
# NOT proven (named in check.sh's header): this does not FETCH the URL. That it returns HTTP 200 with
|
|
# real content was proven by an in-cluster probe recorded in the commit message; a live probe needs a
|
|
# cluster, which L0 by definition does not have. What is asserted here is every static precondition for
|
|
# that 200. It also does not judge whether a runbook is correct or current — (3) is a floor, not a
|
|
# review — and it does not prove Alertmanager delivers the annotation to a human (that is the L3
|
|
# `alerting` test). R-OBS-04 is tagged for the narrow slice only: the platform's own health alerts carry
|
|
# runbook links that resolve to real, served procedure rather than to a 404.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: alert-runbooks-resolvable
|
|
labels:
|
|
conformance.home.k3s/R-OBS-04: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-OBS-04"
|
|
conformance.home.k3s/layer: "L0"
|
|
# WHOLE-REPO SCOPE — this is what makes the check unconditional. Stage 3b of ci/l0.sh runs
|
|
# `conformance/run.sh --layer l0 --include-scope whole-repo`, which UNIONS every test carrying this
|
|
# annotation into the selection regardless of which requirements the change touched. That union is
|
|
# load-bearing here: ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" —
|
|
# NOT R-OBS-04 — and maps platform/kubevirt/* and runbooks/* to nothing at all, so the single most
|
|
# likely breaking change (adding an alert with a runbook_url, or adding a runbook without its
|
|
# configMapGenerator entry) attributes a requirement set that would otherwise FILTER THIS CHECK OUT
|
|
# of exactly the change that introduces the bug. Declaring the scope replaces the hand-written
|
|
# per-test ci/l0.sh stage that this repo used to need (and that nobody remembered to add).
|
|
conformance.home.k3s/scope: "whole-repo"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Every runbook_url on every alert rule resolves to a runbook that exists in git, clears a quality
|
|
floor, and is actually projected into the server that answers that path — so an alert can never
|
|
hand an operator a dead link.
|
|
spec:
|
|
steps:
|
|
- name: audit-every-alert-runbook-link
|
|
description: >-
|
|
Parse every runbook_url out of every PrometheusRule and assert the runbook exists, is real, is
|
|
served at that exact path, and that the serving tier is wired and unsubstituted.
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|