# ============================================================================= # L0 STATIC — every alert's runbook_url resolves to a runbook that is actually served # Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. # # Every platform-health alert rule in this repo carries # `runbook_url: "https://${BASE_DOMAIN}/runbooks/"` — 42 annotations across four files, over 34 # distinct names. For the whole of phases 4-6 there was NO runbooks/ directory and NO HTTPRoute serving # that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. # So an alert reaching a human handed them a dead link at the one moment documentation is read, while the # annotation made the problem look solved — the same green-surface-over-absent-mechanism class as the # import Job that reported success while storing placeholder secrets, and the guard whose glob silently # stopped matching. # # The link is served now (runbooks/, clusters/base/runbooks.yaml). This test is the guard that keeps it # true, and it asserts BOTH halves of the claim because the regression is silent in both directions: an # alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its # configMapGenerator entry is in git, reviewed, merged — and still 404s in the cluster. "The file exists" # and "the URL resolves" are different claims; only the second one helps an operator. # # GIVEN every `runbook_url` annotation on every PrometheusRule in the repo, # WHEN each is parsed out of git and resolved against runbooks/, # THEN (1) each URL has EXACTLY the shape the server resolves — https://${BASE_DOMAIN}/runbooks/, # the ${BASE_DOMAIN} token (never a literal hostname), no `.md`, no trailing slash, no query; # (2) every referenced has a runbooks/.md in git; # (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the # dev-container kubectl invocation with --context k3s-prod — so a stub cannot satisfy (2) and # make the link look fixed while still helping nobody; # (4) every runbook is PROJECTED INTO THE SERVER by runbooks/kustomization.yaml's # configMapGenerator under the extensionless key the URL asks for; # (5) route, listener and server agree with the annotations (PathPrefix == the annotations' prefix, # a sectionName that platform/gateway/gateway.yaml actually defines, the namespace opted in to # the Gateway, and an nginx location for that same prefix with a text/plain default_type); # (6) the Flux tier is wired into clusters/base AND declares NO postBuild.substituteFrom — # substitution would run envsubst over the runbook prose and silently empty every `${...}` # shell example in it (it fails open: DECISIONS-LOG 2026-09-01, the emptied Cilium # k8sServiceHost), and the flux-postbuild-substitution check cannot see it because that check # scans files reachable through `resources:`, not configMapGenerator inputs; # AND (7) the check is LOAD-BEARING: this Test DECLARES # `conformance.home.k3s/scope: whole-repo`, so ci/l0.sh Stage 3b's # `--include-scope whole-repo` unions it into EVERY gate invocation rather than only the runs # whose touched requirements happen to include R-OBS-04; and ci/gate.sh sets the merge block # when the l0 stage fails. A SKIP of a scope-declaring check is UNWIRED, never green. # # STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library only); # conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test wraps the SAME # script. # # NOT proven (named in check.sh's header): this does not FETCH the URL. That it returns HTTP 200 with # real content was proven by an in-cluster probe recorded in the commit message; a live probe needs a # cluster, which L0 by definition does not have. What is asserted here is every static precondition for # that 200. It also does not judge whether a runbook is correct or current — (3) is a floor, not a # review — and it does not prove Alertmanager delivers the annotation to a human (that is the L3 # `alerting` test). R-OBS-04 is tagged for the narrow slice only: the platform's own health alerts carry # runbook links that resolve to real, served procedure rather than to a 404. # ============================================================================= apiVersion: chainsaw.kyverno.io/v1alpha1 kind: Test metadata: name: alert-runbooks-resolvable labels: conformance.home.k3s/R-OBS-04: "true" annotations: conformance.home.k3s/requirements: "R-OBS-04" conformance.home.k3s/layer: "L0" # WHOLE-REPO SCOPE — this is what makes the check unconditional. Stage 3b of ci/l0.sh runs # `conformance/run.sh --layer l0 --include-scope whole-repo`, which UNIONS every test carrying this # annotation into the selection regardless of which requirements the change touched. That union is # load-bearing here: ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" — # NOT R-OBS-04 — and maps platform/kubevirt/* and runbooks/* to nothing at all, so the single most # likely breaking change (adding an alert with a runbook_url, or adding a runbook without its # configMapGenerator entry) attributes a requirement set that would otherwise FILTER THIS CHECK OUT # of exactly the change that introduces the bug. Declaring the scope replaces the hand-written # per-test ci/l0.sh stage that this repo used to need (and that nobody remembered to add). conformance.home.k3s/scope: "whole-repo" conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- Every runbook_url on every alert rule resolves to a runbook that exists in git, clears a quality floor, and is actually projected into the server that answers that path — so an alert can never hand an operator a dead link. spec: steps: - name: audit-every-alert-runbook-link description: >- Parse every runbook_url out of every PrometheusRule and assert the runbook exists, is real, is served at that exact path, and that the serving tier is wired and unsubstituted. try: - script: content: bash ./check.sh