jakman85andClaude Opus 5
cf7d97e546
Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`,
and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO
rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}`
series existed and the rule could not fire. Verified on prod before this change:
`count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}`
and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it
either — a target that was never discovered has no `up` series to be 0.
THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett,
qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics
without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled;
Keycloak does, on the operator's management port, already scraped by
platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so
its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them
had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships
default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a
correct port would have been dropped by policy — while renderer-contract.md claimed the
baseline carried such an allow.
FIXED IN THE GENERATOR, not in twelve rendered files:
* `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the
service's own workload object —
`kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`,
`kube_statefulset_status_replicas_ready` for a StatefulSet, and
`absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom
KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no
exporter and no new target. The `or absent(...)` half is load-bearing: the signal
DISAPPEARING now fires the alert instead of silencing it.
* schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A
ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working
scrape together: the named Service port, the monitor endpoint, and the
monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve
dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus
kube-state-metrics liveness.
* schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses
it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape
would have made its `absent()` half fire FOREVER — the other failure mode this repo has
shipped.
* renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI
must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port
must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED.
PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by
platform/monitoring/tests/run.sh over the RENDERED manifests):
* quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT;
the VM asserted both gone and in a non-running phase; per-app isolation asserted, and
keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment,
keycloak-db StatefulSet ready in the same namespace).
* mutation-checked: six mutations, each turning its suite red — reverting plex to
`up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment;
capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace;
inverting plex's `== 0` to `!= 0`. Control run green.
* live on prod with `promtool query instant`: all ten deployed app Deployments export
`kube_deployment_status_replicas_available`,
`kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2,
`kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve
SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The
twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed
— its rule is not applied either.
DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed:
kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady /
KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a
generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares.
platform/monitoring/alerts.yaml carries no generic app-readiness rule at all.
GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant
conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the
alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR
across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only
thing that made the broken monitors visible to the sibling check, which is exactly why this
axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no
ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements,
vm-demo's monitor matching no Service at all, and the three generator/contract assertions —
and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair
accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix
there are zero in-repo monitors left for it to judge and a guard whose only evidence is
"nothing failed" cannot be told apart from one that stopped working. It also asserts
r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve
services the same wrong one.
Also fixed, found on the way:
* platform/monitoring/tests/run.sh discovered the SUITE convention only inside
platform/monitoring/tests/ while documenting it as general — a suite added anywhere else
was never run and still looked like coverage. Now discovered tree-wide; a misplaced
*.test.yaml fails loudly instead of vanishing.
* KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve
apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the
gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails.
* renderer-contract.md claimed the default-deny baseline carried a standing monitoring
scrape allow. It never did; the claim is corrected and the allow is now real, per declared
metrics endpoint and scoped to that one port.
Co-Authored-By: Claude Opus 5 <[email protected]>
Conformance tests — layout & requirement-tagging convention
Every test is a Chainsaw Test in its own directory:
tests/
l0-static/ L0 — whole-repo invariants over the tracked tree (no cluster)
l1-rendering/ L1 — ServiceDefinition -> rendered resources match golden
l2-behavioral/ L2 — real runtime behavior (probes against a cluster)
l3-fault-injection/ L3 — break something; assert fail-closed / no-data-loss
meta/ harness self-checks (R-TEST-*), cluster-independent
<test-name>/chainsaw-test.yaml
The layer a test belongs to is its subtree and its .../layer annotation; the
coverage generator cross-checks the two.
L0 static (l0-static/) — what belongs here, and what does not
docs/14's L0 row is "Pre-apply, in CI · manifests/ServiceDefinitions obey invariants".
Kyverno (platform/kyverno/) covers the invariants expressible over a single
resource. This subtree covers the L0 invariants that are properties of the repository
as a whole and that no per-resource policy can state — e.g. no plaintext secret exists
anywhere in the tracked tree, no tracked RoleBinding grants a workload identity a
privileged role, the spec-immutability gate actually blocks a modified spec.
Each l0-static/<name>/ dir carries a check.sh: a self-contained offline assertion
script (git + Python stdlib/PyYAML; no cluster, no network) that exits 0/1. It is executed:
- by
conformance/run.sh --layer l0(and as the offline leg of a full run), vialib/run_l1_offline.py, producing a REAL pass/fail JUnit entry — never a skip; and - by the merge gate's
ci/l0.shstage, so these assertions run on every PR; and - by the wrapping Chainsaw
Test, which runs the SAMEcheck.sh, so an in-cluster run executes byte-identical assertions.
Two authoring rules specific to this layer:
- State the gap. A statically-provable half of a requirement is still only half.
Every
check.shheader ends with an explicit "what this does NOT prove" list naming the cluster-layer remainder. Covering a requirement at L0 does not finish it. - Exemptions must be self-policing. Where a check must tolerate something (a
deliberately-plaintext policy fixture, an
.exampletemplate), the exemption is conditioned on a property that would break if the exemption were abused — e.g. the plaintext Secret fixture is exempt only while akyverno-test.yamlasserts the policy rejects it. Never a bare path allowlist.
Requirement-tagging convention (authoritative)
A test declares the requirement(s) it verifies three ways; the label is the source of truth, the annotations must agree with it (the coverage generator fails on drift):
metadata:
name: default-deny
labels:
# ONE label per verified R-* ID. Machine-selectable + coverage source of truth.
conformance.home.k3s/R-NET-01: "true"
annotations:
# Human-readable canonical list — MUST equal the label set.
conformance.home.k3s/requirements: "R-NET-01"
# Declared layer (L0|L1|L2|L3|META) — must match the subtree.
conformance.home.k3s/layer: "L2"
# true => R-TEST-05 requires this to be (or have) an L3 fault-injection test.
conformance.home.k3s/safety-critical: "false"
# Given/When/Then intent, for the coverage report and reviewers.
conformance.home.k3s/intent: "Given ... When ... Then ..."
Why a label per requirement (not a comma list): Kubernetes label values can't hold
a comma-separated list, and chainsaw test --selector selects on label keys/values.
One conformance.home.k3s/<R-ID>=true label per requirement makes both subset-selection
and coverage extraction exact and robust.
How this drives the tooling
- Subset runs (merge gate):
run.sh --requirements R-NET-01,R-NET-06resolves the R-IDs to the set of test dirs carrying those labels (union, deduped) and runs each once. - Coverage (
lib/coverage.py): reads these labels from everychainsaw-test.yaml, maps each registry requirement -> its verifying test(s), and flags gaps + any label/annotation mismatch + tags not in the registry. - JUnit: results (real or skipped-scaffold) carry the R-IDs in
classname+ properties, so the JUnit itself stays traceable (R-TEST-04).
Authoring rules (R-TEST-01/02/06)
-
Behavior, not existence. Assert what the system does (a probe cannot connect), never that an object merely exists (a policy can exist and not enforce).
-
Given/When/Then structure with
given-*/when-*/then-*step names. -
Hermetic & self-cleaning. Rely on Chainsaw's per-test namespace + auto-teardown; put any out-of-namespace fixture cleanup in a
finallyblock. A leaked fixture is a failure. -
Bounded, deterministic waits. Use assert/error polling within the config timeouts; never
sleep-and-hope. -
Safety-critical => fault-injection. For a safety-critical requirement, actually break the thing and assert fail-closed / no-data-loss, with
finallyrestoring state. -
Never weaken a test to go green (R-TEST-06). If the spec can't be met, escalate to the PM — do not relax the assertion or the requirement.
-
Never write a per-environment value into a test (R-ENV-02). No hostname suffix, no Gateway VIP, no LB-pool range. One authored suite must run green against dev and prod, so a test asks the cluster it is pointed at what it is. The mechanism is a chainsaw binding over the same ConfigMap Flux substitutes from:
spec: bindings: - name: baseDomain value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)Declare it at test level (step
outputs/bindingsare step-scoped), then consume it:($baseDomain)for a podenv:value,(join('.', ['myhost', $baseDomain]))where a full hostname is needed — including inside afixtures/*.yamlapplied withapply: file:, which chainsaw templates with the same bindings. Prefer reading the actual observed value over restating a range (e.g. assert the DNS answer equalsGATEWAY_VIP, or equals the LB IP the Service was assigned) — it is both stricter and environment-independent. Never take the environment from a CLI flag or an exported variable: a flag someone forgets is how a suite ends up silently asserting the other cluster's expectations. L1 goldens are the one exception: they are rendering fixtures and carry the literal${BASE_DOMAIN}token.
How a scaffold declares itself (and how status.py tells one from a verifier)
Scaffolds in this tree carry TODO(T-x.y) markers where a fixture/assertion lands as a
later roadmap phase delivers the capability under test. lib/status.py classifies each
test EXECUTABLE or SCAFFOLD from the test's own files — chainsaw-test.yaml,
render-check.sh, check.sh — and either signal below outranks
conformance.home.k3s/executable: "true", so a placeholder body can never be reported as
an executable test. Declare a scaffold in both positions:
# STATUS: scaffold — <what is missing>in the file's header comment block. The declared state must be the first token afterSTATUS:(executable tests say# STATUS: EXECUTABLE/IMPLEMENTED/RUNNABLE).- A
TODO/FIXMEmarker carrying a message, in the body — the file from its first non-comment line onward.echo "TODO(T-4.2): inject a fault to fire a rule", or an inline comment on a missing fixture:file: fixtures/smtp-sink.yaml # TODO(T-4.1): hermetic SMTP capture sink.
What is not a scaffold marker, and why the distinction is load-bearing: a marker
mentioned in the header prose (the Given/When/Then and the mandatory "what this does
NOT prove" list), and a marker that is the entire content of a quoted literal — a
placeholder detector's own vocabulary, e.g. SEED = ["TBD", "TODO", "FIXME", ...] in
l0-static/placeholder-guard-coupling/check.sh. A whole-file "TODO" in text test used to
classify that 32-assertion check as a scaffold and, through it, report the safety-critical
R-STATE-03 as exec=1 scaffold=1. If your check must hold a marker as data, keep each
idiom a standalone quoted literal in a list; a marker with text attached (a regex
alternation, several idioms in one string) still classifies as a scaffold, because
ambiguity here only ever downgrades.