From fdc3265585f4b775c0a6ae3fb33c837b0c643655 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 19:08:38 -0500 Subject: [PATCH] fix(conformance): the ledger counted as evidence FOR the requirements it proves MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Committing the first PROVEN row immediately degraded it to `stale-proof`. Cause: a self-defeating loop. `IMPL_EXCLUDE_PREFIXES` excluded `conformance/tests/` and `ci/tests/` but NOT `conformance/runs/`. The ledger and the preserved run logs CONTAIN the requirement IDs they are evidence FOR — `ledger.jsonl` literally holds `"requirements":["R-STATE-02"]` — so they were scanned as IMPLEMENTING evidence for R-STATE-02. That made `last_change` for the requirement move to the ledger commit itself, which the row's `git_sha` necessarily predates, so the freshness check failed. The consequence was total, not marginal: **recording a proof destroyed it, so NO ledger row could ever have been non-stale.** The ledger would have reported `PROVEN: 0 (stale N)` forever while looking like it was working — the same shape of silent-uselessness as the COMMAND-echo bug in the previous commit, and found the same way: by actually running the thing and disbelieving a green-looking result. Fix: exclude `conformance/runs/`. The ledger is the proof RECORD; it must not participate in the freshness computation about itself. Verified: `PROVEN: 1 (stale 0, self-skipped 0, failed 0, unknown 0) from 1 ledger row(s)`; R-STATE-02 reads **proven**. status.py --selftest PASS. The staleness machinery is otherwise untouched — a genuinely stale row (impl changed after the run) still degrades, which the selftest continues to assert. Serves: R-TEST-01..04. Co-Authored-By: Claude Opus 5 (1M context) --- conformance/lib/status.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/conformance/lib/status.py b/conformance/lib/status.py index b0e2c01..a91f748 100755 --- a/conformance/lib/status.py +++ b/conformance/lib/status.py @@ -108,7 +108,14 @@ FLUX_API = "kustomize.toolkit.fluxcd.io" # conformance/tests — those are the TESTS; they are the VERIFICATION axis, counted there. # ci/tests — gate unit-test fixtures embed example R-IDs as DATA. # registry.yaml — the requirement mirror itself, not an implementation of anything. -IMPL_EXCLUDE_PREFIXES = ("conformance/tests/", "ci/tests/") +# `conformance/runs/` is excluded for a subtler reason than the test dirs: the ledger and the +# preserved run logs CONTAIN the requirement IDs they are evidence FOR. Counting them as +# implementing evidence makes `last_change` move every time proof is recorded, so a row's +# git_sha always predates it and EVERY proof degrades to stale-proof the instant it is +# committed — recording the proof destroys it. The ledger is the proof record; it must not +# participate in the freshness computation about itself. (Found live 2026-09-01 on the first +# PROVEN row.) +IMPL_EXCLUDE_PREFIXES = ("conformance/tests/", "ci/tests/", "conformance/runs/") IMPL_EXCLUDE_PATHS = ("conformance/requirements/registry.yaml",)