jakman85andClaude Opus 5
e9a4b669d4
status.py has always reported PROVEN: 0 for all 51 requirements, and named the fix in its
own header: "a committed, machine-readable run ledger". There was none — _artifacts/ is
gitignored, the root .gitignore kills *.junit.xml, and a stray JUnit was surfaced only as
a quarantined mtime advisory. Built BEFORE the first real L3 execution so that run is
captured rather than reconstructed from memory.
THE TRAP THIS EXISTS TO AVOID. These tests exit 0 and Chainsaw records `passed` on a
SELF-SKIP exactly as on a real run, and Chainsaw's JUnit carries no stdout — only
properties. So committing the JUnit as proof would let DR_RESTORE_SKIPPED_OK be recorded
as proof of R-STATE-03: authoritative-looking and wrong, which is worse than no ledger.
The sentinel token is the ONLY thing distinguishing proven from skipped, and it lives
solely in the run log. The ledger therefore records the LOG alongside the JUnit and
derives proof from the sentinel, never from the JUnit result.
Sentinel tokens are DERIVED, not hand-listed: record_run.py parses every
contains($stdout,'TOKEN') out of each test's own chainsaw-test.yaml, so the list cannot
drift from the tests. 23 of 35 tests declare no sentinel at all.
decide() is the single place `proven` is emitted, ordered WORST-FIRST:
JUnit failure/error -> failed
JUnit <skipped/> -> junit-skipped
ANY declared skip token in the log -> skipped (beats a green JUnit AND beats a pass
token present in the same log)
no sentinel declared -> proven ONLY for L0/L1 (whose JUnit is truthful on its own, since
run_l1_offline.py maps exit 3 to <skipped/>); written as an
explicit exempt-set so an unknown/blank layer fails CLOSED
EVERY declared pass token present -> proven
otherwise -> unknown-sentinel-absent
Log attribution is a state machine over Chainsaw's per-test headers; a log with no
recognisable header falls back to whole-log scanning restricted to GLOBALLY-UNIQUE tokens,
so DNS_OK (owned by two tests) can never prove either. record_run.py exits 3 if the JUnit
or the log is missing — nothing recorded, never green.
STALENESS: a proven row is honoured only while `git merge-base --is-ancestor` holds
between the requirement's last implementing/test change and the row's git_sha; otherwise
it degrades to stale-proof. A -dirty sha, an unknown rev, or an uncommitted test path all
fail ancestry the same way. A requirement reads proven only when EVERY verifying test
does (worst-of), and the latest row per test wins — not best-ever.
REFUSAL POSTURE PRESERVED: with an empty ledger the report is unchanged in intent — every
requirement not-derivable, headline "PROVEN: 0 — NOT DERIVABLE". This replaces "not
derivable" with real evidence only where evidence exists, never with optimism.
--record is explicit, never implicit (a run that silently records itself is how a bad run
becomes permanent false proof), is rejected for --status/--coverage-only/--lint, and
re-execs under tee returning ${PIPESTATUS[0]} so recording can never soften a gate. It
rm -f's a stale _artifacts/report.junit.xml first — a hazard hit live during development,
where a stale JUnit got paired with a fresh log.
Selftests: record_run.py 23 checks (incl. end-to-end Chainsaw-passed + NODE_LOSS_SKIPPED_OK
-> proof "skipped"), status.py +16 against a throwaway git repo (self-skip -> skipped;
proven-at-older-sha -> stale-proof; dirty/unresolvable -> stale-proof; worst-of;
latest-row-wins; malformed lines dropped). No regressions: --coverage-only, --status,
--lint, --strict-coverage and the merge-gate shape all rc=0 and unchanged; nothing in ci/
references status.py.
Root .gitignore needed no change — conformance/runs/ is not itself excluded, so the deeper
negations override the root's global *.junit.xml. Verified with git check-ignore.
Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
30 lines
1.5 KiB
Plaintext
30 lines
1.5 KiB
Plaintext
# Generated artifacts — regenerated by run.sh on every run; never committed.
|
|
# CI collects these from the workspace / results PVC; they are inputs to the gate,
|
|
# not source. (Root .gitignore also covers *.junit.xml and coverage/.)
|
|
_artifacts/
|
|
|
|
# Downloaded pinned tool binaries (run.sh may fetch chainsaw/kubectl here).
|
|
.bin/
|
|
|
|
# Python bytecode from lib/ (status.py imports coverage.py); never committed.
|
|
__pycache__/
|
|
|
|
# --- THE ONE EXCEPTION: the committed run ledger (runs/) -------------------------------
|
|
# `runs/<UTC>-<short-sha>/{report.junit.xml,run.log}` + `runs/ledger.jsonl` are the
|
|
# machine-readable proof-of-pass record that lib/status.py's docstring asks for. They are
|
|
# SOURCE, not regenerated artifacts: a run's evidence cannot be reconstructed after the
|
|
# fact, so it must survive in git.
|
|
#
|
|
# The negations are needed because the ROOT .gitignore ignores `*.junit.xml` globally
|
|
# (and `coverage/`). A deeper .gitignore overrides a shallower one, and nothing ignores
|
|
# the `conformance/runs/` directory itself, so re-including the files here works (git
|
|
# cannot re-include a file whose PARENT DIRECTORY is excluded — that is why this must not
|
|
# live under _artifacts/).
|
|
#
|
|
# The run LOG is the load-bearing half: Chainsaw's JUnit carries no stdout, and the L2/L3
|
|
# tests self-skip GREEN, so the sentinel token in run.log is the only thing separating
|
|
# "proven" from "skipped". Never commit a report.junit.xml without its run.log.
|
|
!runs/
|
|
!runs/**
|
|
!runs/**/*.junit.xml
|