feat(conformance): committed run ledger — the only thing that can move PROVEN off zero
status.py has always reported PROVEN: 0 for all 51 requirements, and named the fix in its
own header: "a committed, machine-readable run ledger". There was none — _artifacts/ is
gitignored, the root .gitignore kills *.junit.xml, and a stray JUnit was surfaced only as
a quarantined mtime advisory. Built BEFORE the first real L3 execution so that run is
captured rather than reconstructed from memory.
THE TRAP THIS EXISTS TO AVOID. These tests exit 0 and Chainsaw records `passed` on a
SELF-SKIP exactly as on a real run, and Chainsaw's JUnit carries no stdout — only
properties. So committing the JUnit as proof would let DR_RESTORE_SKIPPED_OK be recorded
as proof of R-STATE-03: authoritative-looking and wrong, which is worse than no ledger.
The sentinel token is the ONLY thing distinguishing proven from skipped, and it lives
solely in the run log. The ledger therefore records the LOG alongside the JUnit and
derives proof from the sentinel, never from the JUnit result.
Sentinel tokens are DERIVED, not hand-listed: record_run.py parses every
contains($stdout,'TOKEN') out of each test's own chainsaw-test.yaml, so the list cannot
drift from the tests. 23 of 35 tests declare no sentinel at all.
decide() is the single place `proven` is emitted, ordered WORST-FIRST:
JUnit failure/error -> failed
JUnit <skipped/> -> junit-skipped
ANY declared skip token in the log -> skipped (beats a green JUnit AND beats a pass
token present in the same log)
no sentinel declared -> proven ONLY for L0/L1 (whose JUnit is truthful on its own, since
run_l1_offline.py maps exit 3 to <skipped/>); written as an
explicit exempt-set so an unknown/blank layer fails CLOSED
EVERY declared pass token present -> proven
otherwise -> unknown-sentinel-absent
Log attribution is a state machine over Chainsaw's per-test headers; a log with no
recognisable header falls back to whole-log scanning restricted to GLOBALLY-UNIQUE tokens,
so DNS_OK (owned by two tests) can never prove either. record_run.py exits 3 if the JUnit
or the log is missing — nothing recorded, never green.
STALENESS: a proven row is honoured only while `git merge-base --is-ancestor` holds
between the requirement's last implementing/test change and the row's git_sha; otherwise
it degrades to stale-proof. A -dirty sha, an unknown rev, or an uncommitted test path all
fail ancestry the same way. A requirement reads proven only when EVERY verifying test
does (worst-of), and the latest row per test wins — not best-ever.
REFUSAL POSTURE PRESERVED: with an empty ledger the report is unchanged in intent — every
requirement not-derivable, headline "PROVEN: 0 — NOT DERIVABLE". This replaces "not
derivable" with real evidence only where evidence exists, never with optimism.
--record is explicit, never implicit (a run that silently records itself is how a bad run
becomes permanent false proof), is rejected for --status/--coverage-only/--lint, and
re-execs under tee returning ${PIPESTATUS[0]} so recording can never soften a gate. It
rm -f's a stale _artifacts/report.junit.xml first — a hazard hit live during development,
where a stale JUnit got paired with a fresh log.
Selftests: record_run.py 23 checks (incl. end-to-end Chainsaw-passed + NODE_LOSS_SKIPPED_OK
-> proof "skipped"), status.py +16 against a throwaway git repo (self-skip -> skipped;
proven-at-older-sha -> stale-proof; dirty/unresolvable -> stale-proof; worst-of;
latest-row-wins; malformed lines dropped). No regressions: --coverage-only, --status,
--lint, --strict-coverage and the merge-gate shape all rc=0 and unchanged; nothing in ci/
references status.py.
Root .gitignore needed no change — conformance/runs/ is not itself excluded, so the deeper
negations override the root's global *.junit.xml. Verified with git check-ignore.
Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
aa0a50a143
commit
e9a4b669d4
6 files changed
+1347
-42
No files matched your search
+93
-3
@@ -14,12 +14,33 @@
|
||||
# ./run.sh --status # DERIVED STATUS report only (no cluster): per
|
||||
# # requirement, is it implemented, is it actually
|
||||
# # reconciled by Flux, is its test executable or a
|
||||
# # scaffold. Proof-of-pass is deliberately NOT
|
||||
# # derived — see lib/status.py.
|
||||
# # scaffold, and whether the committed run ledger
|
||||
# # (runs/ledger.jsonl) actually proves it — see
|
||||
# # lib/status.py. No ledger row => not-derivable.
|
||||
# ./run.sh --lint # structural-validate test defs (no cluster)
|
||||
# ./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
||||
# ./run.sh --requirements R-NET-01 # ALSO asserts those IDs are covered (merge gate)
|
||||
# ./run.sh --require-all # phase-exit: every requirement must be covered
|
||||
# ./run.sh --record # RUN AND CAPTURE: tee the whole run to
|
||||
# # runs/<UTC>-<short-sha>/run.log, copy the JUnit
|
||||
# # beside it, and append one ledger row per
|
||||
# # testcase to runs/ledger.jsonl (lib/record_run.py)
|
||||
#
|
||||
# --record — PROOF-OF-PASS, CAPTURED AT RUN TIME (why it is a separate flag)
|
||||
# --------------------------------------------------------------------------
|
||||
# Without it, nothing about a green run ever reaches repo state: _artifacts/ is gitignored
|
||||
# and the root .gitignore kills *.junit.xml globally, which is why `--status` reports
|
||||
# PROVEN 0 for all 51 requirements. `--record` writes the one tracked thing that fixes
|
||||
# that — `conformance/runs/` (see conformance/.gitignore for the negations).
|
||||
#
|
||||
# RECORDING IS DELIBERATELY EXPLICIT. A run that silently records itself is how a bad run
|
||||
# becomes a permanent false proof, so the plain `./run.sh` writes no ledger row at all.
|
||||
#
|
||||
# BOTH FILES ARE KEPT, and the LOG is the load-bearing one. The L2/L3 tests self-skip
|
||||
# GREEN: when a precondition is missing they echo e.g. `DR_RESTORE_SKIPPED_OK` and exit 0,
|
||||
# and Chainsaw records `passed` exactly as on a real run. Chainsaw's JUnit carries no
|
||||
# stdout, so the sentinel token in run.log is the ONLY thing that separates proven from
|
||||
# skipped. A JUnit committed on its own would record a skip as proof of R-STATE-03.
|
||||
#
|
||||
# Artifacts (always written under conformance/_artifacts/, gitignored):
|
||||
# _artifacts/report.junit.xml JUnit (Chainsaw output, or skipped-scaffold shim)
|
||||
@@ -28,6 +49,11 @@
|
||||
# _artifacts/status/status.md human requirement-status report (--status)
|
||||
# _artifacts/status/status.json machine requirement-status report (--status)
|
||||
#
|
||||
# Run ledger (TRACKED — commit these; --record only):
|
||||
# runs/<UTC>-<short-sha>/run.log full tee'd run output (carries the sentinels)
|
||||
# runs/<UTC>-<short-sha>/report.junit.xml the JUnit for that same run
|
||||
# runs/ledger.jsonl append-only, one row per testcase per run
|
||||
#
|
||||
# Phase 0: no cluster exists. The behavior tests are valid scaffolds that cannot
|
||||
# execute yet, so run.sh emits a *skipped* JUnit and a coverage report, and exits
|
||||
# green — the harness is proven reproducible while later phases fill in behavior.
|
||||
@@ -47,12 +73,17 @@ STATUS_OUT="${ARTIFACTS}/status"
|
||||
JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
# Every argument as given, so `--record` can re-exec this script unchanged (minus itself)
|
||||
# under `tee`. Captured BEFORE parsing so nothing is normalized away.
|
||||
ORIG_ARGS=("$@")
|
||||
|
||||
LAYER="all"
|
||||
REQS=""
|
||||
MODE="run" # run | coverage-only | status | lint
|
||||
STRICT_COVERAGE=0
|
||||
REQUIRE_ALL=0
|
||||
FORCE_NO_CLUSTER=0
|
||||
RECORD=0
|
||||
|
||||
# Documented, out-of-scope coverage gaps exempt from the completeness gate: requirements
|
||||
# owned by another workstream that WS-TEST does not yet scaffold. Listed here so the
|
||||
@@ -83,7 +114,9 @@ FORCE_NO_CLUSTER=0
|
||||
# longer a gap and has been removed from this list; the coverage report counts it as covered.
|
||||
DOCUMENTED_GAPS=""
|
||||
|
||||
usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; }
|
||||
# Print the header block (line 2 through the closing `####` rule, which is dropped) as
|
||||
# help. Bounded by the rule rather than a fixed line number so the header can grow.
|
||||
usage() { sed -n '2,/^#####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; }
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
@@ -95,6 +128,7 @@ while [ $# -gt 0 ]; do
|
||||
--strict-coverage) STRICT_COVERAGE=1; shift ;;
|
||||
--require-all) REQUIRE_ALL=1; shift ;;
|
||||
--no-cluster) FORCE_NO_CLUSTER=1; shift ;;
|
||||
--record) RECORD=1; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) echo "unknown arg: $1" >&2; usage; exit 2 ;;
|
||||
esac
|
||||
@@ -104,6 +138,62 @@ mkdir -p "${ARTIFACTS}" "${COVERAGE_OUT}" "${STATUS_OUT}"
|
||||
|
||||
log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; }
|
||||
|
||||
###############################################################################
|
||||
# --record: capture this run as committed evidence, then hand it to the ledger.
|
||||
#
|
||||
# Implemented as a ONE-LEVEL re-exec of this same script under `tee`, so the run itself is
|
||||
# byte-for-byte the ordinary run (same code path, same artifacts) and the log is the exact
|
||||
# operator-visible output — no second, differently-behaving "recording mode" to drift.
|
||||
# CONF_RECORD_CHILD guards against recursion.
|
||||
#
|
||||
# The recorder runs REGARDLESS of the run's exit status: a red run must be recorded as red
|
||||
# (that is evidence too), and run.sh still exits with the child's real rc, so no gate is
|
||||
# softened by recording. `${PIPESTATUS[0]}` is what carries that rc past `tee`.
|
||||
###############################################################################
|
||||
if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
|
||||
case "${MODE}" in
|
||||
run) ;;
|
||||
# --record is about EVIDENCE OF A RUN. --coverage-only/--status/--lint execute no
|
||||
# test, so a ledger row from one of them would be a row about nothing.
|
||||
*) echo "--record captures a TEST RUN; --${MODE} executes none." >&2; exit 2 ;;
|
||||
esac
|
||||
RUN_SHA="$(git -C "${CONF_ROOT}" rev-parse --short HEAD 2>/dev/null || echo nogit)"
|
||||
RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-${RUN_SHA}"
|
||||
RUN_DIR="${CONF_ROOT}/runs/${RUN_ID}"
|
||||
mkdir -p "${RUN_DIR}"
|
||||
# Drop any JUnit left behind by a PREVIOUS run before starting, so this run dir can only
|
||||
# ever capture a report THIS run produced. Copying a stale artifact would pair someone
|
||||
# else's result with this run's log — a fabricated pairing, which is precisely the class
|
||||
# of false proof the ledger exists to prevent.
|
||||
rm -f "${JUNIT_OUT}"
|
||||
# Strip --record from the argv handed to the child; everything else passes through.
|
||||
CHILD_ARGS=()
|
||||
for a in ${ORIG_ARGS[@]+"${ORIG_ARGS[@]}"}; do
|
||||
[ "$a" = "--record" ] || CHILD_ARGS+=("$a")
|
||||
done
|
||||
log "RECORD: capturing this run to runs/${RUN_ID}/ (run.log + report.junit.xml)"
|
||||
set +e
|
||||
CONF_RECORD_CHILD=1 "${BASH_SOURCE[0]}" ${CHILD_ARGS[@]+"${CHILD_ARGS[@]}"} 2>&1 \
|
||||
| tee "${RUN_DIR}/run.log"
|
||||
RUN_RC=${PIPESTATUS[0]}
|
||||
set -e
|
||||
if [ -f "${JUNIT_OUT}" ]; then
|
||||
cp "${JUNIT_OUT}" "${RUN_DIR}/report.junit.xml"
|
||||
else
|
||||
log "RECORD: no ${JUNIT_OUT} was produced — the ledger step will SKIP (never green)."
|
||||
fi
|
||||
# Explicit, separate step: read the JUnit + the log and append one row per testcase.
|
||||
# rc 3 = inputs absent (nothing recorded, not green); it must not mask the run's own rc.
|
||||
# CONF_LEDGER exists so the harness's own tests can exercise this path without writing
|
||||
# to the real ledger; operators should never set it.
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
|
||||
--run-dir "${RUN_DIR}" --env "${CONF_ENV:-dev}" \
|
||||
--ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
|
||||
log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
|
||||
log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
|
||||
exit "${RUN_RC}"
|
||||
fi
|
||||
|
||||
# --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
|
||||
gen_coverage() {
|
||||
# Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no
|
||||
|
||||
Reference in new issue
Block a user