Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

feat(conformance): committed run ledger — the only thing that can move PROVEN off zero

Browse Source 
status.py has always reported PROVEN: 0 for all 51 requirements, and named the fix in its
own header: "a committed, machine-readable run ledger". There was none — _artifacts/ is
gitignored, the root .gitignore kills *.junit.xml, and a stray JUnit was surfaced only as
a quarantined mtime advisory. Built BEFORE the first real L3 execution so that run is
captured rather than reconstructed from memory.

THE TRAP THIS EXISTS TO AVOID. These tests exit 0 and Chainsaw records `passed` on a
SELF-SKIP exactly as on a real run, and Chainsaw's JUnit carries no stdout — only
properties. So committing the JUnit as proof would let DR_RESTORE_SKIPPED_OK be recorded
as proof of R-STATE-03: authoritative-looking and wrong, which is worse than no ledger.
The sentinel token is the ONLY thing distinguishing proven from skipped, and it lives
solely in the run log. The ledger therefore records the LOG alongside the JUnit and
derives proof from the sentinel, never from the JUnit result.

Sentinel tokens are DERIVED, not hand-listed: record_run.py parses every
contains($stdout,'TOKEN') out of each test's own chainsaw-test.yaml, so the list cannot
drift from the tests. 23 of 35 tests declare no sentinel at all.

decide() is the single place `proven` is emitted, ordered WORST-FIRST:
  JUnit failure/error -> failed
  JUnit <skipped/>    -> junit-skipped
  ANY declared skip token in the log -> skipped   (beats a green JUnit AND beats a pass
                                                   token present in the same log)
  no sentinel declared -> proven ONLY for L0/L1 (whose JUnit is truthful on its own, since
                          run_l1_offline.py maps exit 3 to <skipped/>); written as an
                          explicit exempt-set so an unknown/blank layer fails CLOSED
  EVERY declared pass token present -> proven
  otherwise -> unknown-sentinel-absent
Log attribution is a state machine over Chainsaw's per-test headers; a log with no
recognisable header falls back to whole-log scanning restricted to GLOBALLY-UNIQUE tokens,
so DNS_OK (owned by two tests) can never prove either. record_run.py exits 3 if the JUnit
or the log is missing — nothing recorded, never green.

STALENESS: a proven row is honoured only while `git merge-base --is-ancestor` holds
between the requirement's last implementing/test change and the row's git_sha; otherwise
it degrades to stale-proof. A -dirty sha, an unknown rev, or an uncommitted test path all
fail ancestry the same way. A requirement reads proven only when EVERY verifying test
does (worst-of), and the latest row per test wins — not best-ever.

REFUSAL POSTURE PRESERVED: with an empty ledger the report is unchanged in intent — every
requirement not-derivable, headline "PROVEN: 0 — NOT DERIVABLE". This replaces "not
derivable" with real evidence only where evidence exists, never with optimism.

--record is explicit, never implicit (a run that silently records itself is how a bad run
becomes permanent false proof), is rejected for --status/--coverage-only/--lint, and
re-execs under tee returning ${PIPESTATUS[0]} so recording can never soften a gate. It
rm -f's a stale _artifacts/report.junit.xml first — a hazard hit live during development,
where a stale JUnit got paired with a fresh log.

Selftests: record_run.py 23 checks (incl. end-to-end Chainsaw-passed + NODE_LOSS_SKIPPED_OK
-> proof "skipped"), status.py +16 against a throwaway git repo (self-skip -> skipped;
proven-at-older-sha -> stale-proof; dirty/unresolvable -> stale-proof; worst-of;
latest-row-wins; malformed lines dropped). No regressions: --coverage-only, --status,
--lint, --strict-coverage and the merge-gate shape all rc=0 and unchanged; nothing in ci/
references status.py.

Root .gitignore needed no change — conformance/runs/ is not itself excluded, so the deeper
negations override the root's global *.junit.xml. Verified with git check-ignore.

Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-01 18:49:01 -05:00
1 parent aa0a50a143
commit e9a4b669d4
6 files changed
+1347 -42
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

conformance/lib/status.py
+447 -39
View File
@@ -15,26 +15,42 @@ This generator derives, from the REPOSITORY ALONE, three orthogonal axes per req
from a Flux Kustomization entrypoint (i.e. actually reconciled)?
2. VERIFICATION — is there a requirement-tagged test, at which layer, and is it an
EXECUTABLE test or a TODO scaffold?
3. PROOF — has that test actually passed? *** DELIBERATELY NOT DERIVED. ***
3. PROOF — has that test actually passed? Derived ONLY from the committed
run ledger, and ONLY as far as that ledger's evidence reaches.
WHAT THIS TOOL REFUSES TO DERIVE (and why)
------------------------------------------
PROOF-OF-PASS IS NOT DERIVABLE FROM REPO STATE. Nothing tracked in git records a test
result: `conformance/_artifacts/` and `results/` are git-ignored, and the only prose
record of a green run lives in DECISIONS-LOG.md — hand-maintained, unstructured, and
exactly the kind of source the operator has ruled out. The project's own culture is that
"merged" is never "done" (agents/definition-of-done.md, R-TEST-03), so this report will
NOT infer, guess, or default a requirement to green. Every requirement's proof state is
reported as `not-derivable`, and the headline says so. If an untracked JUnit artifact
happens to be lying in the workspace it is surfaced ONLY as a quarantined advisory with
its mtime — never counted, never mixed into a status.
PROOF — WHAT CHANGED, AND WHAT DID NOT
--------------------------------------
This report used to say `not-derivable` for every requirement, because nothing tracked in
git recorded a test result and the only prose record of a green run lived in
DECISIONS-LOG.md — hand-maintained, unstructured, exactly the source the operator ruled
out. The fix this docstring asked for now exists: `conformance/runs/ledger.jsonl`, an
append-only ledger written by `lib/record_run.py` from a run's JUnit **and its run log**
(see conformance/.gitignore for the negations that keep both in git).
The honest fix is a committed, machine-readable run ledger; until that exists, the most
this report can say is "wired and testable", never "proven".
THE REFUSAL POSTURE IS UNCHANGED. Evidence replaced "not derivable" only where evidence
exists; nowhere is optimism substituted for it:
* A ledger row reaches `proven` only if `record_run.py` saw that test's own PASS
sentinel token in the run log. The L2/L3 tests SELF-SKIP GREEN — Chainsaw records
`passed` on `DR_RESTORE_SKIPPED_OK` exactly as on a real run, and its JUnit carries no
stdout — so a skip token maps to `skipped` and can never become proof.
* A proof is honoured only while it still describes the current code: if any of the
requirement's implementing manifests or verifying tests changed in a commit that is
NOT an ancestor of the run's `git_sha`, the row degrades to `stale-proof`, which is
not green. A run recorded from a dirty tree (`<sha>-dirty`) is never honoured.
* A requirement with no ledger row, or whose verifying tests did not ALL report proven,
stays `not-derivable` / `unknown` — never rounded up.
* The untracked `_artifacts/report.junit.xml` is STILL only a quarantined advisory with
its mtime. It is not the ledger, it is not evidence, and it feeds no status.
With an empty or absent ledger this report behaves exactly as it did before: every
requirement `not-derivable`, headline PROVEN 0, "the most this report can say is wired and
testable, never proven".
READS (all read-only)
requirements/registry.yaml the authoritative R-* mirror (via lib/coverage.py)
tests/**/chainsaw-test.yaml requirement tags + layer + executable annotation
runs/ledger.jsonl COMMITTED proof-of-pass rows (via lib/record_run.py)
clusters/<env>/ Flux entrypoint -> the reconciliation graph
`git ls-files` TRACKED files only; untracked scratch is not repo state
../home-network-k3s/docs/00-vision-and-scope.md OPTIONAL registry<->spec parity check
@@ -343,6 +359,169 @@ def spec_requirement_ids(spec_root: Path) -> set[str] | None:
return set(RID_RE.findall(doc.read_text()))
# ========================================================================== ledger
# PROOF axis. Everything here reads `conformance/runs/ledger.jsonl` (written ONLY by an
# explicit `lib/record_run.py` / `run.sh --record` invocation) and downgrades relentlessly.
# Nothing in this section may ever turn absent, ambiguous or aged evidence into green.
DEFAULT_LEDGER = CONF_ROOT / "runs" / "ledger.jsonl"
# Severity ranking, WORST FIRST. A requirement's proof is the MINIMUM over its verifying
# tests, so one failed/skipped/missing test drags the whole requirement down and a single
# proven test can never carry the others. `stale-proof` ranks below `proven` on purpose:
# it is a real past pass that no longer describes the current code, and it is NOT green.
PROOF_ORDER = {
"failed": 0, # a ledger row recorded a JUnit failure/error
"unknown": 1, # ran, but the log could not evidence pass vs self-skip
"not-derivable": 2, # no ledger row for this test at all (the pre-ledger default)
"junit-skipped": 3, # JUnit <skipped/> — did not execute
"skipped": 4, # SELF-SKIPPED green: exited 0 without exercising the behaviour
"stale-proof": 5, # proven at a sha the implementation/test has since moved past
"proven": 6, # the ONLY green state
}
# record_run.py's two "we cannot tell" states collapse to one reported state here; the
# per-test detail in status.json keeps the distinction for whoever has to fix it.
_PROOF_ALIASES = {
"unknown-no-sentinel-declared": "unknown",
"unknown-sentinel-absent": "unknown",
}
def read_ledger(path: Path) -> list[dict]:
"""Parse the append-only JSONL ledger. A malformed line is SKIPPED, never guessed at.
Absent or empty file -> [] , which reproduces this report's pre-ledger behaviour
exactly (every requirement `not-derivable`, headline PROVEN 0).
"""
if not path.is_file():
return []
rows: list[dict] = []
for line in path.read_text(errors="replace").splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
try:
obj = json.loads(line)
except json.JSONDecodeError:
continue
if isinstance(obj, dict) and obj.get("test"):
rows.append(obj)
return rows
def latest_by_test(rows: list[dict]) -> dict[str, dict]:
"""Most recent ledger row per test name (ts_utc, then file order as the tiebreak).
Latest wins rather than best-ever: a test that has since gone back to self-skipping
must stop reading as proven.
"""
latest: dict[str, dict] = {}
for i, r in enumerate(rows):
key = r["test"]
prev = latest.get(key)
if prev is None or (r.get("ts_utc", ""), i) >= (prev.get("ts_utc", ""),
prev["_i"]):
latest[key] = {**r, "_i": i}
return latest
class _GitOracle:
"""Small cached wrapper for the two git questions the staleness rule asks."""
def __init__(self, repo: Path):
self.repo = repo
self._last: dict[tuple[str, ...], str | None] = {}
self._anc: dict[tuple[str, str], bool] = {}
def _git(self, *args: str) -> subprocess.CompletedProcess:
return subprocess.run(["git", "-C", str(self.repo), *args],
capture_output=True, text=True)
def last_change(self, paths: tuple[str, ...]) -> str | None:
"""Commit that last touched any of `paths`; None if git knows none of them."""
if not paths:
return None
if paths in self._last:
return self._last[paths]
p = self._git("log", "-1", "--format=%H", "--", *paths)
sha = p.stdout.strip() if p.returncode == 0 else ""
self._last[paths] = sha or None
return self._last[paths]
def is_ancestor(self, older: str, newer: str) -> bool:
"""True iff `older` is an ancestor of (or equal to) `newer`.
False whenever either rev is unresolvable — an unknown or `-dirty` sha (a run
recorded from a modified working tree) can never satisfy the freshness test, which
is the intended fail-closed direction.
"""
key = (older, newer)
if key in self._anc:
return self._anc[key]
ok = self._git("merge-base", "--is-ancestor", older, newer).returncode == 0
self._anc[key] = ok
return ok
def derive_proof(git: _GitOracle, rid: str, verifiers: list[dict],
impl_files: list[dict], latest: dict[str, dict]) -> dict:
"""PROOF for one requirement, from the ledger, with the staleness rule applied.
STALENESS RULE — a proof is a claim about a specific tree, not a permanent badge:
relevant paths = every implementing artifact this report found for the requirement
+ every verifying test's directory
last_change = `git log -1 -- <relevant paths>`
A row's `proven` is honoured only if `last_change` is an ANCESTOR of the row's
`git_sha`, i.e. the run happened at or after the newest change to the code it
claims to prove. Otherwise it degrades to `stale-proof` (not green). A `-dirty`
sha, a sha this clone does not have (rebased/dropped), or a path git has never
seen (an uncommitted test) all fail the ancestry test and degrade the same way.
The requirement's state is the MINIMUM over its verifying tests: every verifying test
must be proven and fresh before the requirement reads `proven`. A test with no ledger
row contributes `not-derivable`, so a partially-recorded run cannot manufacture green.
"""
paths = sorted({f["path"] for f in impl_files}
| {os.path.dirname(v["path"]) for v in verifiers if v.get("path")})
last_change = git.last_change(tuple(paths))
per_test: list[dict] = []
for v in verifiers:
row = latest.get(v["name"])
if row is None:
per_test.append({"test": v["name"], "state": "not-derivable",
"reason": "no ledger row — this test has never been "
"recorded by run.sh --record / record_run.py"})
continue
base = _PROOF_ALIASES.get(row.get("proof", ""), row.get("proof", "unknown"))
if base not in PROOF_ORDER:
base = "unknown"
state, reason = base, row.get("note", "")
if base == "proven":
sha = row.get("git_sha", "")
if last_change and not git.is_ancestor(last_change, sha):
state = "stale-proof"
reason = (f"proven at {sha[:12]}, but the implementing manifests / test "
f"changed in {last_change[:12]}, which that run predates")
per_test.append({"test": v["name"], "state": state, "reason": reason,
"run_id": row.get("run_id"), "ts_utc": row.get("ts_utc"),
"git_sha": row.get("git_sha"), "env": row.get("env"),
"sentinel": row.get("sentinel"),
"junit_result": row.get("junit_result")})
if not per_test:
state = "not-derivable"
reason = "no requirement-tagged test exists to be proven"
else:
worst = min(per_test, key=lambda d: PROOF_ORDER[d["state"]])
state, reason = worst["state"], worst["reason"]
if state == "not-derivable" and not reason:
reason = "no committed ledger row for this requirement's verifying test(s)"
return {"state": state, "reason": reason, "last_change_sha": last_change,
"evidence_paths": paths, "per_test": per_test}
# ========================================================================== build
IMPL_ORDER = {"none": 0, "off-cluster": 1, "unwired": 2, "reconciled-partial": 3,
"reconciled": 4}
@@ -369,7 +548,13 @@ LABELS = {
def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, list[dict]],
applied: set[str], gitops_roots: set[str], spec_ids: set[str] | None,
env: str) -> dict:
env: str, ledger_rows: list[dict] | None = None) -> dict:
# PROOF axis inputs. `ledger_rows=None` / `[]` reproduces the pre-ledger behaviour:
# `latest` is empty, so every requirement derives `not-derivable` exactly as before.
ledger_rows = ledger_rows or []
latest = latest_by_test(ledger_rows)
git = _GitOracle(repo)
classified = [classify_test(t) for t in tests]
by_req_tests: dict[str, list[dict]] = {r["id"]: [] for r in registry}
for t, c in zip(tests, classified):
@@ -439,11 +624,9 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
"blocked_by": sorted({t["blocked_by"] for t in ts if t["blocked_by"]}),
"tests": ts,
},
# NEVER derived. See the module docstring.
"proof": {"state": "not-derivable",
"reason": "no committed, machine-readable record of a test run "
"exists in the repo (_artifacts/ and results/ are "
"git-ignored); DECISIONS-LOG.md prose is not a signal"},
# Derived ONLY from the committed ledger, and only as far as it reaches.
# With no ledger row this stays `not-derivable`, as it always was.
"proof": derive_proof(git, rid, ts, ev, latest),
"status": LABELS[(impl_state, verify_state)],
}
@@ -492,8 +675,19 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
and v["verification"]["state"] == "executable"),
"safety_critical_total": count(lambda v: v["safety_critical"]),
"safety_critical_without_executable_l3": len(safety_scaffold),
"proven": 0,
"proven_derivable": False,
# PROVEN is now a real count — but only of requirements whose EVERY verifying
# test has a committed ledger row that saw its pass sentinel, at a sha the
# implementation has not moved past. `stale-proof` and `skipped` are counted
# separately and are NOT green.
"proven": count(lambda v: v["proof"]["state"] == "proven"),
"proof_stale": count(lambda v: v["proof"]["state"] == "stale-proof"),
"proof_skipped": count(lambda v: v["proof"]["state"] == "skipped"),
"proof_failed": count(lambda v: v["proof"]["state"] == "failed"),
"proof_unknown": count(lambda v: v["proof"]["state"] == "unknown"),
"proof_not_derivable": count(
lambda v: v["proof"]["state"] in ("not-derivable", "junit-skipped")),
"ledger_rows": len(ledger_rows),
"proven_derivable": bool(ledger_rows),
"coverage_style_covered": count(lambda v: v["verification"]["state"] != "none"),
}
summary["uncited_by_any_reconciled_manifest"] = count(
@@ -552,16 +746,39 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
o.append(f"| Safety-critical requirements whose L3 test is a scaffold | "
f"**{s['safety_critical_without_executable_l3']}** of "
f"{s['safety_critical_total']} |")
o.append(f"| **PROVEN (a test actually passed)** | **{s['proven']} — NOT DERIVABLE** |")
o.append("")
o.append("> **Proof is deliberately not derived.** Nothing tracked in git records a "
"test result: `conformance/_artifacts/` and `results/` are git-ignored, and "
"the only record of a green run is DECISIONS-LOG.md prose — hand-maintained, "
"which is the very thing this report replaces. Per "
"`agents/definition-of-done.md` a requirement is done only when its tagged "
"test **passes**, so the strongest claim this report can make is "
"*wired and testable*. Closing the gap needs a committed, machine-readable "
"run ledger; until then, treat every row below as unproven.\n")
if s["proven_derivable"]:
o.append(f"| **PROVEN (every verifying test passed, evidenced)** | "
f"**{s['proven']}** |")
o.append(f"| Proof exists but is STALE (code changed after the run) | "
f"{s['proof_stale']} |")
o.append(f"| Verifying test SELF-SKIPPED green (not proof) | {s['proof_skipped']} |")
o.append(f"| Verifying test FAILED in the recorded run | {s['proof_failed']} |")
o.append(f"| Ran, but the log could not evidence pass vs self-skip | "
f"{s['proof_unknown']} |")
o.append(f"| No committed ledger row yet | {s['proof_not_derivable']} |")
o.append("")
o.append(f"> **Proof comes from `conformance/runs/ledger.jsonl`** "
f"({s['ledger_rows']} row(s)), written only by an explicit "
f"`run.sh --record` / `lib/record_run.py` invocation. A row reaches "
f"`proven` ONLY when the run log carried that test's own pass sentinel "
f"token — the L2/L3 tests self-skip GREEN (Chainsaw records `passed` on "
f"`DR_RESTORE_SKIPPED_OK` exactly as on a real run, and its JUnit carries "
f"no stdout), so a skip token maps to `skipped` and can never become "
f"proof. A proof is honoured only while the implementing manifests and "
f"the test itself have not changed since the recorded `git_sha`; "
f"otherwise it degrades to `stale-proof`, which is not green.\n")
else:
o.append(f"| **PROVEN (a test actually passed)** | **{s['proven']} — NOT DERIVABLE** |")
o.append("")
o.append("> **No proof is derivable: `conformance/runs/ledger.jsonl` is absent or "
"empty.** The ledger is the committed, machine-readable run record; until "
"a run is captured with `run.sh --record`, nothing tracked in git records "
"a test result (`conformance/_artifacts/` and `results/` are git-ignored, "
"and DECISIONS-LOG.md prose is hand-maintained — the very thing this "
"report replaces). Per `agents/definition-of-done.md` a requirement is "
"done only when its tagged test **passes**, so the strongest claim this "
"report can make is *wired and testable*; treat every row below as "
"unproven.\n")
o.append(f"> For contrast, `coverage.md` counts **{s['coverage_style_covered']}** "
f"requirements as covered — that count includes the "
f"{s['verify_scaffold_only']} whose only verifying test is a TODO scaffold.\n")
@@ -612,7 +829,15 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
if ver["blocked_by"]:
vcell += f" · blocked-by: {'; '.join(ver['blocked_by'])}"
sc = " ⚠safety" if v["safety_critical"] else ""
o.append(f"| {rid}{sc} | **{v['status']}** | {icell} | {vcell} | not-derivable |")
pr = v["proof"]
if pr["state"] == "not-derivable":
pcell = "not-derivable"
else:
runs = sorted({d["run_id"] for d in pr["per_test"] if d.get("run_id")})
pcell = f"**{pr['state']}**" if pr["state"] == "proven" else pr["state"]
if runs:
pcell += f" ({', '.join(runs)})"
o.append(f"| {rid}{sc} | **{v['status']}** | {icell} | {vcell} | {pcell} |")
if rep["safety_critical_without_executable_l3"]:
o.append("\n## Safety-critical requirements whose L3 fault test is a SCAFFOLD "
@@ -701,22 +926,76 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
o.append("| executable vs scaffold | TODO markers in the test body outrank the "
"`executable` annotation; a real body without the annotation still counts "
"and is flagged | conservative — ambiguity never upgrades |")
o.append("| **test actually passed** | **NOT DERIVED** — nothing in git records a "
"run result | refused |")
o.append("| **test actually passed** | the committed `runs/ledger.jsonl` row for "
"every verifying test: JUnit result **plus** that test's own pass sentinel "
"token grepped from the run log | exact where a run was recorded, refused "
"everywhere else — a self-skip token or a missing/unrecognised sentinel "
"NEVER yields `proven` |")
o.append("| proof still current | the row's `git_sha` must be a descendant of the "
"last commit touching the requirement's implementing manifests or its test "
"| conservative — an unresolvable or `-dirty` sha degrades to `stale-proof` |")
o.append("| cluster liveness (pods/CRs healthy) | **NOT DERIVED** — no cluster is "
"contacted; `reconciled` means Flux *would* apply it, not that it is "
"healthy | refused |")
o.append("| phase/board status | **NOT DERIVED** — `agents/task-board.md` is stale "
"and is not a signal | refused |")
o.append("\n## Run ledger — the committed proof-of-pass evidence\n")
if not s["proven_derivable"]:
o.append("- `conformance/runs/ledger.jsonl` holds no rows. Capture a run with "
"`./run.sh --record` (recording is never implicit: a run that silently "
"records itself is how a bad run becomes a permanent false proof), then "
"commit `conformance/runs/<run-id>/` **with its `run.log`** and the "
"ledger. The log is the load-bearing half — Chainsaw's JUnit carries no "
"stdout, so without it a self-skip is indistinguishable from a pass.")
else:
o.append(f"- {s['ledger_rows']} row(s) in `conformance/runs/ledger.jsonl`. "
f"Per-test evidence for every non-`not-derivable` requirement below; "
f"full rows (env, chainsaw/kube versions, sentinel, note) are in "
f"`status.json` under `by_requirement.<R-ID>.proof.per_test`.\n")
o.append("| Requirement | Proof | Test | Ledger state | Sentinel | Run |")
o.append("|---|---|---|---|---|---|")
for rid, v in rep["by_requirement"].items():
pr = v["proof"]
if pr["state"] == "not-derivable":
continue
for d in pr["per_test"]:
o.append(f"| {rid} | {pr['state']} | `{d['test']}` | {d['state']} | "
f"`{d.get('sentinel') or '—'}` | {d.get('run_id') or '—'} |")
stale = [(rid, v) for rid, v in rep["by_requirement"].items()
if v["proof"]["state"] == "stale-proof"]
if stale:
o.append("\n### Stale proofs — a real pass that no longer describes this tree\n")
o.append("These requirements DID pass, but their implementing manifests or "
"their verifying test changed in a commit the recorded run predates. "
"A stale proof is not green; re-run with `--record`.\n")
for rid, v in stale:
pr = v["proof"]
o.append(f"- **{rid}** — {pr['reason']}")
skipped = [(rid, v) for rid, v in rep["by_requirement"].items()
if v["proof"]["state"] == "skipped"]
if skipped:
o.append("\n### SELF-SKIPPED green — recorded, and deliberately NOT proof\n")
o.append("Chainsaw reported `passed` for these, because the test exited 0 "
"after emitting its skip token: a precondition (Longhorn "
"StorageClass, Velero CRDs, >=3 Ready nodes) was absent, so the "
"behaviour was never exercised. This is exactly the row a "
"JUnit-only ledger would have recorded as green.\n")
for rid, v in skipped:
toks = ", ".join(sorted({d.get("sentinel") or "?"
for d in v["proof"]["per_test"]
if d["state"] == "skipped"}))
o.append(f"- **{rid}** — skip token(s): `{toks}`")
o.append("\n## Advisory — untracked local run artifact (NOT counted)\n")
if advisory:
o.append(f"- `{advisory['path']}` (untracked, git-ignored), last written "
f"{advisory['mtime']}: {advisory['tests']} testcase(s), "
f"{advisory['failures']} failure(s), {advisory['skipped']} skipped.")
o.append(f"- requirement IDs it names: {', '.join(advisory['requirements']) or '—'}")
o.append("- This is one developer's local workspace, not repo state. It is shown "
"for orientation only and contributes to NO status above.")
o.append("- This is one developer's local workspace, not repo state, and it is NOT "
"the ledger. It is shown for orientation only and contributes to NO "
"status above — a JUnit alone cannot tell a self-skip from a pass.")
else:
o.append("- none present.")
return "\n".join(o) + "\n"
@@ -841,6 +1120,122 @@ def _selftest() -> int:
classify_test({"name": "s", "path": str(f), "layer": "L2"})["state"]
== "scaffold")
# ---------------------------------------------------------------- PROOF / ledger
# These are the assertions that stand between this report and a fake green.
check("empty ledger parses to no rows (pre-ledger behaviour is preserved)",
read_ledger(Path("/nonexistent/ledger.jsonl")) == [])
def row(**kw):
base = {"ts_utc": "2026-09-01T12:00:00Z", "git_sha": "deadbeef", "env": "dev",
"chainsaw_version": "v0.2.12", "kube_version": "v1.36.4", "layer": "L3",
"test": "dr-restore", "requirements": ["R-STATE-03"],
"junit_result": "passed", "sentinel": None, "proof": "skipped",
"run_id": "20260901T120000Z-deadbee", "note": ""}
base.update(kw)
return base
verifiers = [{"name": "dr-restore",
"path": "conformance/tests/l3-fault-injection/dr-restore/"
"chainsaw-test.yaml"}]
with tempfile.TemporaryDirectory() as td:
gr = Path(td)
env = {**os.environ, "GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@t",
"GIT_COMMITTER_NAME": "t", "GIT_COMMITTER_EMAIL": "t@t"}
def git(*a):
return subprocess.run(["git", "-C", str(gr), *a], capture_output=True,
text=True, env=env)
git("init", "-q", "-b", "main")
(gr / "platform").mkdir()
(gr / "platform" / "velero.yaml").write_text("kind: X # R-STATE-03\n")
git("add", "-A"); git("commit", "-qm", "one")
sha_a = git("rev-parse", "HEAD").stdout.strip()
(gr / "platform" / "velero.yaml").write_text("kind: X # R-STATE-03 v2\n")
git("add", "-A"); git("commit", "-qm", "two")
sha_b = git("rev-parse", "HEAD").stdout.strip()
oracle = _GitOracle(gr)
impl = [{"path": "platform/velero.yaml"}]
# *** THE CASE THIS WHOLE FEATURE EXISTS FOR ***
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
latest_by_test([row(proof="skipped", junit_result="passed",
sentinel="DR_RESTORE_SKIPPED_OK",
git_sha=sha_b)]))
check("a SELF-SKIP row on a GREEN JUnit is 'skipped', NEVER 'proven'",
pr["state"] == "skipped")
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
latest_by_test([row(proof="proven", git_sha=sha_b,
sentinel="DR_RESTORED_OK")]))
check("a proven row at the CURRENT sha is honoured", pr["state"] == "proven")
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
latest_by_test([row(proof="proven", git_sha=sha_a,
sentinel="DR_RESTORED_OK")]))
check("STALENESS: proven at a sha the implementation has moved past degrades "
"to stale-proof", pr["state"] == "stale-proof")
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
latest_by_test([row(proof="proven", git_sha=f"{sha_b}-dirty",
sentinel="DR_RESTORED_OK")]))
check("a run recorded from a DIRTY tree is never honoured as proof",
pr["state"] == "stale-proof")
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
latest_by_test([row(proof="proven", git_sha="0" * 40)]))
check("a git_sha this clone cannot resolve degrades, never upgrades",
pr["state"] == "stale-proof")
for bad, expect in (("failed", "failed"),
("junit-skipped", "junit-skipped"),
("unknown-sentinel-absent", "unknown"),
("unknown-no-sentinel-declared", "unknown"),
("something-new-from-the-future", "unknown")):
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
latest_by_test([row(proof=bad, git_sha=sha_b)]))
check(f"ledger state {bad!r} maps to {expect!r}, never to proven",
pr["state"] == expect)
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl, {})
check("no ledger row at all stays not-derivable",
pr["state"] == "not-derivable")
# Two verifying tests, only one proven: the requirement must NOT read proven.
two = verifiers + [{"name": "data-survival",
"path": "conformance/tests/l3-fault-injection/"
"data-survival/chainsaw-test.yaml"}]
pr = derive_proof(oracle, "R-STATE-03", two, impl,
latest_by_test([row(proof="proven", git_sha=sha_b)]))
check("a requirement is proven only when EVERY verifying test is",
pr["state"] == "not-derivable")
pr = derive_proof(oracle, "R-STATE-03", two, impl,
latest_by_test([row(proof="proven", git_sha=sha_b),
row(test="data-survival", proof="skipped",
git_sha=sha_b)]))
check("worst-of wins across verifying tests (skip drags proven down)",
pr["state"] == "skipped")
# Latest row wins, not best-ever: a test that regressed to self-skipping must
# stop reading as proven.
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
latest_by_test([row(ts_utc="2026-09-01T10:00:00Z",
proof="proven", git_sha=sha_b),
row(ts_utc="2026-09-02T10:00:00Z",
proof="skipped", git_sha=sha_b)]))
check("the LATEST ledger row wins, not the best one ever recorded",
pr["state"] == "skipped")
# A malformed ledger line is skipped, never guessed at.
lf = gr / "ledger.jsonl"
lf.write_text('{"test":"a","proof":"proven"}\nnot json at all\n\n'
'{"proof":"proven"}\n')
rows = read_ledger(lf)
check("malformed / test-less ledger lines are dropped, not guessed at",
len(rows) == 1 and rows[0]["test"] == "a")
if not ok:
return 1
sys.stderr.write("status.py selftest: PASS\n")
@@ -865,6 +1260,10 @@ def main() -> int:
help="exit 1 on defects this tool owns: registry/spec drift, "
"executable-annotation conflicts, or a safety-critical "
"requirement whose only L3 test is a scaffold")
ap.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER,
help=f"committed proof-of-pass ledger written by lib/record_run.py "
f"(default {DEFAULT_LEDGER}). Absent or empty => every "
f"requirement stays `not-derivable`, as before the ledger existed")
ap.add_argument("--selftest", action="store_true", help="run the built-in self-test")
args = ap.parse_args()
@@ -895,7 +1294,12 @@ def main() -> int:
applied, nodes = reconciled_files(repo, entry)
gitops_roots = {n["resolved"].split("/", 1)[0] for n in nodes if n["exists"]}
impl = scan_implementation(repo, tracked_files(repo))
rep = build(repo, registry, tests, impl, applied, gitops_roots, spec_ids, args.env)
ledger_rows = read_ledger(args.ledger)
rep = build(repo, registry, tests, impl, applied, gitops_roots, spec_ids, args.env,
ledger_rows)
rep["ledger"] = {"path": os.path.relpath(args.ledger, REPO_ROOT),
"rows": len(ledger_rows),
"present": args.ledger.is_file()}
rep["flux_graph"] = nodes
rep["gitops_roots"] = sorted(gitops_roots)
rep["reconciled_file_count"] = len(applied)
@@ -915,7 +1319,11 @@ def main() -> int:
f"(+{s['wired_or_partial_and_testable'] - s['wired_and_testable']} partial) | "
f"scaffold-only {s['verify_scaffold_only']} | "
f"safety-critical w/o executable L3 "
f"{s['safety_critical_without_executable_l3']} | PROVEN: not derivable")
f"{s['safety_critical_without_executable_l3']} | "
+ (f"PROVEN: {s['proven']} (stale {s['proof_stale']}, self-skipped "
f"{s['proof_skipped']}, failed {s['proof_failed']}, unknown "
f"{s['proof_unknown']}) from {s['ledger_rows']} ledger row(s)"
if s["proven_derivable"] else "PROVEN: not derivable (empty run ledger)"))
print(f"[status] wrote {args.out/'status.md'} and {args.out/'status.json'}")
rc = 0