feat(conformance): repo-derived spec status report (conformance/run.sh --status)
Operator ruling: "I care about the state of the repository and what the status of each spec is based on that state" — not a hand-maintained task board. This is that report. conformance/lib/status.py (stdlib-only, --selftest) sits beside lib/coverage.py and IMPORTS it rather than forking the registry/test parsers. Driven by a new `conformance/run.sh --status`; artifacts land in _artifacts/status/ alongside coverage and are git-ignored by the existing convention. Exit codes follow ci/lib.sh (0/1/3); --strict-coverage also applies the status hygiene gate. Signals derived per requirement: is there an implementing artifact; is it RECONCILED (walk resources: from clusters/dev, expand each Flux Kustomization's spec.path, recurse — so a commented-out resource correctly does not count); is there a tagged test and is its body executable or a TODO scaffold (TODO markers OUTRANK an executable: "true" annotation — ambiguity never upgrades a row). DELIBERATELY NOT DERIVED, and reported as such rather than defaulting to green: proof-of-pass (nothing tracked in git records a run result — _artifacts/ and results/ are git-ignored) and cluster liveness. "Reconciled" means Flux WOULD apply it, not that it is healthy. PROVEN is reported as "not derivable" for all 51. WHAT IT SHOWS THAT coverage.md DOES NOT. Coverage reports 37/51 covered, 0 safety-critical gaps. Three of those claims do not survive repo state: 1. 5 of the 37 "covered" requirements have only a TODO-body scaffold (R-STATE-03, R-OBS-02, R-OBS-04, R-GITOPS-03, R-GITOPS-02). Coverage counts a tagged file; it never looks inside it. 2. R-STATE-03 is SAFETY-CRITICAL and its L3 fault test asserts nothing. Coverage reports "0 safety-critical gaps" only because dr-restore/chainsaw-test.yaml exists and carries an L3 label. --strict now fails on this. 3. R-UPD-01/02/03 are NOT reconciled: platform/image-automation/ is commented out of platform/kustomization.yaml. Coverage shows them covered with executable L3 tests, which reads as delivered; automated updates are actually 0/3 wired+testable. Headline: 51 requirements | reconciled 20 (+12 partial) | executable test 32 | wired+testable 22 (+8 partial) | scaffold-only 5 | PROVEN 0 (not derivable). Also surfaces 18 requirements with no reconciled manifest citing their R-ID — listed as a traceability gap (R-TEST-04) rather than guessed either way. Annotating those manifests makes the report self-correct, so it acts as a forcing function instead of another hand-maintained map. Serves: R-TEST-01..06. Gate: MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
666cc0d809
commit
1252823d80
4 files changed
+1007
-2
No files matched your search
@@ -18,6 +18,7 @@ behavioral / L3 fault-injection** (this harness).
|
||||
./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
|
||||
./run.sh --layer l2 # one layer: l1 | l2 | l3 | meta | all
|
||||
./run.sh --coverage-only # regenerate the coverage report (no cluster)
|
||||
./run.sh --status # derived requirement-STATUS report (no cluster)
|
||||
./run.sh --lint # structural-validate test defs (no cluster)
|
||||
./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
||||
./run.sh --require-all # phase-exit gate: every requirement must be covered
|
||||
@@ -37,6 +38,33 @@ behavioral / L3 fault-injection** (this harness).
|
||||
used to be here; it now has a real L3 test — `tests/l3-fault-injection/node-loss` — and
|
||||
is covered, not exempted.)
|
||||
|
||||
### Coverage vs status — two different questions
|
||||
|
||||
`coverage.md` answers **"does a requirement-tagged test EXIST for R-x?"** (R-TEST-04).
|
||||
`status.md` (`./run.sh --status`, generator `lib/status.py`) answers the operator's
|
||||
question — **"what is the real state of each requirement given what is actually in the
|
||||
repo?"** — deriving three orthogonal axes per requirement:
|
||||
|
||||
1. **implementation** — is there a tracked artifact naming the requirement?
|
||||
2. **reconciliation** — is that artifact reachable by walking `resources:` from
|
||||
`clusters/<env>/kustomization.yaml` and expanding every Flux `Kustomization`'s
|
||||
`spec.path`? This is what separates **authored** from **actually reconciled**: a
|
||||
commented-out `resources:` entry (e.g. `platform/image-automation`) is correctly
|
||||
reported as in-git-but-not-on-any-cluster.
|
||||
3. **verification** — is the tagged test EXECUTABLE, or a TODO scaffold? Coverage counts
|
||||
both as "covered"; status does not. TODO markers in a body always outrank an
|
||||
`executable: "true"` annotation, so ambiguity never upgrades a test.
|
||||
|
||||
**It deliberately refuses to derive proof-of-pass.** Nothing tracked in git records a test
|
||||
result — `_artifacts/` and `results/` are git-ignored and DECISIONS-LOG.md prose is not a
|
||||
signal — so every requirement's proof state is reported as `not-derivable` rather than
|
||||
defaulting to green. A status report that overstates completion is worse than none
|
||||
(`agents/definition-of-done.md`). Closing that gap needs a **committed, machine-readable
|
||||
run ledger**; until then the strongest claim the report makes is *wired and testable*.
|
||||
`--strict-coverage` also applies status's hygiene gate (registry↔spec drift,
|
||||
annotation/body conflicts, a safety-critical requirement whose only L3 test is a scaffold).
|
||||
Exit codes follow `ci/lib.sh`: `0` pass / `1` fail / `3` skipped — a skip is never green.
|
||||
|
||||
`run.sh` is the **only** entrypoint. Local runs and the in-cluster CI Chainsaw Job call
|
||||
the *same* script with the same config, so results are identical.
|
||||
|
||||
@@ -46,6 +74,8 @@ the *same* script with the same config, so results are identical.
|
||||
| `_artifacts/report.junit.xml` | JUnit — Chainsaw results, or a **skipped-scaffold** shim when no cluster | CI gate |
|
||||
| `_artifacts/coverage/coverage.md` | Human requirement-coverage report | PR reviewers / logs |
|
||||
| `_artifacts/coverage/coverage.json` | Machine coverage: `{requirement: [tests]}`, gaps, safety gaps | CI gate |
|
||||
| `_artifacts/status/status.md` | Human requirement-STATUS report derived from repo state | operator / PM |
|
||||
| `_artifacts/status/status.json` | Machine status: implementation / reconciliation / verification per requirement | tooling |
|
||||
|
||||
## Reproducibility contract (R-TEST-02 — what the harness guarantees)
|
||||
- **Single command.** `./run.sh` (or a tagged/layer subset) runs everything and emits the
|
||||
@@ -74,6 +104,7 @@ requirements/registry.yaml mirror of EVERY spec R-* ID (coverage enumerates fr
|
||||
lib/
|
||||
common.sh tagging convention + test discovery/selection helpers
|
||||
coverage.py requirement-coverage generator (stdlib only)
|
||||
status.py requirement-STATUS generator (stdlib only; --selftest)
|
||||
emit_skipped_junit.py Phase-0 skipped-JUnit shim (no cluster)
|
||||
tests/
|
||||
l1-rendering/ l2-behavioral/ l3-fault-injection/ meta/ (see tests/README.md)
|
||||
|
||||
Reference in new issue
Block a user