feat(conformance): repo-derived spec status report (conformance/run.sh --status)

Operator ruling: "I care about the state of the repository and what the status of each
spec is based on that state" — not a hand-maintained task board. This is that report.

conformance/lib/status.py (stdlib-only, --selftest) sits beside lib/coverage.py and
IMPORTS it rather than forking the registry/test parsers. Driven by a new
`conformance/run.sh --status`; artifacts land in _artifacts/status/ alongside coverage
and are git-ignored by the existing convention. Exit codes follow ci/lib.sh (0/1/3);
--strict-coverage also applies the status hygiene gate.

Signals derived per requirement: is there an implementing artifact; is it RECONCILED
(walk resources: from clusters/dev, expand each Flux Kustomization's spec.path, recurse
— so a commented-out resource correctly does not count); is there a tagged test and is
its body executable or a TODO scaffold (TODO markers OUTRANK an executable: "true"
annotation — ambiguity never upgrades a row).

DELIBERATELY NOT DERIVED, and reported as such rather than defaulting to green:
proof-of-pass (nothing tracked in git records a run result — _artifacts/ and results/
are git-ignored) and cluster liveness. "Reconciled" means Flux WOULD apply it, not that
it is healthy. PROVEN is reported as "not derivable" for all 51.

WHAT IT SHOWS THAT coverage.md DOES NOT. Coverage reports 37/51 covered, 0
safety-critical gaps. Three of those claims do not survive repo state:

1. 5 of the 37 "covered" requirements have only a TODO-body scaffold (R-STATE-03,
   R-OBS-02, R-OBS-04, R-GITOPS-03, R-GITOPS-02). Coverage counts a tagged file; it
   never looks inside it.
2. R-STATE-03 is SAFETY-CRITICAL and its L3 fault test asserts nothing. Coverage reports
   "0 safety-critical gaps" only because dr-restore/chainsaw-test.yaml exists and carries
   an L3 label. --strict now fails on this.
3. R-UPD-01/02/03 are NOT reconciled: platform/image-automation/ is commented out of
   platform/kustomization.yaml. Coverage shows them covered with executable L3 tests,
   which reads as delivered; automated updates are actually 0/3 wired+testable.

Headline: 51 requirements | reconciled 20 (+12 partial) | executable test 32 |
wired+testable 22 (+8 partial) | scaffold-only 5 | PROVEN 0 (not derivable).

Also surfaces 18 requirements with no reconciled manifest citing their R-ID — listed as
a traceability gap (R-TEST-04) rather than guessed either way. Annotating those manifests
makes the report self-correct, so it acts as a forcing function instead of another
hand-maintained map.

Serves: R-TEST-01..06. Gate: MERGE GATE: PASS.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-08-31 22:44:10 -05:00
1 parent 666cc0d809
commit 1252823d80
4 files changed
+1007 -2

No files matched your search

+31
View File
@@ -18,6 +18,7 @@ behavioral / L3 fault-injection** (this harness).
./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
./run.sh --layer l2 # one layer: l1 | l2 | l3 | meta | all
./run.sh --coverage-only # regenerate the coverage report (no cluster)
./run.sh --status # derived requirement-STATUS report (no cluster)
./run.sh --lint # structural-validate test defs (no cluster)
./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
./run.sh --require-all # phase-exit gate: every requirement must be covered
@@ -37,6 +38,33 @@ behavioral / L3 fault-injection** (this harness).
used to be here; it now has a real L3 test — `tests/l3-fault-injection/node-loss` — and
is covered, not exempted.)
### Coverage vs status — two different questions
`coverage.md` answers **"does a requirement-tagged test EXIST for R-x?"** (R-TEST-04).
`status.md` (`./run.sh --status`, generator `lib/status.py`) answers the operator's
question — **"what is the real state of each requirement given what is actually in the
repo?"** — deriving three orthogonal axes per requirement:
1. **implementation** — is there a tracked artifact naming the requirement?
2. **reconciliation** — is that artifact reachable by walking `resources:` from
`clusters/<env>/kustomization.yaml` and expanding every Flux `Kustomization`'s
`spec.path`? This is what separates **authored** from **actually reconciled**: a
commented-out `resources:` entry (e.g. `platform/image-automation`) is correctly
reported as in-git-but-not-on-any-cluster.
3. **verification** — is the tagged test EXECUTABLE, or a TODO scaffold? Coverage counts
both as "covered"; status does not. TODO markers in a body always outrank an
`executable: "true"` annotation, so ambiguity never upgrades a test.
**It deliberately refuses to derive proof-of-pass.** Nothing tracked in git records a test
result — `_artifacts/` and `results/` are git-ignored and DECISIONS-LOG.md prose is not a
signal — so every requirement's proof state is reported as `not-derivable` rather than
defaulting to green. A status report that overstates completion is worse than none
(`agents/definition-of-done.md`). Closing that gap needs a **committed, machine-readable
run ledger**; until then the strongest claim the report makes is *wired and testable*.
`--strict-coverage` also applies status's hygiene gate (registry↔spec drift,
annotation/body conflicts, a safety-critical requirement whose only L3 test is a scaffold).
Exit codes follow `ci/lib.sh`: `0` pass / `1` fail / `3` skipped — a skip is never green.
`run.sh` is the **only** entrypoint. Local runs and the in-cluster CI Chainsaw Job call
the *same* script with the same config, so results are identical.
@@ -46,6 +74,8 @@ the *same* script with the same config, so results are identical.
| `_artifacts/report.junit.xml` | JUnit — Chainsaw results, or a **skipped-scaffold** shim when no cluster | CI gate |
| `_artifacts/coverage/coverage.md` | Human requirement-coverage report | PR reviewers / logs |
| `_artifacts/coverage/coverage.json` | Machine coverage: `{requirement: [tests]}`, gaps, safety gaps | CI gate |
| `_artifacts/status/status.md` | Human requirement-STATUS report derived from repo state | operator / PM |
| `_artifacts/status/status.json` | Machine status: implementation / reconciliation / verification per requirement | tooling |
## Reproducibility contract (R-TEST-02 — what the harness guarantees)
- **Single command.** `./run.sh` (or a tagged/layer subset) runs everything and emits the
@@ -74,6 +104,7 @@ requirements/registry.yaml mirror of EVERY spec R-* ID (coverage enumerates fr
lib/
common.sh tagging convention + test discovery/selection helpers
coverage.py requirement-coverage generator (stdlib only)
status.py requirement-STATUS generator (stdlib only; --selftest)
emit_skipped_junit.py Phase-0 skipped-JUnit shim (no cluster)
tests/
l1-rendering/ l2-behavioral/ l3-fault-injection/ meta/ (see tests/README.md)