feat(conformance): repo-derived spec status report (conformance/run.sh --status)
Operator ruling: "I care about the state of the repository and what the status of each spec is based on that state" — not a hand-maintained task board. This is that report. conformance/lib/status.py (stdlib-only, --selftest) sits beside lib/coverage.py and IMPORTS it rather than forking the registry/test parsers. Driven by a new `conformance/run.sh --status`; artifacts land in _artifacts/status/ alongside coverage and are git-ignored by the existing convention. Exit codes follow ci/lib.sh (0/1/3); --strict-coverage also applies the status hygiene gate. Signals derived per requirement: is there an implementing artifact; is it RECONCILED (walk resources: from clusters/dev, expand each Flux Kustomization's spec.path, recurse — so a commented-out resource correctly does not count); is there a tagged test and is its body executable or a TODO scaffold (TODO markers OUTRANK an executable: "true" annotation — ambiguity never upgrades a row). DELIBERATELY NOT DERIVED, and reported as such rather than defaulting to green: proof-of-pass (nothing tracked in git records a run result — _artifacts/ and results/ are git-ignored) and cluster liveness. "Reconciled" means Flux WOULD apply it, not that it is healthy. PROVEN is reported as "not derivable" for all 51. WHAT IT SHOWS THAT coverage.md DOES NOT. Coverage reports 37/51 covered, 0 safety-critical gaps. Three of those claims do not survive repo state: 1. 5 of the 37 "covered" requirements have only a TODO-body scaffold (R-STATE-03, R-OBS-02, R-OBS-04, R-GITOPS-03, R-GITOPS-02). Coverage counts a tagged file; it never looks inside it. 2. R-STATE-03 is SAFETY-CRITICAL and its L3 fault test asserts nothing. Coverage reports "0 safety-critical gaps" only because dr-restore/chainsaw-test.yaml exists and carries an L3 label. --strict now fails on this. 3. R-UPD-01/02/03 are NOT reconciled: platform/image-automation/ is commented out of platform/kustomization.yaml. Coverage shows them covered with executable L3 tests, which reads as delivered; automated updates are actually 0/3 wired+testable. Headline: 51 requirements | reconciled 20 (+12 partial) | executable test 32 | wired+testable 22 (+8 partial) | scaffold-only 5 | PROVEN 0 (not derivable). Also surfaces 18 requirements with no reconciled manifest citing their R-ID — listed as a traceability gap (R-TEST-04) rather than guessed either way. Annotating those manifests makes the report self-correct, so it acts as a forcing function instead of another hand-maintained map. Serves: R-TEST-01..06. Gate: MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
666cc0d809
commit
1252823d80
4 files changed
+1007
-2
No files matched your search
+39
-2
@@ -11,6 +11,11 @@
|
||||
# ./run.sh --layer l2 # one layer
|
||||
# ./run.sh --requirements R-NET-01,R-NET-02 --layer l2 # intersect: tagged AND in layer
|
||||
# ./run.sh --coverage-only # regenerate coverage report only (no cluster)
|
||||
# ./run.sh --status # DERIVED STATUS report only (no cluster): per
|
||||
# # requirement, is it implemented, is it actually
|
||||
# # reconciled by Flux, is its test executable or a
|
||||
# # scaffold. Proof-of-pass is deliberately NOT
|
||||
# # derived — see lib/status.py.
|
||||
# ./run.sh --lint # structural-validate test defs (no cluster)
|
||||
# ./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
||||
# ./run.sh --requirements R-NET-01 # ALSO asserts those IDs are covered (merge gate)
|
||||
@@ -20,6 +25,8 @@
|
||||
# _artifacts/report.junit.xml JUnit (Chainsaw output, or skipped-scaffold shim)
|
||||
# _artifacts/coverage/coverage.md human coverage report
|
||||
# _artifacts/coverage/coverage.json machine coverage report
|
||||
# _artifacts/status/status.md human requirement-status report (--status)
|
||||
# _artifacts/status/status.json machine requirement-status report (--status)
|
||||
#
|
||||
# Phase 0: no cluster exists. The behavior tests are valid scaffolds that cannot
|
||||
# execute yet, so run.sh emits a *skipped* JUnit and a coverage report, and exits
|
||||
@@ -36,12 +43,13 @@ source "${CONF_ROOT}/versions.env"
|
||||
|
||||
ARTIFACTS="${CONF_ROOT}/_artifacts"
|
||||
COVERAGE_OUT="${ARTIFACTS}/coverage"
|
||||
STATUS_OUT="${ARTIFACTS}/status"
|
||||
JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
LAYER="all"
|
||||
REQS=""
|
||||
MODE="run" # run | coverage-only | lint
|
||||
MODE="run" # run | coverage-only | status | lint
|
||||
STRICT_COVERAGE=0
|
||||
REQUIRE_ALL=0
|
||||
FORCE_NO_CLUSTER=0
|
||||
@@ -84,6 +92,7 @@ while [ $# -gt 0 ]; do
|
||||
--layer) LAYER="$2"; shift 2 ;;
|
||||
--requirements) REQS="$(conf_normalize_reqs "$2")"; shift 2 ;;
|
||||
--coverage-only) MODE="coverage-only"; shift ;;
|
||||
--status) MODE="status"; shift ;;
|
||||
--lint) MODE="lint"; shift ;;
|
||||
--strict-coverage) STRICT_COVERAGE=1; shift ;;
|
||||
--require-all) REQUIRE_ALL=1; shift ;;
|
||||
@@ -93,7 +102,7 @@ while [ $# -gt 0 ]; do
|
||||
esac
|
||||
done
|
||||
|
||||
mkdir -p "${ARTIFACTS}" "${COVERAGE_OUT}"
|
||||
mkdir -p "${ARTIFACTS}" "${COVERAGE_OUT}" "${STATUS_OUT}"
|
||||
|
||||
log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; }
|
||||
|
||||
@@ -111,6 +120,23 @@ gen_coverage() {
|
||||
"${flags[@]}"
|
||||
}
|
||||
|
||||
# --- Derived STATUS report (no cluster) ------------------------------------------------
|
||||
# Coverage answers "does a tagged test EXIST?". Status answers the operator's question —
|
||||
# "what is the real state of each requirement given what is actually in the repo?" — by
|
||||
# deriving, per requirement: implementing artifact? reachable from a Flux Kustomization
|
||||
# (reconciled, not merely authored)? tagged test that is EXECUTABLE rather than a TODO
|
||||
# scaffold? It REFUSES to derive whether a test passed: nothing tracked in git records a
|
||||
# run result, and a status report that defaults to green is worse than none.
|
||||
gen_status() {
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/status.py" \
|
||||
--repo "$(cd "${CONF_ROOT}/.." && pwd)" \
|
||||
--registry "${CONF_ROOT}/requirements/registry.yaml" \
|
||||
--tests "${CONF_ROOT}/tests" \
|
||||
--out "${STATUS_OUT}" \
|
||||
--env "${CONF_ENV:-dev}" \
|
||||
"$@"
|
||||
}
|
||||
|
||||
# --- Structural lint (no cluster): parse each test def and check the tag convention ----
|
||||
lint_tests() {
|
||||
log "Linting test definitions (structural, no cluster)"
|
||||
@@ -287,6 +313,17 @@ case "${MODE}" in
|
||||
log "coverage-only: done. See ${COVERAGE_OUT}/coverage.md"
|
||||
exit 0
|
||||
;;
|
||||
status)
|
||||
# rc: 0 report written, 1 a derivation/strict failure, 3 a check could not run
|
||||
# (SKIPPED, never counted as green — ci/lib.sh convention). Propagated as-is.
|
||||
rc=0
|
||||
# --strict-coverage is the harness's hygiene switch; apply it to status too.
|
||||
status_flags=()
|
||||
[ "${STRICT_COVERAGE}" = 1 ] && status_flags+=(--strict)
|
||||
gen_status ${status_flags[@]+"${status_flags[@]}"} || rc=$?
|
||||
log "status: done. See ${STATUS_OUT}/status.md"
|
||||
exit "${rc}"
|
||||
;;
|
||||
lint)
|
||||
lint_tests
|
||||
log "lint: done."
|
||||
|
||||
Reference in new issue
Block a user