jakman85andClaude Opus 5
e9a4b669d4
status.py has always reported PROVEN: 0 for all 51 requirements, and named the fix in its
own header: "a committed, machine-readable run ledger". There was none — _artifacts/ is
gitignored, the root .gitignore kills *.junit.xml, and a stray JUnit was surfaced only as
a quarantined mtime advisory. Built BEFORE the first real L3 execution so that run is
captured rather than reconstructed from memory.
THE TRAP THIS EXISTS TO AVOID. These tests exit 0 and Chainsaw records `passed` on a
SELF-SKIP exactly as on a real run, and Chainsaw's JUnit carries no stdout — only
properties. So committing the JUnit as proof would let DR_RESTORE_SKIPPED_OK be recorded
as proof of R-STATE-03: authoritative-looking and wrong, which is worse than no ledger.
The sentinel token is the ONLY thing distinguishing proven from skipped, and it lives
solely in the run log. The ledger therefore records the LOG alongside the JUnit and
derives proof from the sentinel, never from the JUnit result.
Sentinel tokens are DERIVED, not hand-listed: record_run.py parses every
contains($stdout,'TOKEN') out of each test's own chainsaw-test.yaml, so the list cannot
drift from the tests. 23 of 35 tests declare no sentinel at all.
decide() is the single place `proven` is emitted, ordered WORST-FIRST:
JUnit failure/error -> failed
JUnit <skipped/> -> junit-skipped
ANY declared skip token in the log -> skipped (beats a green JUnit AND beats a pass
token present in the same log)
no sentinel declared -> proven ONLY for L0/L1 (whose JUnit is truthful on its own, since
run_l1_offline.py maps exit 3 to <skipped/>); written as an
explicit exempt-set so an unknown/blank layer fails CLOSED
EVERY declared pass token present -> proven
otherwise -> unknown-sentinel-absent
Log attribution is a state machine over Chainsaw's per-test headers; a log with no
recognisable header falls back to whole-log scanning restricted to GLOBALLY-UNIQUE tokens,
so DNS_OK (owned by two tests) can never prove either. record_run.py exits 3 if the JUnit
or the log is missing — nothing recorded, never green.
STALENESS: a proven row is honoured only while `git merge-base --is-ancestor` holds
between the requirement's last implementing/test change and the row's git_sha; otherwise
it degrades to stale-proof. A -dirty sha, an unknown rev, or an uncommitted test path all
fail ancestry the same way. A requirement reads proven only when EVERY verifying test
does (worst-of), and the latest row per test wins — not best-ever.
REFUSAL POSTURE PRESERVED: with an empty ledger the report is unchanged in intent — every
requirement not-derivable, headline "PROVEN: 0 — NOT DERIVABLE". This replaces "not
derivable" with real evidence only where evidence exists, never with optimism.
--record is explicit, never implicit (a run that silently records itself is how a bad run
becomes permanent false proof), is rejected for --status/--coverage-only/--lint, and
re-execs under tee returning ${PIPESTATUS[0]} so recording can never soften a gate. It
rm -f's a stale _artifacts/report.junit.xml first — a hazard hit live during development,
where a stale JUnit got paired with a fresh log.
Selftests: record_run.py 23 checks (incl. end-to-end Chainsaw-passed + NODE_LOSS_SKIPPED_OK
-> proof "skipped"), status.py +16 against a throwaway git repo (self-skip -> skipped;
proven-at-older-sha -> stale-proof; dirty/unresolvable -> stale-proof; worst-of;
latest-row-wins; malformed lines dropped). No regressions: --coverage-only, --status,
--lint, --strict-coverage and the merge-gate shape all rc=0 and unchanged; nothing in ci/
references status.py.
Root .gitignore needed no change — conformance/runs/ is not itself excluded, so the deeper
negations override the root's global *.junit.xml. Verified with git check-ignore.
Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
479 lines
23 KiB
Bash
Executable File
479 lines
23 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
###############################################################################
|
|
# conformance/run.sh — the single-command conformance runner (R-TEST-02/03/04)
|
|
#
|
|
# ONE command runs the suite (or a requirement-/layer-scoped subset) and always
|
|
# emits JUnit + a requirement-coverage report. Local runs and the in-cluster CI
|
|
# Chainsaw Job invoke this same script so results are identical (reproducibility).
|
|
#
|
|
# ./run.sh # full suite; skips L1/L2/L3 if no cluster
|
|
# ./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
|
|
# ./run.sh --layer l2 # one layer (l0|l1|l2|l3|meta|all; l0/l1 need no cluster)
|
|
# ./run.sh --requirements R-NET-01,R-NET-02 --layer l2 # intersect: tagged AND in layer
|
|
# ./run.sh --coverage-only # regenerate coverage report only (no cluster)
|
|
# ./run.sh --status # DERIVED STATUS report only (no cluster): per
|
|
# # requirement, is it implemented, is it actually
|
|
# # reconciled by Flux, is its test executable or a
|
|
# # scaffold, and whether the committed run ledger
|
|
# # (runs/ledger.jsonl) actually proves it — see
|
|
# # lib/status.py. No ledger row => not-derivable.
|
|
# ./run.sh --lint # structural-validate test defs (no cluster)
|
|
# ./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
|
# ./run.sh --requirements R-NET-01 # ALSO asserts those IDs are covered (merge gate)
|
|
# ./run.sh --require-all # phase-exit: every requirement must be covered
|
|
# ./run.sh --record # RUN AND CAPTURE: tee the whole run to
|
|
# # runs/<UTC>-<short-sha>/run.log, copy the JUnit
|
|
# # beside it, and append one ledger row per
|
|
# # testcase to runs/ledger.jsonl (lib/record_run.py)
|
|
#
|
|
# --record — PROOF-OF-PASS, CAPTURED AT RUN TIME (why it is a separate flag)
|
|
# --------------------------------------------------------------------------
|
|
# Without it, nothing about a green run ever reaches repo state: _artifacts/ is gitignored
|
|
# and the root .gitignore kills *.junit.xml globally, which is why `--status` reports
|
|
# PROVEN 0 for all 51 requirements. `--record` writes the one tracked thing that fixes
|
|
# that — `conformance/runs/` (see conformance/.gitignore for the negations).
|
|
#
|
|
# RECORDING IS DELIBERATELY EXPLICIT. A run that silently records itself is how a bad run
|
|
# becomes a permanent false proof, so the plain `./run.sh` writes no ledger row at all.
|
|
#
|
|
# BOTH FILES ARE KEPT, and the LOG is the load-bearing one. The L2/L3 tests self-skip
|
|
# GREEN: when a precondition is missing they echo e.g. `DR_RESTORE_SKIPPED_OK` and exit 0,
|
|
# and Chainsaw records `passed` exactly as on a real run. Chainsaw's JUnit carries no
|
|
# stdout, so the sentinel token in run.log is the ONLY thing that separates proven from
|
|
# skipped. A JUnit committed on its own would record a skip as proof of R-STATE-03.
|
|
#
|
|
# Artifacts (always written under conformance/_artifacts/, gitignored):
|
|
# _artifacts/report.junit.xml JUnit (Chainsaw output, or skipped-scaffold shim)
|
|
# _artifacts/coverage/coverage.md human coverage report
|
|
# _artifacts/coverage/coverage.json machine coverage report
|
|
# _artifacts/status/status.md human requirement-status report (--status)
|
|
# _artifacts/status/status.json machine requirement-status report (--status)
|
|
#
|
|
# Run ledger (TRACKED — commit these; --record only):
|
|
# runs/<UTC>-<short-sha>/run.log full tee'd run output (carries the sentinels)
|
|
# runs/<UTC>-<short-sha>/report.junit.xml the JUnit for that same run
|
|
# runs/ledger.jsonl append-only, one row per testcase per run
|
|
#
|
|
# Phase 0: no cluster exists. The behavior tests are valid scaffolds that cannot
|
|
# execute yet, so run.sh emits a *skipped* JUnit and a coverage report, and exits
|
|
# green — the harness is proven reproducible while later phases fill in behavior.
|
|
###############################################################################
|
|
set -euo pipefail
|
|
|
|
CONF_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
export CONF_ROOT
|
|
# shellcheck source=lib/common.sh
|
|
source "${CONF_ROOT}/lib/common.sh"
|
|
# shellcheck source=versions.env
|
|
source "${CONF_ROOT}/versions.env"
|
|
|
|
ARTIFACTS="${CONF_ROOT}/_artifacts"
|
|
COVERAGE_OUT="${ARTIFACTS}/coverage"
|
|
STATUS_OUT="${ARTIFACTS}/status"
|
|
JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
|
|
PYTHON="${PYTHON:-python3}"
|
|
|
|
# Every argument as given, so `--record` can re-exec this script unchanged (minus itself)
|
|
# under `tee`. Captured BEFORE parsing so nothing is normalized away.
|
|
ORIG_ARGS=("$@")
|
|
|
|
LAYER="all"
|
|
REQS=""
|
|
MODE="run" # run | coverage-only | status | lint
|
|
STRICT_COVERAGE=0
|
|
REQUIRE_ALL=0
|
|
FORCE_NO_CLUSTER=0
|
|
RECORD=0
|
|
|
|
# Documented, out-of-scope coverage gaps exempt from the completeness gate: requirements
|
|
# owned by another workstream that WS-TEST does not yet scaffold. Listed here so the
|
|
# exemption is explicit and reviewable, not silent.
|
|
#
|
|
# EMPTY as of 2026-09-01. R-ENV-02 and R-GITOPS-01 were the last two entries — exempted
|
|
# because they were unscaffolded while the coarse path->requirement map keys clusters/*
|
|
# and ci/* to them, so any overlay/CI edit would have false-blocked. They now have a real
|
|
# verifying L0 test (tests/l0-static/flux-postbuild-substitution), which asserts the
|
|
# substitution mechanism that IS the "authored once, promoted" contract, so they are
|
|
# covered rather than exempt and the completeness gate can hold them to it. That test
|
|
# names its own limits in its header: it proves the STATIC property (no manifest renders
|
|
# an undefined variable), not the live-cluster half of either requirement.
|
|
#
|
|
# FOLLOW-UP (WS-TEST) — HALF DONE. The debt was "make the matcher stop keying on doc-header
|
|
# tokens so real requirements need not be exempted". The CONTENT-SCAN half is now fixed in
|
|
# ci/touched-requirements.sh: it scans the added/removed lines of the diff, and skips *.md, so
|
|
# a requirement named in a header or in prose no longer counts as a declaration. Consequence
|
|
# to be aware of: a manifest whose HEADER declares the requirements it serves no longer
|
|
# contributes those IDs unless the change actually touches a line carrying them — the
|
|
# authoritative signal is the `conformance.home.k3s/R-*` label on a test, which still is.
|
|
# The PATH-MAP half remains: ci/touched-requirements.sh keys clusters/* and ci/* to
|
|
# R-ENV-02 + R-GITOPS-01 wholesale, which is why those two must stay exempted here. Tighten
|
|
# the path map (or scaffold the two requirements) and this list empties.
|
|
#
|
|
# R-STATE-02 (node-loss) WAS listed here (owned by WS-STORAGE, "delivered with Longhorn/infra")
|
|
# but now has a real verifying L3 test — tests/l3-fault-injection/node-loss — so it is no
|
|
# longer a gap and has been removed from this list; the coverage report counts it as covered.
|
|
DOCUMENTED_GAPS=""
|
|
|
|
# Print the header block (line 2 through the closing `####` rule, which is dropped) as
|
|
# help. Bounded by the rule rather than a fixed line number so the header can grow.
|
|
usage() { sed -n '2,/^#####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; }
|
|
|
|
while [ $# -gt 0 ]; do
|
|
case "$1" in
|
|
--layer) LAYER="$2"; shift 2 ;;
|
|
--requirements) REQS="$(conf_normalize_reqs "$2")"; shift 2 ;;
|
|
--coverage-only) MODE="coverage-only"; shift ;;
|
|
--status) MODE="status"; shift ;;
|
|
--lint) MODE="lint"; shift ;;
|
|
--strict-coverage) STRICT_COVERAGE=1; shift ;;
|
|
--require-all) REQUIRE_ALL=1; shift ;;
|
|
--no-cluster) FORCE_NO_CLUSTER=1; shift ;;
|
|
--record) RECORD=1; shift ;;
|
|
-h|--help) usage; exit 0 ;;
|
|
*) echo "unknown arg: $1" >&2; usage; exit 2 ;;
|
|
esac
|
|
done
|
|
|
|
mkdir -p "${ARTIFACTS}" "${COVERAGE_OUT}" "${STATUS_OUT}"
|
|
|
|
log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; }
|
|
|
|
###############################################################################
|
|
# --record: capture this run as committed evidence, then hand it to the ledger.
|
|
#
|
|
# Implemented as a ONE-LEVEL re-exec of this same script under `tee`, so the run itself is
|
|
# byte-for-byte the ordinary run (same code path, same artifacts) and the log is the exact
|
|
# operator-visible output — no second, differently-behaving "recording mode" to drift.
|
|
# CONF_RECORD_CHILD guards against recursion.
|
|
#
|
|
# The recorder runs REGARDLESS of the run's exit status: a red run must be recorded as red
|
|
# (that is evidence too), and run.sh still exits with the child's real rc, so no gate is
|
|
# softened by recording. `${PIPESTATUS[0]}` is what carries that rc past `tee`.
|
|
###############################################################################
|
|
if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
|
|
case "${MODE}" in
|
|
run) ;;
|
|
# --record is about EVIDENCE OF A RUN. --coverage-only/--status/--lint execute no
|
|
# test, so a ledger row from one of them would be a row about nothing.
|
|
*) echo "--record captures a TEST RUN; --${MODE} executes none." >&2; exit 2 ;;
|
|
esac
|
|
RUN_SHA="$(git -C "${CONF_ROOT}" rev-parse --short HEAD 2>/dev/null || echo nogit)"
|
|
RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-${RUN_SHA}"
|
|
RUN_DIR="${CONF_ROOT}/runs/${RUN_ID}"
|
|
mkdir -p "${RUN_DIR}"
|
|
# Drop any JUnit left behind by a PREVIOUS run before starting, so this run dir can only
|
|
# ever capture a report THIS run produced. Copying a stale artifact would pair someone
|
|
# else's result with this run's log — a fabricated pairing, which is precisely the class
|
|
# of false proof the ledger exists to prevent.
|
|
rm -f "${JUNIT_OUT}"
|
|
# Strip --record from the argv handed to the child; everything else passes through.
|
|
CHILD_ARGS=()
|
|
for a in ${ORIG_ARGS[@]+"${ORIG_ARGS[@]}"}; do
|
|
[ "$a" = "--record" ] || CHILD_ARGS+=("$a")
|
|
done
|
|
log "RECORD: capturing this run to runs/${RUN_ID}/ (run.log + report.junit.xml)"
|
|
set +e
|
|
CONF_RECORD_CHILD=1 "${BASH_SOURCE[0]}" ${CHILD_ARGS[@]+"${CHILD_ARGS[@]}"} 2>&1 \
|
|
| tee "${RUN_DIR}/run.log"
|
|
RUN_RC=${PIPESTATUS[0]}
|
|
set -e
|
|
if [ -f "${JUNIT_OUT}" ]; then
|
|
cp "${JUNIT_OUT}" "${RUN_DIR}/report.junit.xml"
|
|
else
|
|
log "RECORD: no ${JUNIT_OUT} was produced — the ledger step will SKIP (never green)."
|
|
fi
|
|
# Explicit, separate step: read the JUnit + the log and append one row per testcase.
|
|
# rc 3 = inputs absent (nothing recorded, not green); it must not mask the run's own rc.
|
|
# CONF_LEDGER exists so the harness's own tests can exercise this path without writing
|
|
# to the real ledger; operators should never set it.
|
|
"${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
|
|
--run-dir "${RUN_DIR}" --env "${CONF_ENV:-dev}" \
|
|
--ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
|
|
log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
|
|
log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
|
|
exit "${RUN_RC}"
|
|
fi
|
|
|
|
# --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
|
|
gen_coverage() {
|
|
# Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no
|
|
# requirement is exempt, and handing the flag an empty string would be a silent no-op
|
|
# that reads like an exemption in the CLI log.
|
|
local flags=()
|
|
[ -n "${DOCUMENTED_GAPS}" ] && flags+=(--allow-gap "${DOCUMENTED_GAPS}")
|
|
[ "${STRICT_COVERAGE}" = 1 ] && flags+=(--strict)
|
|
# A tagged subset run is also the merge gate: the touched requirements MUST be covered.
|
|
[ -n "${REQS}" ] && flags+=(--require "${REQS}")
|
|
[ "${REQUIRE_ALL}" = 1 ] && flags+=(--require-all)
|
|
"${PYTHON}" "${CONF_ROOT}/lib/coverage.py" \
|
|
--registry "${CONF_ROOT}/requirements/registry.yaml" \
|
|
--tests "${CONF_ROOT}/tests" \
|
|
--out "${COVERAGE_OUT}" \
|
|
"${flags[@]}"
|
|
}
|
|
|
|
# --- Derived STATUS report (no cluster) ------------------------------------------------
|
|
# Coverage answers "does a tagged test EXIST?". Status answers the operator's question —
|
|
# "what is the real state of each requirement given what is actually in the repo?" — by
|
|
# deriving, per requirement: implementing artifact? reachable from a Flux Kustomization
|
|
# (reconciled, not merely authored)? tagged test that is EXECUTABLE rather than a TODO
|
|
# scaffold? It REFUSES to derive whether a test passed: nothing tracked in git records a
|
|
# run result, and a status report that defaults to green is worse than none.
|
|
gen_status() {
|
|
"${PYTHON}" "${CONF_ROOT}/lib/status.py" \
|
|
--repo "$(cd "${CONF_ROOT}/.." && pwd)" \
|
|
--registry "${CONF_ROOT}/requirements/registry.yaml" \
|
|
--tests "${CONF_ROOT}/tests" \
|
|
--out "${STATUS_OUT}" \
|
|
--env "${CONF_ENV:-dev}" \
|
|
"$@"
|
|
}
|
|
|
|
# --- Structural lint (no cluster): parse each test def and check the tag convention ----
|
|
lint_tests() {
|
|
log "Linting test definitions (structural, no cluster)"
|
|
local rc=0 f
|
|
if command -v chainsaw >/dev/null 2>&1; then
|
|
# Prefer chainsaw's own linter when the pinned binary is available.
|
|
while IFS= read -r f; do
|
|
chainsaw lint test --file "$f" >/dev/null || { echo "LINT FAIL: $f" >&2; rc=1; }
|
|
done < <(conf_all_test_files)
|
|
else
|
|
log "chainsaw not on PATH; falling back to structural check"
|
|
# Minimal YAML-shape check: kind Test + at least one requirement label per file.
|
|
while IFS= read -r f; do
|
|
grep -Eq '^kind:\s*Test' "$f" || { echo "LINT FAIL (no kind: Test): $f" >&2; rc=1; }
|
|
grep -Eq "^[[:space:]]*${CONF_LABEL_PREFIX}/R-[A-Z0-9]+-[0-9]+:" "$f" \
|
|
|| { echo "LINT FAIL (no requirement label): $f" >&2; rc=1; }
|
|
done < <(conf_all_test_files)
|
|
fi
|
|
return $rc
|
|
}
|
|
|
|
# --- Select which test directories Chainsaw should run --------------------------------
|
|
select_dirs() {
|
|
# --requirements AND --layer together => intersection (tests carrying the tag
|
|
# AND living in that layer subtree). Either alone keeps its prior meaning, so
|
|
# this is backward compatible (LAYER defaults to "all"). Both helpers emit
|
|
# sorted, de-duplicated dirs, so `comm -12` yields the intersection.
|
|
if [ -n "${REQS}" ] && [ "${LAYER}" != all ]; then
|
|
comm -12 \
|
|
<(conf_dirs_for_requirements "${REQS}") \
|
|
<(conf_dirs_for_layer "${LAYER}")
|
|
elif [ -n "${REQS}" ]; then
|
|
conf_dirs_for_requirements "${REQS}"
|
|
else
|
|
conf_dirs_for_layer "${LAYER}"
|
|
fi
|
|
}
|
|
|
|
# --- HEAVY tests: the storage/VM/DR tests whose KubeVirt-CDI imports + Longhorn volume ops
|
|
# spike embedded-etcd I/O on the small dev VMs. Run SERIALLY (parallel=1) in their own
|
|
# pass so their etcd spikes never coincide with ANOTHER test's setup — the coincidence
|
|
# that made a single heavy test's apiserver blip fail-close Kyverno admission and abort
|
|
# every in-flight test's 0.01s setup (see config/chainsaw-config.yaml). Keyed by the test
|
|
# DIRECTORY basename (stable); override via CONF_HEAVY_TESTS to add/remove without code
|
|
# changes. Listed here even for tests still scaffolded (vm-lifecycle, dr-restore, ...) so
|
|
# the serialization is already correct the moment they grow real KubeVirt/Longhorn bodies.
|
|
CONF_HEAVY_TESTS="${CONF_HEAVY_TESTS:-vm-lifecycle vm-data-survival vm-default-deny data-survival dr-restore node-loss}"
|
|
|
|
conf_is_heavy_dir() {
|
|
# $1 = test dir; true if its basename is in the heavy set.
|
|
local base t
|
|
base="$(basename "$1")"
|
|
for t in ${CONF_HEAVY_TESTS}; do [ "$base" = "$t" ] && return 0; done
|
|
return 1
|
|
}
|
|
|
|
# One Chainsaw invocation over a specific set of dirs, at a given parallelism, writing to a
|
|
# named report file. Returns chainsaw's rc (0=all passed) WITHOUT aborting run.sh (callers
|
|
# collect both passes' rc so a light-pass failure still lets the heavy pass run, and vice
|
|
# versa). failFast stays false (config) so every test in the pass runs regardless.
|
|
run_chainsaw_pass() {
|
|
local report_name="$1" parallel="$2"; shift 2
|
|
local -a dirs=("$@")
|
|
local -a test_dir_args=()
|
|
local d
|
|
for d in "${dirs[@]}"; do test_dir_args+=(--test-dir "$d"); done
|
|
# Chainsaw writes JUnit per config/chainsaw-config.yaml (report.* under _artifacts).
|
|
# --report-path is passed ABSOLUTE (overriding the config's relative `_artifacts`) so the
|
|
# report write succeeds regardless of the caller's CWD. --report-name/--parallel override
|
|
# the config per-pass so the two passes don't clobber each other's report and the heavy
|
|
# pass forces serial execution.
|
|
local rc=0
|
|
chainsaw test \
|
|
--config "${CONF_ROOT}/config/chainsaw-config.yaml" \
|
|
--report-path "${ARTIFACTS}" \
|
|
--report-name "${report_name}" \
|
|
--parallel "${parallel}" \
|
|
"${test_dir_args[@]}" || rc=$?
|
|
return $rc
|
|
}
|
|
|
|
# --- Execute the Chainsaw suite against a live cluster ---------------------------------
|
|
# Split into a LIGHT pass (concurrent, config parallelism) and a HEAVY pass (serial,
|
|
# parallel=1). Robust single-pass green: the heavy etcd spikes are isolated in time from the
|
|
# light tests' setup, so a heavy test's transient apiserver blip can no longer cascade into
|
|
# mass fail-closed setup-aborts of unrelated tests. The two JUnit reports are stitched back
|
|
# into the one stable artifact CI collects (report.junit.xml) via lib/merge_junit.py, so the
|
|
# split is invisible to everything downstream.
|
|
run_chainsaw() {
|
|
local -a dirs
|
|
mapfile -t dirs < <(select_dirs)
|
|
if [ "${#dirs[@]}" -eq 0 ]; then
|
|
log "No tests match the selection (layer=${LAYER} reqs='${REQS}')."
|
|
return 0
|
|
fi
|
|
|
|
local -a light=() heavy=()
|
|
local d
|
|
for d in "${dirs[@]}"; do
|
|
if conf_is_heavy_dir "$d"; then heavy+=("$d"); else light+=("$d"); fi
|
|
done
|
|
|
|
mkdir -p "${ARTIFACTS}"
|
|
local light_report="${ARTIFACTS}/chainsaw-report-light.xml"
|
|
local heavy_report="${ARTIFACTS}/chainsaw-report-heavy.xml"
|
|
rm -f "${light_report}" "${heavy_report}"
|
|
local light_rc=0 heavy_rc=0
|
|
|
|
# Determine the light pass's parallelism from the shared config (single source of truth),
|
|
# falling back to 2 if unreadable — so bumping the config on prod-class hw needs no run.sh edit.
|
|
local light_parallel
|
|
light_parallel="$(sed -nE 's/^[[:space:]]*parallel:[[:space:]]*([0-9]+).*/\1/p' \
|
|
"${CONF_ROOT}/config/chainsaw-config.yaml" | head -1)"
|
|
[ -n "${light_parallel}" ] || light_parallel=2
|
|
|
|
if [ "${#light[@]}" -gt 0 ]; then
|
|
log "LIGHT pass: ${#light[@]} test(s), parallel=${light_parallel} (pinned ${CHAINSAW_VERSION})"
|
|
run_chainsaw_pass "chainsaw-report-light" "${light_parallel}" "${light[@]}" || light_rc=$?
|
|
else
|
|
log "LIGHT pass: no tests selected."
|
|
fi
|
|
|
|
if [ "${#heavy[@]}" -gt 0 ]; then
|
|
log "HEAVY pass: ${#heavy[@]} test(s) SERIAL (parallel=1) — ${heavy[*]##*/}"
|
|
run_chainsaw_pass "chainsaw-report-heavy" 1 "${heavy[@]}" || heavy_rc=$?
|
|
else
|
|
log "HEAVY pass: no heavy tests selected."
|
|
fi
|
|
|
|
# Stitch both passes' JUnit into the one stable artifact CI collects.
|
|
"${PYTHON}" "${CONF_ROOT}/lib/merge_junit.py" \
|
|
--out "${JUNIT_OUT}" --name chainsaw-report \
|
|
"${light_report}" "${heavy_report}"
|
|
|
|
# Overall result is the union: fail if EITHER pass had a failing/erroring test. Real
|
|
# failures still surface (nothing is masked); the split only changes WHEN heavy tests run.
|
|
if [ "${light_rc}" -ne 0 ] || [ "${heavy_rc}" -ne 0 ]; then
|
|
log "Chainsaw reported failures (light rc=${light_rc}, heavy rc=${heavy_rc})."
|
|
return 1
|
|
fi
|
|
return 0
|
|
}
|
|
|
|
# --- L0/L1: the cluster-INDEPENDENT layers (docs/14) ------------------------------------
|
|
# docs/14 puts TWO layers in CI without a cluster:
|
|
# L0 static/policy tests/l0-static/** each dir carries a `check.sh` (repo invariants)
|
|
# L1 rendering tests/l1-rendering/** each dir carries a `render-check.sh` (render == golden)
|
|
# Both are executed directly here and emit a REAL pass/fail JUnit, instead of being skipped
|
|
# with the cluster-gated layers. Honors the same --layer/--requirements selection as the rest
|
|
# of the suite (select_dirs), filtered to those two subtrees.
|
|
OFFLINE_SUBTREES='/l0-static/|/l1-rendering/'
|
|
|
|
offline_selected_dirs() {
|
|
select_dirs | grep -E "${OFFLINE_SUBTREES}" || true
|
|
}
|
|
|
|
# True when EVERY requested layer is cluster-independent, so the offline result is the
|
|
# whole result and run.sh must exit on it (the ci/l0.sh + ci/l1.sh merge-gate paths).
|
|
layers_are_offline_only() {
|
|
local l
|
|
[ "${LAYER}" = all ] && return 1
|
|
for l in ${LAYER//,/ }; do
|
|
case "$l" in l0|l1) ;; *) return 1 ;; esac
|
|
done
|
|
return 0
|
|
}
|
|
|
|
run_offline_checks() {
|
|
local -a dirs
|
|
mapfile -t dirs < <(offline_selected_dirs)
|
|
if [ "${#dirs[@]}" -eq 0 ]; then
|
|
log "No L0/L1 offline checks match the selection (layer=${LAYER} reqs='${REQS}')."
|
|
return 0
|
|
fi
|
|
log "Running ${#dirs[@]} offline L0/L1 check(s) (no cluster; docs/14)."
|
|
"${PYTHON}" "${CONF_ROOT}/lib/run_l1_offline.py" \
|
|
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${dirs[@]}"
|
|
}
|
|
|
|
###############################################################################
|
|
# Main
|
|
###############################################################################
|
|
log "Coverage report first (always)"
|
|
gen_coverage
|
|
|
|
case "${MODE}" in
|
|
coverage-only)
|
|
log "coverage-only: done. See ${COVERAGE_OUT}/coverage.md"
|
|
exit 0
|
|
;;
|
|
status)
|
|
# rc: 0 report written, 1 a derivation/strict failure, 3 a check could not run
|
|
# (SKIPPED, never counted as green — ci/lib.sh convention). Propagated as-is.
|
|
rc=0
|
|
# --strict-coverage is the harness's hygiene switch; apply it to status too.
|
|
status_flags=()
|
|
[ "${STRICT_COVERAGE}" = 1 ] && status_flags+=(--strict)
|
|
gen_status ${status_flags[@]+"${status_flags[@]}"} || rc=$?
|
|
log "status: done. See ${STATUS_OUT}/status.md"
|
|
exit "${rc}"
|
|
;;
|
|
lint)
|
|
lint_tests
|
|
log "lint: done."
|
|
exit 0
|
|
;;
|
|
esac
|
|
|
|
# L0 static + L1 rendering are cluster-independent (docs/14): run the selected offline
|
|
# checks and treat their result as authoritative REGARDLESS of cluster availability. An
|
|
# offline check failure is a real failure (nothing cluster-gated about it), so it always blocks.
|
|
OFFLINE_RC=0
|
|
run_offline_checks || OFFLINE_RC=$?
|
|
if layers_are_offline_only; then
|
|
# Offline-only run (the ci/l0.sh + ci/l1.sh merge-gate paths): the offline result IS the result.
|
|
if [ "${OFFLINE_RC}" -eq 0 ]; then
|
|
log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
|
|
else
|
|
log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}"
|
|
fi
|
|
exit "${OFFLINE_RC}"
|
|
fi
|
|
if [ "${OFFLINE_RC}" -ne 0 ]; then
|
|
log "Offline L0/L1 checks FAILED (rc=${OFFLINE_RC}) — blocking (not cluster-gated)."
|
|
exit "${OFFLINE_RC}"
|
|
fi
|
|
|
|
# run mode: need a cluster for L2/L3. Without one (Phase 0), emit skipped JUnit for the
|
|
# cluster-gated layers. (L1 already ran offline above and passed.)
|
|
if [ "${FORCE_NO_CLUSTER}" = 1 ] || ! conf_cluster_reachable; then
|
|
log "No reachable cluster — emitting SKIPPED JUnit scaffold for L2/L3 (Phase 0 behavior)."
|
|
log "NOTE: the offline L0/L1 results above already ran for real; this shim covers L2/L3 only."
|
|
log "Also running structural lint so scaffolds stay valid."
|
|
lint_tests || { log "lint failed"; exit 1; }
|
|
"${PYTHON}" "${CONF_ROOT}/lib/emit_skipped_junit.py" \
|
|
--tests "${CONF_ROOT}/tests" --out "${JUNIT_OUT}"
|
|
log "Done (scaffold mode). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
|
|
exit 0
|
|
fi
|
|
|
|
command -v chainsaw >/dev/null 2>&1 || {
|
|
echo "ERROR: cluster reachable but 'chainsaw' (${CHAINSAW_VERSION}) not on PATH." >&2
|
|
echo "Install the pinned binary or run via the in-cluster Job (incluster/)." >&2
|
|
exit 3
|
|
}
|
|
run_chainsaw
|
|
log "Done. JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
|