From a775082ae1ce15c5812b75a3eac550f4bd53ec58 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 8 Nov 2026 00:18:57 -0600 Subject: [PATCH 1/2] feat(ci): run L0/L1 offline checks, mutation suites and yamllint in bounded parallel pools - conformance/lib/run_l1_offline.py: checks run in a thread pool (--jobs / CONF_OFFLINE_JOBS, default min(8, usable CPUs)); results consumed in submission order, so log lines, JUnit, step summary and the 0/1/3 aggregation are unchanged. Exports a smaller MUTATION_JOBS to the children when parallel, so nested pools do not oversubscribe. - keycloak-{passkey,client-secret}-reconciler/mutations.py: anchors checked first (serial), then the --selftest subprocesses run in a pool (MUTATION_JOBS, default min(8, CPUs)); verdicts reported in MUTATIONS order. Output byte-identical to the serial loop. - ci/lint.sh: yamllint over LINT_JOBS contiguous slices concurrently, per-slice output buffered and printed in order; stage rc = highest slice rc. Co-Authored-By: Claude Opus 5.5 --- ci/lint.sh | 40 +++++- conformance/lib/run_l1_offline.py | 117 ++++++++++++++++-- .../mutations.py | 110 ++++++++++++---- .../keycloak-passkey-reconciler/mutations.py | 110 ++++++++++++---- 4 files changed, 312 insertions(+), 65 deletions(-) diff --git a/ci/lint.sh b/ci/lint.sh index 39aaf86..70d3653 100755 --- a/ci/lint.sh +++ b/ci/lint.sh @@ -25,8 +25,44 @@ if [ "${#files[@]}" -eq 0 ]; then exit 0 fi -yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${files[@]}" -ok "yaml/lint passed (${#files[@]} files)" +# yamllint is single-threaded, so the file list is split into LINT_JOBS CONTIGUOUS slices of the +# sorted list, one yamllint per slice, run concurrently. Each slice's output goes to its own file +# and the files are printed in slice order afterwards: the log is the same text, in the same +# file order, as one serial `yamllint "${files[@]}"`, and a finding is never interleaved with +# another process's output (yamllint's standard format puts the FILE NAME on its own line above +# that file's `line:col` findings, so interleaving would misattribute them). The stage's exit +# code is the highest any slice returned (yamllint: 1 = error, 2 = warning under strict), so a +# single failing slice fails the stage exactly as the serial call did. LINT_JOBS=1 is serial. +lint_jobs="${LINT_JOBS:-$(n="$(nproc 2>/dev/null || echo 1)"; [ "$n" -gt 8 ] && n=8; echo "$n")}" +case "$lint_jobs" in ''|*[!0-9]*|0) err "LINT_JOBS must be a positive integer, got '${lint_jobs}'"; exit 2 ;; esac +[ "$lint_jobs" -gt "${#files[@]}" ] && lint_jobs="${#files[@]}" +lint_tmp="$(mktemp -d)" +trap 'rm -rf "$lint_tmp"' EXIT +per=$(( (${#files[@]} + lint_jobs - 1) / lint_jobs )) +pids=() +for (( i = 0; i < lint_jobs; i++ )); do + slice=("${files[@]:$(( i * per )):${per}}") + [ "${#slice[@]}" -gt 0 ] || break + ( rc=0 + yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${slice[@]}" \ + >"${lint_tmp}/${i}.out" 2>&1 || rc=$? + echo "$rc" >"${lint_tmp}/${i}.rc" ) & + pids+=("$!") +done +wait "${pids[@]}" || true +lint_rc=0 +for (( i = 0; i < ${#pids[@]}; i++ )); do + cat "${lint_tmp}/${i}.out" + # A slice that left no rc file died before yamllint returned: that is a failure, not a pass. + slice_rc="$(cat "${lint_tmp}/${i}.rc" 2>/dev/null || true)" + case "$slice_rc" in ''|*[!0-9]*) slice_rc=1 ;; esac + [ "$slice_rc" -gt "$lint_rc" ] && lint_rc="$slice_rc" +done +if [ "$lint_rc" -ne 0 ]; then + err "yaml/lint failed (yamllint exit ${lint_rc}; ${#files[@]} files in ${#pids[@]} parallel slice(s))" + exit "$lint_rc" +fi +ok "yaml/lint passed (${#files[@]} files, ${#pids[@]} parallel slice(s))" # A workflow's `configurationFile:` must name a file that exists. Renovate's did not after #79 # renamed renovate.json -> renovate.json5, and every scheduled run failed with ENOENT for 11 days diff --git a/conformance/lib/run_l1_offline.py b/conformance/lib/run_l1_offline.py index c39abf9..199c78c 100644 --- a/conformance/lib/run_l1_offline.py +++ b/conformance/lib/run_l1_offline.py @@ -43,13 +43,35 @@ Exit codes: 0 all selected checks passed / 1 at least one FAILED / 3 nothing fai scope-declared check was UNWIRED. conformance/run.sh propagates 3 verbatim; ci/l0.sh turns it into the gate's own rc 3. This is the ONE place the "unwired, not green" rule lives now, instead of once per hand-written stage. + +PARALLELISM (--jobs, env CONF_OFFLINE_JOBS) +------------------------------------------- +The checks are independent processes, so they run in a bounded thread pool (one `bash` child +per worker). Default: min(8, usable CPUs); `--jobs 1` is the old serial runner exactly. Each +check's stdout+stderr is CAPTURED and only reported after every check has finished, in the +ORDER THE DIRS WERE GIVEN, so the log, the JUnit and the step summary are byte-for-byte +independent of which check finished first (only the trailing "[N.Ns]" wall-time on each +status line varies). The 0/1/3 aggregation runs over the same ordered case list as before. + +Parallel-safety is a property of the CHECKS, not of this runner: every check must keep its +scratch state in a private mktemp/mkdtemp dir and must not write into the tree or a fixed +path. The audit that established that for the current set is recorded in +decisions/2026-11-08-parallel-l0-checks.md; a new check that needs shared state must say so. + +Nested pools: the keycloak reconciler checks run their own mutation suites in a pool +(mutations.py, env MUTATION_JOBS). When this runner is itself parallel it exports a SMALLER +MUTATION_JOBS (unless the caller already set one), so the two suites share the CPUs the +other jobs-2 checks are not using instead of each taking min(8, CPUs). See inner_jobs(). """ from __future__ import annotations import argparse +import os import re import subprocess import sys +import time +from concurrent.futures import ThreadPoolExecutor import xml.etree.ElementTree as ET from pathlib import Path from xml.dom import minidom @@ -97,6 +119,46 @@ def collect_meta(test_yaml: Path, prefix: str): return name, reqs +def usable_cpus() -> int: + """CPUs this process may run on (the cgroup/affinity view, not the host's count).""" + try: + return max(1, len(os.sched_getaffinity(0))) + except (AttributeError, OSError): + return max(1, os.cpu_count() or 1) + + +def positive_int(text: str) -> int: + n = int(text) + if n < 1: + raise argparse.ArgumentTypeError(f"must be >= 1, got {n}") + return n + + +def inner_jobs(cpus: int, outer: int) -> int: + """MUTATION_JOBS to hand the checks when this runner runs `outer` of them at once. + + Two checks (the keycloak client-secret and passkey reconcilers) fan out internally; the + other outer-2 workers are single processes. Give the two suites an equal share of the CPUs + those others leave free, never fewer than 2 each (a 1-wide pool would make the 40-mutation + passkey suite the critical path of the whole stage), never more than 8. Peak process count + is then about (outer-2) + 2*inner, i.e. <= cpus on any machine with cpus >= outer + 2, and + at worst ~1.25-1.5x cpus on a small runner — CPU-bound oversubscription of that size costs + latency per process, not throughput. + """ + if outer <= 1: + return min(8, cpus) + return max(2, min(8, (cpus - outer + 2) // 2)) + + +def run_check(check: Path, env: dict) -> tuple[int, str, float]: + t0 = time.monotonic() + proc = subprocess.run( + ["bash", str(check)], capture_output=True, text=True, + stdin=subprocess.DEVNULL, env=env, + ) + return proc.returncode, (proc.stdout + proc.stderr).strip(), time.monotonic() - t0 + + def main() -> int: ap = argparse.ArgumentParser(description=__doc__) ap.add_argument("--junit", type=Path, required=True) @@ -108,26 +170,54 @@ def main() -> int: "(makes this runner exit 3); repeatable") ap.add_argument("--summary-out", type=Path, default=None, help="write one Markdown line per case here, for the CI step summary") + ap.add_argument("--jobs", type=positive_int, default=None, + help="checks to run at once (default: env CONF_OFFLINE_JOBS, else " + "min(8, usable CPUs)); 1 = serial") ap.add_argument("dirs", nargs="+", type=Path) args = ap.parse_args() strict_scopes = set(args.unwired_scope) - cases = [] # (name, reqs, dir, rc, output, scope) + cpus = usable_cpus() + if args.jobs is not None: + jobs = args.jobs + elif os.environ.get("CONF_OFFLINE_JOBS", "").strip(): + try: + jobs = positive_int(os.environ["CONF_OFFLINE_JOBS"].strip()) + except (ValueError, argparse.ArgumentTypeError) as e: + ap.error(f"CONF_OFFLINE_JOBS={os.environ['CONF_OFFLINE_JOBS']!r}: {e}") + else: + jobs = min(8, cpus) + child_env = dict(os.environ) + if not child_env.get("MUTATION_JOBS", "").strip(): + child_env["MUTATION_JOBS"] = str(inner_jobs(cpus, jobs)) + print(f"[offline] running {len(args.dirs)} check(s), jobs={jobs} " + f"(cpus={cpus}, MUTATION_JOBS={child_env['MUTATION_JOBS']})", file=sys.stderr) + + # Metadata first, in the given order; then the checks themselves through the pool. The + # results are consumed in SUBMISSION order, so `cases` is identical to the serial runner's. + planned = [] # (name, reqs, dir, scope, check-or-None) for d in args.dirs: check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None) name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix) scope = scope_of(d / "chainsaw-test.yaml", args.scope_annotation) - if check is None: - cases.append((name, reqs, d, 3, - f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}", - scope)) - continue - proc = subprocess.run( - ["bash", str(check)], capture_output=True, text=True - ) - out = (proc.stdout + proc.stderr).strip() - cases.append((name, reqs, d, proc.returncode, out, scope)) + planned.append((name, reqs, d, scope, check)) + + cases = [] # (name, reqs, dir, rc, output, scope) + elapsed = [] # wall seconds per case, same order (None: nothing ran) + with ThreadPoolExecutor(max_workers=jobs) as pool: + futures = [pool.submit(run_check, check, child_env) if check is not None else None + for (_n, _r, _d, _s, check) in planned] + for (name, reqs, d, scope, check), fut in zip(planned, futures): + if fut is None: + cases.append((name, reqs, d, 3, + f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}", + scope)) + elapsed.append(None) + continue + rc, out, secs = fut.result() + cases.append((name, reqs, d, rc, out, scope)) + elapsed.append(secs) failures = sum(1 for c in cases if c[3] not in (0, 3)) skipped = sum(1 for c in cases if c[3] == 3) @@ -145,7 +235,7 @@ def main() -> int: }, ) summary_lines = [] - for name, reqs, d, rc, out, scope in cases: + for (name, reqs, d, rc, out, scope), secs in zip(cases, elapsed): tc = ET.SubElement( suite, "testcase", @@ -164,8 +254,9 @@ def main() -> int: fail.text = out status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL") mark = f" scope={scope}" if scope else "" + took = f" [{secs:.1f}s]" if secs is not None else "" print(f"[offline] {layer_of(d):3s} {status:4s} {name} " - f"({','.join(reqs) or 'untagged'}){mark}", + f"({','.join(reqs) or 'untagged'}){mark}{took}", file=sys.stderr) if rc not in (0, 3): print(out, file=sys.stderr) diff --git a/conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py b/conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py index cbc99f3..545535f 100755 --- a/conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py +++ b/conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py @@ -25,15 +25,19 @@ # # python3 mutations.py # -# Runtime is ~20s (one subprocess per mutation, each running the full selftest -# against an in-process stub Keycloak; no network, no cluster). +# Runtime is ~11s serial (one subprocess per mutation, each running the full selftest +# against an in-process stub Keycloak; no network, no cluster). The selftests run in a +# bounded pool (env MUTATION_JOBS, default min(8, CPUs); 1 = serial); output order and +# verdicts are independent of the pool size. # ============================================================================= from __future__ import annotations +import os import re import subprocess import sys import tempfile +from concurrent.futures import ThreadPoolExecutor from pathlib import Path @@ -136,43 +140,99 @@ MUTATIONS: list[tuple[str, str, str, str]] = [ ] +def mutation_jobs() -> int: + """Selftests to run at once: env MUTATION_JOBS, else min(8, usable CPUs). + + conformance/lib/run_l1_offline.py exports a smaller MUTATION_JOBS when it is itself + running several checks in parallel, so the nested pools do not oversubscribe the host. + MUTATION_JOBS=1 reproduces the old serial loop exactly. + """ + raw = os.environ.get("MUTATION_JOBS", "").strip() + if raw: + try: + n = int(raw) + except ValueError: + n = 0 + if n < 1: + raise ValueError("MUTATION_JOBS must be a positive integer, got %r" % raw) + return n + try: + cpus = len(os.sched_getaffinity(0)) + except (AttributeError, OSError): + cpus = os.cpu_count() or 1 + return max(1, min(8, cpus)) + + +def run_selftest(mutated_src: str) -> subprocess.CompletedProcess: + with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh: + fh.write(mutated_src) + path = fh.name + try: + return subprocess.run([sys.executable, path, "--selftest"], + capture_output=True, text=True, timeout=240, + stdin=subprocess.DEVNULL) + finally: + Path(path).unlink(missing_ok=True) + + def main() -> int: if len(sys.argv) != 2: print("usage: mutations.py ", file=sys.stderr) return 2 src = Path(sys.argv[1]).read_text() - problems: list[str] = [] - killed = 0 + try: + jobs = mutation_jobs() + except ValueError as e: + print("mutations.py: %s" % e, file=sys.stderr) + return 2 + + # Pass 1, serial and cheap: every anchor is checked BEFORE anything runs, exactly as the + # serial loop did, and each mutation's source is built here. None = anchor failure. + planned: list[tuple[str, str, str | None, str | None]] = [] for name, why, old, new in MUTATIONS: old, new = dedent4(old), dedent4(new) # An anchor that no longer matches is a FAILURE, not a skip: it means the code # moved and this mutation silently stopped proving anything. n = src.count(old) if n != 1: - problems.append("%s: anchor matches %d times (expected exactly 1) -- the " + planned.append((name, why, + "%s: anchor matches %d times (expected exactly 1) -- the " "code moved and this mutation no longer proves anything. " - "Re-anchor it against the current source." % (name, n)) + "Re-anchor it against the current source." % (name, n), None)) continue - with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh: - fh.write(src.replace(old, new, 1)) - path = fh.name - try: - proc = subprocess.run([sys.executable, path, "--selftest"], - capture_output=True, text=True, timeout=240) - finally: - Path(path).unlink(missing_ok=True) - if proc.returncode == 0: - problems.append("%s SURVIVED -- --selftest still passed with this regression " - "applied, so nothing in the suite actually checks it. The " - "mistake it models: %s" % (name, why)) - continue - killed += 1 - out = proc.stdout + proc.stderr - m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out) - how = ("%s assertion(s) caught it" % m.group(1) if m - else "the selftest aborted (caught, less cleanly)") - print(" killed %-42s %s" % (name, how)) + planned.append((name, why, None, src.replace(old, new, 1))) + + # Pass 2: the selftests, `jobs` at a time. Each child gets its own temp file, its own + # mkdtemp dirs and its own 127.0.0.1:0 stub port, so they share nothing. Results are + # consumed in MUTATIONS order, so every line below is printed in the same order, with the + # same verdict, as the serial loop produced -- whichever child finishes first. + problems: list[str] = [] + killed = 0 + pool = ThreadPoolExecutor(max_workers=jobs) + try: + futures = [pool.submit(run_selftest, mutated) if mutated is not None else None + for (_n, _w, _a, mutated) in planned] + for (name, why, anchor_problem, _m), fut in zip(planned, futures): + if anchor_problem is not None: + problems.append(anchor_problem) + continue + proc = fut.result() + if proc.returncode == 0: + problems.append("%s SURVIVED -- --selftest still passed with this regression " + "applied, so nothing in the suite actually checks it. The " + "mistake it models: %s" % (name, why)) + continue + killed += 1 + out = proc.stdout + proc.stderr + m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out) + how = ("%s assertion(s) caught it" % m.group(1) if m + else "the selftest aborted (caught, less cleanly)") + print(" killed %-42s %s" % (name, how)) + finally: + # A timeout (or any other exception) still propagates and fails the check, as it did + # serially; queued mutations are cancelled rather than run for nothing. + pool.shutdown(wait=True, cancel_futures=True) print() if problems: diff --git a/conformance/tests/l0-static/keycloak-passkey-reconciler/mutations.py b/conformance/tests/l0-static/keycloak-passkey-reconciler/mutations.py index 53ba2c8..a231fb7 100644 --- a/conformance/tests/l0-static/keycloak-passkey-reconciler/mutations.py +++ b/conformance/tests/l0-static/keycloak-passkey-reconciler/mutations.py @@ -40,15 +40,19 @@ # # python3 mutations.py # -# Runtime is ~60s (one subprocess per mutation, each running the full selftest -# against an in-process stub Keycloak; no network, no cluster). +# Runtime is ~85s serial (one subprocess per mutation, each running the full selftest +# against an in-process stub Keycloak; no network, no cluster). The selftests run in a +# bounded pool (env MUTATION_JOBS, default min(8, CPUs); 1 = serial); output order and +# verdicts are independent of the pool size. # ============================================================================= from __future__ import annotations +import os import re import subprocess import sys import tempfile +from concurrent.futures import ThreadPoolExecutor from pathlib import Path # (name, the mistake being modelled, anchor, replacement) @@ -403,42 +407,98 @@ MUTATIONS: list[tuple[str, str, str, str]] = [ ] +def mutation_jobs() -> int: + """Selftests to run at once: env MUTATION_JOBS, else min(8, usable CPUs). + + conformance/lib/run_l1_offline.py exports a smaller MUTATION_JOBS when it is itself + running several checks in parallel, so the nested pools do not oversubscribe the host. + MUTATION_JOBS=1 reproduces the old serial loop exactly. + """ + raw = os.environ.get("MUTATION_JOBS", "").strip() + if raw: + try: + n = int(raw) + except ValueError: + n = 0 + if n < 1: + raise ValueError("MUTATION_JOBS must be a positive integer, got %r" % raw) + return n + try: + cpus = len(os.sched_getaffinity(0)) + except (AttributeError, OSError): + cpus = os.cpu_count() or 1 + return max(1, min(8, cpus)) + + +def run_selftest(mutated_src: str) -> subprocess.CompletedProcess: + with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh: + fh.write(mutated_src) + path = fh.name + try: + return subprocess.run([sys.executable, path, "--selftest"], + capture_output=True, text=True, timeout=300, + stdin=subprocess.DEVNULL) + finally: + Path(path).unlink(missing_ok=True) + + def main() -> int: if len(sys.argv) != 2: print("usage: mutations.py ", file=sys.stderr) return 2 src = Path(sys.argv[1]).read_text() - problems: list[str] = [] - killed = 0 + try: + jobs = mutation_jobs() + except ValueError as e: + print("mutations.py: %s" % e, file=sys.stderr) + return 2 + + # Pass 1, serial and cheap: every anchor is checked BEFORE anything runs, exactly as the + # serial loop did, and each mutation's source is built here. None = anchor failure. + planned: list[tuple[str, str, str | None, str | None]] = [] for name, why, old, new in MUTATIONS: # An anchor that no longer matches is a FAILURE, not a skip: it means the code # moved and this mutation silently stopped proving anything. n = src.count(old) if n != 1: - problems.append("%s: anchor matches %d times (expected exactly 1) -- the " + planned.append((name, why, + "%s: anchor matches %d times (expected exactly 1) -- the " "code moved and this mutation no longer proves anything. " - "Re-anchor it against the current source." % (name, n)) + "Re-anchor it against the current source." % (name, n), None)) continue - with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh: - fh.write(src.replace(old, new, 1)) - path = fh.name - try: - proc = subprocess.run([sys.executable, path, "--selftest"], - capture_output=True, text=True, timeout=300) - finally: - Path(path).unlink(missing_ok=True) - if proc.returncode == 0: - problems.append("%s SURVIVED -- --selftest still passed with this regression " - "applied, so nothing in the suite actually checks it. The " - "mistake it models: %s" % (name, why)) - continue - killed += 1 - out = proc.stdout + proc.stderr - m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out) - how = ("%s assertion(s) caught it" % m.group(1) if m - else "the selftest aborted (caught, less cleanly)") - print(" killed %-54s %s" % (name, how)) + planned.append((name, why, None, src.replace(old, new, 1))) + + # Pass 2: the selftests, `jobs` at a time. Each child gets its own temp file, its own + # mkdtemp dirs and its own 127.0.0.1:0 stub port, so they share nothing. Results are + # consumed in MUTATIONS order, so every line below is printed in the same order, with the + # same verdict, as the serial loop produced -- whichever child finishes first. + problems: list[str] = [] + killed = 0 + pool = ThreadPoolExecutor(max_workers=jobs) + try: + futures = [pool.submit(run_selftest, mutated) if mutated is not None else None + for (_n, _w, _a, mutated) in planned] + for (name, why, anchor_problem, _m), fut in zip(planned, futures): + if anchor_problem is not None: + problems.append(anchor_problem) + continue + proc = fut.result() + if proc.returncode == 0: + problems.append("%s SURVIVED -- --selftest still passed with this regression " + "applied, so nothing in the suite actually checks it. The " + "mistake it models: %s" % (name, why)) + continue + killed += 1 + out = proc.stdout + proc.stderr + m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out) + how = ("%s assertion(s) caught it" % m.group(1) if m + else "the selftest aborted (caught, less cleanly)") + print(" killed %-54s %s" % (name, how)) + finally: + # A timeout (or any other exception) still propagates and fails the check, as it did + # serially; queued mutations are cancelled rather than run for nothing. + pool.shutdown(wait=True, cancel_futures=True) print() if problems: -- 2.54.0 From 89f48e4a5f6d0110aed52bcf54427b9ad70396f8 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 8 Nov 2026 00:34:48 -0600 Subject: [PATCH 2/2] docs(decisions): handoff for parallel L0 checks; runner usage names --jobs Co-Authored-By: Claude Opus 5.5 --- conformance/lib/run_l1_offline.py | 2 +- decisions/2026-11-08-parallel-l0-checks.md | 116 +++++++++++++++++++++ 2 files changed, 117 insertions(+), 1 deletion(-) create mode 100644 decisions/2026-11-08-parallel-l0-checks.md diff --git a/conformance/lib/run_l1_offline.py b/conformance/lib/run_l1_offline.py index 199c78c..1e4ba6a 100644 --- a/conformance/lib/run_l1_offline.py +++ b/conformance/lib/run_l1_offline.py @@ -14,7 +14,7 @@ Pure standard library (matches the harness's no-pip-dependency posture); the che themselves shell out (render checks to catalog/render.py, whose PyYAML dependency lives there; L0 checks to git + PyYAML). -Usage: run_l1_offline.py --junit [--label-prefix P] [ ...] +Usage: run_l1_offline.py --junit [--label-prefix P] [--jobs N] [ ...] Exit 0 iff every selected offline check passes. The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory diff --git a/decisions/2026-11-08-parallel-l0-checks.md b/decisions/2026-11-08-parallel-l0-checks.md new file mode 100644 index 0000000..d2808e2 --- /dev/null +++ b/decisions/2026-11-08-parallel-l0-checks.md @@ -0,0 +1,116 @@ +--- +date: 2026-11-08 +branch: feat/parallel-l0-checks +title: L0 offline checks, mutation suites and yamllint run in bounded parallel pools (gate 184s -> 32s locally) +--- + +## What changed + +The gate's wall time was dominated by serial loops over independent subprocesses. Three of +them now run in bounded pools; none of them changes a verdict, an exit code, or the order of +anything printed. + +- `conformance/lib/run_l1_offline.py` — the per-check loop (L0 and L1 offline checks) runs in + a `ThreadPoolExecutor` of `bash ` children. `--jobs N` / env `CONF_OFFLINE_JOBS`, + default `min(8, usable CPUs)` (`sched_getaffinity`, so a cgroup cpuset is respected); + `--jobs 1` is the old serial runner. Results are consumed in SUBMISSION order, so `cases`, + the JUnit, the step summary and the 0/1/3 aggregation are built from exactly the same + ordered list as before. Additive output only: one `[offline] running N check(s), jobs=…` + line, and a trailing `[N.Ns]` wall time on each per-check status line. Children get + `stdin=/dev/null` (parallel children must not share the runner's stdin). +- `conformance/tests/l0-static/keycloak-{passkey,client-secret}-reconciler/mutations.py` — + pass 1 (serial) checks every anchor and builds each mutated source, exactly as before; + pass 2 runs the `--selftest` children in a pool (env `MUTATION_JOBS`, default + `min(8, CPUs)`, `1` = serial). Verdicts are consumed in `MUTATIONS` order. A timeout + (`TimeoutExpired`) still propagates and fails the check; queued mutations are cancelled. + An invalid `MUTATION_JOBS` exits 2 (fails the check, never skips it). +- **Nested-pool budget.** When the outer runner is parallel it exports + `MUTATION_JOBS = max(2, min(8, (cpus - jobs + 2) // 2))` unless the caller already set + one. Reason: only those two checks fan out; the other `jobs-2` workers are single + processes, so the two suites split the CPUs those leave free. Peak processes ≈ + `(jobs-2) + 2*inner` ≤ cpus whenever cpus ≥ jobs+2 (24 CPUs: 6 + 16 = 22), and ~1.25–1.5× + cpus on a small runner (4 CPUs: 2 + 4 = 6). The floor of 2 is deliberate: a 1-wide inner + pool would make the 40-mutation passkey suite (~85s serial) the critical path of the stage. +- `ci/lint.sh` — yamllint runs over `LINT_JOBS` (default `min(8, nproc)`) CONTIGUOUS slices + of the sorted file list concurrently; each slice's output is buffered to its own file and + printed in slice order, so the text is the same as one serial call and a `file` header is + never separated from its `line:col` findings. Stage rc = the highest slice rc; a slice that + leaves no rc file counts as 1. Invalid `LINT_JOBS` exits 2. + +### Parallel-safety audit (all 25 `l0-static/*/check.sh`, 3 `l1-rendering/*/render-check.sh`) + +Grepped every check for fixed `/tmp` paths, `_artifacts`/`ARTIFACTS`/`COVERAGE_OUT`, +`write_text`/`open(…,"w")`, `mktemp`/`mkdtemp`/`TemporaryDirectory`, sockets/ports, `git` +writes and `run.sh` invocations, and read what each writer writes to: + +- **Scratch state: all private.** Every writer uses its own `mktemp -d` / `mkdtemp` / + `TemporaryDirectory` (placeholder-guard-coupling, spec-immutability-enforced, + whole-repo-invariants-wired, auditability-proofs-producible, alert-rules-unit-tested and its + `platform/monitoring/tests/run.sh`, internal-services-publish-no-public-dns, + oauth2-proxy-{api-routes-per-app,cookie-names-distinct}, both keycloak reconcilers, the three + L1 render checks). No fixed `/tmp` path anywhere. +- **Git writes:** only spec-immutability-enforced, and only `git init/add/commit` inside its + own `mktemp` repo. Everything else is `git ls-files`/`ls-tree`/`log`/`rev-parse` (read-only). +- **Ports:** the reconciler selftests bind `127.0.0.1:0` (kernel-assigned); the passkey + selftest also dials `127.0.0.1:1` as a deliberate refused connection. No fixed listener. +- **Ansible** (dev-nodes-provisioned-as-code): `--syntax-check`/`--list-hosts` only, + `retry_files_enabled = False`; its local tmp is per-process. +- **The one shared writer:** auditability-proofs-producible runs + `./run.sh --coverage-only --requirements …`, which REWRITES + `conformance/_artifacts/coverage/coverage.{json,md}` (it inherits `CONF_ARTIFACTS`). That was + already true serially. No other offline check reads or writes that directory (the only + reader is `tests/meta/harness-selfcheck`, not an offline check), and the outer runner + writes coverage BEFORE and the JUnit AFTER the pool, so the final state is the same as + serial. Not serialised; left as is and recorded here. `lib/status.py` (run by that check) + reads `_artifacts/report.junit.xml` as an advisory — written only after the pool, unchanged. +- whole-repo-invariants-wired calls `run.sh --print-selection` (writes nothing) and the + runner itself against synthetic dirs in its own `mktemp`. + +Result: no check needed fixing or serialising. A new check that needs shared state must say +so; the runner's docstring now states that contract. + +## What I verified + +All in the pinned toolbox (`dev.sh run`, `DEV_KUBECONFIG=/dev/null`, ENGINE shim adding a ro +mount of the main checkout's `.git`; spec repo resolved as a sibling, spec-immutability +`pass`), host 24 CPUs, every `dev.sh` call under `flock`. + +- **Gate, before** (origin/main `1a33814`, detached scratch worktree + spec mount): + `GATE_RC=3`, in-container wall **183.9s**; `l0` stage **168.3s**, `lint` **7.8s**. + "Stages that did not run: gitea-settings" (plus the expected on-base + touched-set-not-determinable note, since main has no branch commits). +- **Gate, after** (this branch, commit `a775082`): `GATE_RC=3`, in-container wall **32.1s**; + `l0` **21.1s**, `lint` **2.4s**, `l1` 2.3s. "Stages that did not run: gitea-settings" — the + only not-run stage. **rc 3, not green.** Caveat: the branch run selected 23 L0 checks (the + touched-requirement scoping dropped dev-nodes-provisioned-as-code and + placeholder-guard-coupling), the main run 25. +- **Same-tree L0 comparison** (all 25 l0 dirs, same tree, same container): + `CONF_OFFLINE_JOBS=1 MUTATION_JOBS=1` **164.0s / 165.3s** vs default (jobs=8, + MUTATION_JOBS=8) **19.2s / 18.7s**. JUnit `cmp`-identical, step summary `cmp`-identical, + stderr identical except the timing suffixes and the JUnit path in the last line. +- **Mutation suites, host:** passkey 83.2s → 10.5s (`MUTATION_JOBS=8`), client-secret + 11.2s → 2.0s; output of `MUTATION_JOBS=1` and `=8` byte-identical to the pre-change script's, + rc 0 all round. +- **Falsification (not committed):** in a scratch worktree at `a775082`, made the passkey + mutation "the protected-subtree screen is never called" a no-op (replacement == anchor). + Host: `MUTATION_JOBS=1` and `=8` both rc 1, output byte-identical (with a second, broken + anchor also injected: both reported). Toolbox: runner rc 1 both serial and parallel, + `L0 FAIL keycloak-passkey-reconciler … 1 of 40 mutation(s) not proven … SURVIVED`; full + gate **rc 1**, `stage l0: FAIL`, `MERGE GATE: BLOCKED`. Reverted; scratch worktree removed. +- **lint.sh, host scratch repo** (31 YAML files, two broken): `LINT_JOBS=1/4/8` all rc 1, + stdout identical, findings named `y/f07.yaml` / `2:1 … key-duplicates` etc.; clean tree rc 0; + `LINT_JOBS=x` rc 2. + +## What I did NOT verify + +- **CI timings at the time of writing.** The PR's CI run is measured after push and reported + on the PR / handoff message, not here. The CI runner's CPU count is UNVERIFIED — the new + `[offline] running … jobs=… cpus=…` line in the CI log is what will show it. On a small + runner the speedup will be smaller than locally. +- **approval-policy** will be red on CI: `ci/**` and `conformance/**` are sensitive paths and + need the operator's review. I did not and must not override it. +- Flakiness under load: the selftests have no wall-clock assertions I found, and 4 toolbox + runs + several host runs were all stable, but that is a handful of runs, not a soak. +- The coverage files under `conformance/_artifacts/coverage/` end in the same state as before + only because a single check rewrites them; I did not change that pre-existing quirk. +- L2/L3 conformance: not run (no cluster in CI, as before). -- 2.54.0