From 89f48e4a5f6d0110aed52bcf54427b9ad70396f8 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 8 Nov 2026 00:34:48 -0600 Subject: [PATCH] docs(decisions): handoff for parallel L0 checks; runner usage names --jobs Co-Authored-By: Claude Opus 5.5 --- conformance/lib/run_l1_offline.py | 2 +- decisions/2026-11-08-parallel-l0-checks.md | 116 +++++++++++++++++++++ 2 files changed, 117 insertions(+), 1 deletion(-) create mode 100644 decisions/2026-11-08-parallel-l0-checks.md diff --git a/conformance/lib/run_l1_offline.py b/conformance/lib/run_l1_offline.py index 199c78c..1e4ba6a 100644 --- a/conformance/lib/run_l1_offline.py +++ b/conformance/lib/run_l1_offline.py @@ -14,7 +14,7 @@ Pure standard library (matches the harness's no-pip-dependency posture); the che themselves shell out (render checks to catalog/render.py, whose PyYAML dependency lives there; L0 checks to git + PyYAML). -Usage: run_l1_offline.py --junit [--label-prefix P] [ ...] +Usage: run_l1_offline.py --junit [--label-prefix P] [--jobs N] [ ...] Exit 0 iff every selected offline check passes. The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory diff --git a/decisions/2026-11-08-parallel-l0-checks.md b/decisions/2026-11-08-parallel-l0-checks.md new file mode 100644 index 0000000..d2808e2 --- /dev/null +++ b/decisions/2026-11-08-parallel-l0-checks.md @@ -0,0 +1,116 @@ +--- +date: 2026-11-08 +branch: feat/parallel-l0-checks +title: L0 offline checks, mutation suites and yamllint run in bounded parallel pools (gate 184s -> 32s locally) +--- + +## What changed + +The gate's wall time was dominated by serial loops over independent subprocesses. Three of +them now run in bounded pools; none of them changes a verdict, an exit code, or the order of +anything printed. + +- `conformance/lib/run_l1_offline.py` — the per-check loop (L0 and L1 offline checks) runs in + a `ThreadPoolExecutor` of `bash ` children. `--jobs N` / env `CONF_OFFLINE_JOBS`, + default `min(8, usable CPUs)` (`sched_getaffinity`, so a cgroup cpuset is respected); + `--jobs 1` is the old serial runner. Results are consumed in SUBMISSION order, so `cases`, + the JUnit, the step summary and the 0/1/3 aggregation are built from exactly the same + ordered list as before. Additive output only: one `[offline] running N check(s), jobs=…` + line, and a trailing `[N.Ns]` wall time on each per-check status line. Children get + `stdin=/dev/null` (parallel children must not share the runner's stdin). +- `conformance/tests/l0-static/keycloak-{passkey,client-secret}-reconciler/mutations.py` — + pass 1 (serial) checks every anchor and builds each mutated source, exactly as before; + pass 2 runs the `--selftest` children in a pool (env `MUTATION_JOBS`, default + `min(8, CPUs)`, `1` = serial). Verdicts are consumed in `MUTATIONS` order. A timeout + (`TimeoutExpired`) still propagates and fails the check; queued mutations are cancelled. + An invalid `MUTATION_JOBS` exits 2 (fails the check, never skips it). +- **Nested-pool budget.** When the outer runner is parallel it exports + `MUTATION_JOBS = max(2, min(8, (cpus - jobs + 2) // 2))` unless the caller already set + one. Reason: only those two checks fan out; the other `jobs-2` workers are single + processes, so the two suites split the CPUs those leave free. Peak processes ≈ + `(jobs-2) + 2*inner` ≤ cpus whenever cpus ≥ jobs+2 (24 CPUs: 6 + 16 = 22), and ~1.25–1.5× + cpus on a small runner (4 CPUs: 2 + 4 = 6). The floor of 2 is deliberate: a 1-wide inner + pool would make the 40-mutation passkey suite (~85s serial) the critical path of the stage. +- `ci/lint.sh` — yamllint runs over `LINT_JOBS` (default `min(8, nproc)`) CONTIGUOUS slices + of the sorted file list concurrently; each slice's output is buffered to its own file and + printed in slice order, so the text is the same as one serial call and a `file` header is + never separated from its `line:col` findings. Stage rc = the highest slice rc; a slice that + leaves no rc file counts as 1. Invalid `LINT_JOBS` exits 2. + +### Parallel-safety audit (all 25 `l0-static/*/check.sh`, 3 `l1-rendering/*/render-check.sh`) + +Grepped every check for fixed `/tmp` paths, `_artifacts`/`ARTIFACTS`/`COVERAGE_OUT`, +`write_text`/`open(…,"w")`, `mktemp`/`mkdtemp`/`TemporaryDirectory`, sockets/ports, `git` +writes and `run.sh` invocations, and read what each writer writes to: + +- **Scratch state: all private.** Every writer uses its own `mktemp -d` / `mkdtemp` / + `TemporaryDirectory` (placeholder-guard-coupling, spec-immutability-enforced, + whole-repo-invariants-wired, auditability-proofs-producible, alert-rules-unit-tested and its + `platform/monitoring/tests/run.sh`, internal-services-publish-no-public-dns, + oauth2-proxy-{api-routes-per-app,cookie-names-distinct}, both keycloak reconcilers, the three + L1 render checks). No fixed `/tmp` path anywhere. +- **Git writes:** only spec-immutability-enforced, and only `git init/add/commit` inside its + own `mktemp` repo. Everything else is `git ls-files`/`ls-tree`/`log`/`rev-parse` (read-only). +- **Ports:** the reconciler selftests bind `127.0.0.1:0` (kernel-assigned); the passkey + selftest also dials `127.0.0.1:1` as a deliberate refused connection. No fixed listener. +- **Ansible** (dev-nodes-provisioned-as-code): `--syntax-check`/`--list-hosts` only, + `retry_files_enabled = False`; its local tmp is per-process. +- **The one shared writer:** auditability-proofs-producible runs + `./run.sh --coverage-only --requirements …`, which REWRITES + `conformance/_artifacts/coverage/coverage.{json,md}` (it inherits `CONF_ARTIFACTS`). That was + already true serially. No other offline check reads or writes that directory (the only + reader is `tests/meta/harness-selfcheck`, not an offline check), and the outer runner + writes coverage BEFORE and the JUnit AFTER the pool, so the final state is the same as + serial. Not serialised; left as is and recorded here. `lib/status.py` (run by that check) + reads `_artifacts/report.junit.xml` as an advisory — written only after the pool, unchanged. +- whole-repo-invariants-wired calls `run.sh --print-selection` (writes nothing) and the + runner itself against synthetic dirs in its own `mktemp`. + +Result: no check needed fixing or serialising. A new check that needs shared state must say +so; the runner's docstring now states that contract. + +## What I verified + +All in the pinned toolbox (`dev.sh run`, `DEV_KUBECONFIG=/dev/null`, ENGINE shim adding a ro +mount of the main checkout's `.git`; spec repo resolved as a sibling, spec-immutability +`pass`), host 24 CPUs, every `dev.sh` call under `flock`. + +- **Gate, before** (origin/main `1a33814`, detached scratch worktree + spec mount): + `GATE_RC=3`, in-container wall **183.9s**; `l0` stage **168.3s**, `lint` **7.8s**. + "Stages that did not run: gitea-settings" (plus the expected on-base + touched-set-not-determinable note, since main has no branch commits). +- **Gate, after** (this branch, commit `a775082`): `GATE_RC=3`, in-container wall **32.1s**; + `l0` **21.1s**, `lint` **2.4s**, `l1` 2.3s. "Stages that did not run: gitea-settings" — the + only not-run stage. **rc 3, not green.** Caveat: the branch run selected 23 L0 checks (the + touched-requirement scoping dropped dev-nodes-provisioned-as-code and + placeholder-guard-coupling), the main run 25. +- **Same-tree L0 comparison** (all 25 l0 dirs, same tree, same container): + `CONF_OFFLINE_JOBS=1 MUTATION_JOBS=1` **164.0s / 165.3s** vs default (jobs=8, + MUTATION_JOBS=8) **19.2s / 18.7s**. JUnit `cmp`-identical, step summary `cmp`-identical, + stderr identical except the timing suffixes and the JUnit path in the last line. +- **Mutation suites, host:** passkey 83.2s → 10.5s (`MUTATION_JOBS=8`), client-secret + 11.2s → 2.0s; output of `MUTATION_JOBS=1` and `=8` byte-identical to the pre-change script's, + rc 0 all round. +- **Falsification (not committed):** in a scratch worktree at `a775082`, made the passkey + mutation "the protected-subtree screen is never called" a no-op (replacement == anchor). + Host: `MUTATION_JOBS=1` and `=8` both rc 1, output byte-identical (with a second, broken + anchor also injected: both reported). Toolbox: runner rc 1 both serial and parallel, + `L0 FAIL keycloak-passkey-reconciler … 1 of 40 mutation(s) not proven … SURVIVED`; full + gate **rc 1**, `stage l0: FAIL`, `MERGE GATE: BLOCKED`. Reverted; scratch worktree removed. +- **lint.sh, host scratch repo** (31 YAML files, two broken): `LINT_JOBS=1/4/8` all rc 1, + stdout identical, findings named `y/f07.yaml` / `2:1 … key-duplicates` etc.; clean tree rc 0; + `LINT_JOBS=x` rc 2. + +## What I did NOT verify + +- **CI timings at the time of writing.** The PR's CI run is measured after push and reported + on the PR / handoff message, not here. The CI runner's CPU count is UNVERIFIED — the new + `[offline] running … jobs=… cpus=…` line in the CI log is what will show it. On a small + runner the speedup will be smaller than locally. +- **approval-policy** will be red on CI: `ci/**` and `conformance/**` are sensitive paths and + need the operator's review. I did not and must not override it. +- Flakiness under load: the selftests have no wall-clock assertions I found, and 4 toolbox + runs + several host runs were all stable, but that is a handful of runs, not a soak. +- The coverage files under `conformance/_artifacts/coverage/` end in the same state as before + only because a single check rewrites them; I did not change that pre-existing quirk. +- L2/L3 conformance: not run (no cluster in CI, as before).