home-network-k3s-impl/decisions/2026-11-08-parallel-l0-checks.md

date, branch, title

date	branch	title
2026-11-08	feat/parallel-l0-checks	L0 offline checks, mutation suites and yamllint run in bounded parallel pools (gate 184s -> 32s locally)

What changed

The gate's wall time was dominated by serial loops over independent subprocesses. Three of them now run in bounded pools; none of them changes a verdict, an exit code, or the order of anything printed.

• conformance/lib/run_l1_offline.py — the per-check loop (L0 and L1 offline checks) runs in a ThreadPoolExecutor of bash <check> children. --jobs N / env CONF_OFFLINE_JOBS, default min(8, usable CPUs) (sched_getaffinity, so a cgroup cpuset is respected); --jobs 1 is the old serial runner. Results are consumed in SUBMISSION order, so cases, the JUnit, the step summary and the 0/1/3 aggregation are built from exactly the same ordered list as before. Additive output only: one [offline] running N check(s), jobs=… line, and a trailing [N.Ns] wall time on each per-check status line. Children get stdin=/dev/null (parallel children must not share the runner's stdin).
• conformance/tests/l0-static/keycloak-{passkey,client-secret}-reconciler/mutations.py — pass 1 (serial) checks every anchor and builds each mutated source, exactly as before; pass 2 runs the --selftest children in a pool (env MUTATION_JOBS, default min(8, CPUs), 1 = serial). Verdicts are consumed in MUTATIONS order. A timeout (TimeoutExpired) still propagates and fails the check; queued mutations are cancelled. An invalid MUTATION_JOBS exits 2 (fails the check, never skips it).
• Nested-pool budget. When the outer runner is parallel it exports MUTATION_JOBS = max(2, min(8, (cpus - jobs + 2) // 2)) unless the caller already set one. Reason: only those two checks fan out; the other jobs-2 workers are single processes, so the two suites split the CPUs those leave free. Peak processes ≈ (jobs-2) + 2*inner ≤ cpus whenever cpus ≥ jobs+2 (24 CPUs: 6 + 16 = 22), and ~1.25–1.5× cpus on a small runner (4 CPUs: 2 + 4 = 6). The floor of 2 is deliberate: a 1-wide inner pool would make the 40-mutation passkey suite (~85s serial) the critical path of the stage.
• ci/lint.sh — yamllint runs over LINT_JOBS (default min(8, nproc)) CONTIGUOUS slices of the sorted file list concurrently; each slice's output is buffered to its own file and printed in slice order, so the text is the same as one serial call and a file header is never separated from its line:col findings. Stage rc = the highest slice rc; a slice that leaves no rc file counts as 1. Invalid LINT_JOBS exits 2.

Parallel-safety audit (all 25 l0-static/*/check.sh, 3 l1-rendering/*/render-check.sh)

Grepped every check for fixed /tmp paths, _artifacts/ARTIFACTS/COVERAGE_OUT, write_text/open(…,"w"), mktemp/mkdtemp/TemporaryDirectory, sockets/ports, git writes and run.sh invocations, and read what each writer writes to:

• Scratch state: all private. Every writer uses its own mktemp -d / mkdtemp / TemporaryDirectory (placeholder-guard-coupling, spec-immutability-enforced, whole-repo-invariants-wired, auditability-proofs-producible, alert-rules-unit-tested and its platform/monitoring/tests/run.sh, internal-services-publish-no-public-dns, oauth2-proxy-{api-routes-per-app,cookie-names-distinct}, both keycloak reconcilers, the three L1 render checks). No fixed /tmp path anywhere.
• Git writes: only spec-immutability-enforced, and only git init/add/commit inside its own mktemp repo. Everything else is git ls-files/ls-tree/log/rev-parse (read-only).
• Ports: the reconciler selftests bind 127.0.0.1:0 (kernel-assigned); the passkey selftest also dials 127.0.0.1:1 as a deliberate refused connection. No fixed listener.
• Ansible (dev-nodes-provisioned-as-code): --syntax-check/--list-hosts only, retry_files_enabled = False; its local tmp is per-process.
• The one shared writer: auditability-proofs-producible runs ./run.sh --coverage-only --requirements …, which REWRITES conformance/_artifacts/coverage/coverage.{json,md} (it inherits CONF_ARTIFACTS). That was already true serially. No other offline check reads or writes that directory (the only reader is tests/meta/harness-selfcheck, not an offline check), and the outer runner writes coverage BEFORE and the JUnit AFTER the pool, so the final state is the same as serial. Not serialised; left as is and recorded here. lib/status.py (run by that check) reads _artifacts/report.junit.xml as an advisory — written only after the pool, unchanged.
• whole-repo-invariants-wired calls run.sh --print-selection (writes nothing) and the runner itself against synthetic dirs in its own mktemp.

Result: no check needed fixing or serialising. A new check that needs shared state must say so; the runner's docstring now states that contract.

What I verified

All in the pinned toolbox (dev.sh run, DEV_KUBECONFIG=/dev/null, ENGINE shim adding a ro mount of the main checkout's .git; spec repo resolved as a sibling, spec-immutability pass), host 24 CPUs, every dev.sh call under flock.

• Gate, before (origin/main 1a33814, detached scratch worktree + spec mount): GATE_RC=3, in-container wall 183.9s; l0 stage 168.3s, lint 7.8s. "Stages that did not run: gitea-settings" (plus the expected on-base touched-set-not-determinable note, since main has no branch commits).
• Gate, after (this branch, commit a775082): GATE_RC=3, in-container wall 32.1s; l0 21.1s, lint 2.4s, l1 2.3s. "Stages that did not run: gitea-settings" — the only not-run stage. rc 3, not green. Caveat: the branch run selected 23 L0 checks (the touched-requirement scoping dropped dev-nodes-provisioned-as-code and placeholder-guard-coupling), the main run 25.
• Same-tree L0 comparison (all 25 l0 dirs, same tree, same container): CONF_OFFLINE_JOBS=1 MUTATION_JOBS=1 164.0s / 165.3s vs default (jobs=8, MUTATION_JOBS=8) 19.2s / 18.7s. JUnit cmp-identical, step summary cmp-identical, stderr identical except the timing suffixes and the JUnit path in the last line.
• Mutation suites, host: passkey 83.2s → 10.5s (MUTATION_JOBS=8), client-secret 11.2s → 2.0s; output of MUTATION_JOBS=1 and =8 byte-identical to the pre-change script's, rc 0 all round.
• Falsification (not committed): in a scratch worktree at a775082, made the passkey mutation "the protected-subtree screen is never called" a no-op (replacement == anchor). Host: MUTATION_JOBS=1 and =8 both rc 1, output byte-identical (with a second, broken anchor also injected: both reported). Toolbox: runner rc 1 both serial and parallel, L0 FAIL keycloak-passkey-reconciler … 1 of 40 mutation(s) not proven … SURVIVED; full gate rc 1, stage l0: FAIL, MERGE GATE: BLOCKED. Reverted; scratch worktree removed.
• lint.sh, host scratch repo (31 YAML files, two broken): LINT_JOBS=1/4/8 all rc 1, stdout identical, findings named y/f07.yaml / 2:1 … key-duplicates etc.; clean tree rc 0; LINT_JOBS=x rc 2.

What I did NOT verify

• CI timings at the time of writing. The PR's CI run is measured after push and reported on the PR / handoff message, not here. The CI runner's CPU count is UNVERIFIED — the new [offline] running … jobs=… cpus=… line in the CI log is what will show it. On a small runner the speedup will be smaller than locally.
• approval-policy will be red on CI: ci/** and conformance/** are sensitive paths and need the operator's review. I did not and must not override it.
• Flakiness under load: the selftests have no wall-clock assertions I found, and 4 toolbox runs + several host runs were all stable, but that is a handful of runs, not a soak.
• The coverage files under conformance/_artifacts/coverage/ end in the same state as before only because a single check rewrites them; I did not change that pre-existing quirk.
• L2/L3 conformance: not run (no cluster in CI, as before).