fix(conformance): one regex made the merge gate take 28 minutes instead of 2 #76

2026-10-18T07:40:25Z

gitea-admin commented

2026-10-18 07:40:25 +00:00

One regex was costing ~26 minutes of every `infra/**` gate run

The merge gate jumped from <2 minutes to ~28 minutes. A single regex in ha-control-plane-as-code/check.sh accounted for essentially all of it:

join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
                      K3S_TEXT, re.S)

With re.S, every . also matches newlines — so .*? and the nested (?:.*\n){0,10}? can each consume the whole document. That is nested unbounded repetition, ten levels deep, and a failed or late match backtracks exponentially.

Measured, not inferred

docker top on the live CI job container for run #299: python3 - at 19:54 of CPU time, parented by ha-control-plane-as-code/check.sh, inside conformance/run.sh --layer l0 --include-scope whole-repo --requirements R-ENV-01.
Reproduced locally: the check did not finish within a 100-second budget on an unmodified checkout.
Localised to the exact line by extracting the inline Python and running it under faulthandler.dump_traceback_later(25), which named re/__init__.py:177 in search from this call.
The pattern is unique in the repo — grep -rlnE "\{0,[0-9]+\}\?" conformance/tests/l0-static/*/check.sh returns only this file.

The fix

[^\n]* instead of ., and no re.S, so the {0,10} is a genuine ten-line window with nothing to backtrack into. Same intent, same verdict.

	before	after
the regex alone, real input (22,179 chars)	did not complete	0.2 ms, match still found
the whole check	>100 s (unfinished)	0.199 s, rc 0, 115 assertions, identical success message

Why it surfaced now — and it was self-inflicted

K3S_TEXT is every *.yml in roles/k3s_server concatenated. The regex was latent for as long as it existed; PR #71 added ~100 lines to that role (the kubelet graceful-shutdown config), growing the input past the point where exponential backtracking stopped being survivable. Nothing about the check changed — its input did.

Why only `infra/**` changes were slow

ci/touched-requirements.sh:268 maps infra/* → R-ENV-01, so those runs become --include-scope whole-repo, which selects this check. The split is clean:

PR	touched	duration
#69	docs	1m55s
#72	`platform/keycloak`	1m53s
#71	`infra/ansible` + script	27m44s
#73 merge	`infra/ansible` + more	27m46s
#75	`infra/ansible`	~28m

Worth merging first

This takes ~26 minutes off every future infra/** gate run, including the ones on #75 and the Renovate cron fix. Those PRs are otherwise waiting on a gate that is mostly this regex.

Not verified

I did not re-time the full gate in CI after the fix. The expectation of a return to ~2 minutes is arithmetic from the 19m54s figure, not a measured post-fix run — this PR's own CI run is the first real datapoint.
I did not audit the other 23 L0 checks for different performance pathologies. This one was found only because it was actively burning CPU in a live job; a merely-slow check would not have stood out. ci/gate.sh has no per-stage timing at all (grep -nE "SECONDS|date \+|elapsed" finds nothing), so there is no data on where the remaining time goes.
Nothing prevents this class returning — no check asserts stage runtime, no lint forbids nested quantifiers under re.S. Both are closeable; neither is in this PR.
The rewrite assumes when: is preceded only by spaces or tabs ([ \t]*). Other whitespace would now fail to match where DOTALL would have succeeded. Unlikely in YAML and the current input matches, but it is a narrowing.

Local gate: rc 3, zero failures — all seven runnable stages pass; only gitea-settings (CI-only token) and L2/L3 conformance (no cluster) did not run.

Full detail: decisions/2026-10-18-ha-check-catastrophic-regex.md

🤖 Generated with Claude Code

## One regex was costing ~26 minutes of every `infra/**` gate run The merge gate jumped from **<2 minutes to ~28 minutes**. A single regex in `ha-control-plane-as-code/check.sh` accounted for essentially all of it: ```python join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'", K3S_TEXT, re.S) ``` With `re.S`, every `.` also matches newlines — so `.*?` and the nested `(?:.*\n){0,10}?` can each consume the whole document. That is **nested unbounded repetition**, ten levels deep, and a failed or late match backtracks exponentially. ## Measured, not inferred - `docker top` on the **live CI job container** for run #299: `python3 -` at **19:54 of CPU time**, parented by `ha-control-plane-as-code/check.sh`, inside `conformance/run.sh --layer l0 --include-scope whole-repo --requirements R-ENV-01`. - Reproduced locally: the check **did not finish within a 100-second budget** on an unmodified checkout. - Localised to the exact line by extracting the inline Python and running it under `faulthandler.dump_traceback_later(25)`, which named `re/__init__.py:177 in search` from this call. - The pattern is **unique in the repo** — `grep -rlnE "\{0,[0-9]+\}\?" conformance/tests/l0-static/*/check.sh` returns only this file. ## The fix `[^\n]*` instead of `.`, and no `re.S`, so the `{0,10}` is a genuine ten-line window with nothing to backtrack into. Same intent, same verdict. | | before | after | |---|---|---| | the regex alone, real input (22,179 chars) | did not complete | **0.2 ms**, match still found | | the whole check | >100 s (unfinished) | **0.199 s**, rc 0, 115 assertions, identical success message | ## Why it surfaced now — and it was self-inflicted `K3S_TEXT` is every `*.yml` in `roles/k3s_server` concatenated. The regex was latent for as long as it existed; **PR #71 added ~100 lines to that role** (the kubelet graceful-shutdown config), growing the input past the point where exponential backtracking stopped being survivable. Nothing about the check changed — its input did. ## Why only `infra/**` changes were slow `ci/touched-requirements.sh:268` maps `infra/*` → `R-ENV-01`, so those runs become `--include-scope whole-repo`, which selects this check. The split is clean: | PR | touched | duration | |---|---|---| | #69 | docs | 1m55s | | #72 | `platform/keycloak` | 1m53s | | #71 | `infra/ansible` + script | **27m44s** | | #73 merge | `infra/ansible` + more | **27m46s** | | #75 | `infra/ansible` | **~28m** | ## Worth merging first This takes ~26 minutes off every future `infra/**` gate run, including the ones on #75 and the Renovate cron fix. Those PRs are otherwise waiting on a gate that is mostly this regex. ## Not verified - **I did not re-time the full gate in CI after the fix.** The expectation of a return to ~2 minutes is arithmetic from the 19m54s figure, not a measured post-fix run — **this PR's own CI run is the first real datapoint.** - **I did not audit the other 23 L0 checks for different performance pathologies.** This one was found only because it was actively burning CPU in a live job; a merely-slow check would not have stood out. **`ci/gate.sh` has no per-stage timing at all** (`grep -nE "SECONDS|date \+|elapsed"` finds nothing), so there is no data on where the remaining time goes. - **Nothing prevents this class returning** — no check asserts stage runtime, no lint forbids nested quantifiers under `re.S`. Both are closeable; neither is in this PR. - The rewrite assumes `when:` is preceded only by spaces or tabs (`[ \t]*`). Other whitespace would now fail to match where DOTALL would have succeeded. Unlikely in YAML and the current input matches, but it is a narrowing. Local gate: **rc 3, zero failures** — all seven runnable stages pass; only `gitea-settings` (CI-only token) and L2/L3 `conformance` (no cluster) did not run. Full detail: `decisions/2026-10-18-ha-check-catastrophic-regex.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-18 07:40:25 +00:00

fix(conformance): one regex made the merge gate take 28 minutes instead of 2

gate / gate (pull_request) Successful in 1m55s

Details

gate / conformance (pull_request) Has been skipped

Details

793c3940a8

ha-control-plane-as-code/check.sh had:

  re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:...'secondary'",
            K3S_TEXT, re.S)

With re.S every `.` matches newlines, so `.*?` and the nested `(?:.*\n){0,10}?`
can each consume the whole document -- nested unbounded repetition, which
backtracks exponentially on a failed or late match. MEASURED in CI via
`docker top` on the live job container: python3 at 19:54 of CPU time, which is
essentially the entire ~28 minute gate.

Localised to the exact line by extracting the inline python and running it under
faulthandler.dump_traceback_later, which named re/__init__.py:177 in search from
this call -- not inferred from reading.

Replaced `.` with `[^\n]*` and dropped re.S, so the {0,10} is a real ten-line
window with nothing to backtrack into. On the real input (22,179 chars):
0.2 ms, match still found. Whole check: 0.199s, rc 0, 115 assertions, identical
success message.

WHY IT SURFACED NOW, AND IT WAS MINE: K3S_TEXT is every *.yml in
roles/k3s_server concatenated. The regex was latent for as long as it existed;
PR #71 added ~100 lines to that role (kubelet graceful-shutdown config), growing
the input past the point where exponential backtracking stopped being
survivable.

WHY ONLY infra/** WAS SLOW: touched-requirements.sh maps infra/* to R-ENV-01, so
the L0 run becomes --include-scope whole-repo --requirements R-ENV-01, which
selects this whole-repo check. Doc/platform changes never did -- #69 1m55s and
#72 1m53s stayed fast while #71 27m44s, the #73 merge 27m46s and #75 all ran
~28m.

The pattern is unique in the repo (grep over all l0-static check.sh files).
Not re-timed in CI post-fix; this PR's own run is the first datapoint.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 approved these changes

2026-10-18 07:44:30 +00:00

jakman85 merged commit c340bbfc4b into main

2026-10-18 07:51:51 +00:00

jakman85 referenced this issue from a commit

2026-10-18 07:51:52 +00:00

Merge pull request 'fix(conformance): one regex made the merge gate take 28 minutes instead of 2' (#76) from fix/ha-check-catastrophic-regex into main

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#76