jakman85andClaude Opus 5
793c3940a8
ha-control-plane-as-code/check.sh had: re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:...'secondary'", K3S_TEXT, re.S) With re.S every `.` matches newlines, so `.*?` and the nested `(?:.*\n){0,10}?` can each consume the whole document -- nested unbounded repetition, which backtracks exponentially on a failed or late match. MEASURED in CI via `docker top` on the live job container: python3 at 19:54 of CPU time, which is essentially the entire ~28 minute gate. Localised to the exact line by extracting the inline python and running it under faulthandler.dump_traceback_later, which named re/__init__.py:177 in search from this call -- not inferred from reading. Replaced `.` with `[^\n]*` and dropped re.S, so the {0,10} is a real ten-line window with nothing to backtrack into. On the real input (22,179 chars): 0.2 ms, match still found. Whole check: 0.199s, rc 0, 115 assertions, identical success message. WHY IT SURFACED NOW, AND IT WAS MINE: K3S_TEXT is every *.yml in roles/k3s_server concatenated. The regex was latent for as long as it existed; PR #71 added ~100 lines to that role (kubelet graceful-shutdown config), growing the input past the point where exponential backtracking stopped being survivable. WHY ONLY infra/** WAS SLOW: touched-requirements.sh maps infra/* to R-ENV-01, so the L0 run becomes --include-scope whole-repo --requirements R-ENV-01, which selects this whole-repo check. Doc/platform changes never did -- #69 1m55s and #72 1m53s stayed fast while #71 27m44s, the #73 merge 27m46s and #75 all ran ~28m. The pattern is unique in the repo (grep over all l0-static check.sh files). Not re-timed in CI post-fix; this PR's own run is the first datapoint. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
77 lines
4.5 KiB
Markdown
77 lines
4.5 KiB
Markdown
---
|
|
date: 2026-10-18
|
|
branch: fix/ha-check-catastrophic-regex
|
|
title: "One regex made the merge gate take 28 minutes instead of 2"
|
|
---
|
|
|
|
## What changed
|
|
|
|
The merge gate started taking **~28 minutes** instead of its usual **<2 minutes**, on every change
|
|
touching `infra/**`. One regex was responsible for essentially all of it.
|
|
|
|
`conformance/tests/l0-static/ha-control-plane-as-code/check.sh` contained:
|
|
|
|
```python
|
|
join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
|
|
K3S_TEXT, re.S)
|
|
```
|
|
|
|
With `re.S`, every `.` also matches newlines, so `.*?` and the nested `(?:.*\n){0,10}?` can each
|
|
consume the entire document. That is **nested unbounded repetition** — ten levels of a quantifier
|
|
that can each swallow everything — and a failed or late match backtracks exponentially.
|
|
|
|
Replaced `.` with `[^\n]*` and dropped `re.S`, so the `{0,10}` is a genuine ten-line window with
|
|
nothing to backtrack into. Same intent, same verdict, no DOTALL.
|
|
|
|
**Why it surfaced now, and it was mine.** `K3S_TEXT` is every `*.yml` in `roles/k3s_server`
|
|
concatenated. The regex was latent for as long as it existed; **PR #71 added ~100 lines to that role
|
|
(the kubelet graceful-shutdown config), which grew the input past the point where exponential
|
|
backtracking stopped being survivable.** Nothing about the check changed — my change grew its input.
|
|
|
|
**Why only `infra/**` changes were slow.** `ci/touched-requirements.sh:268` maps `infra/*` to
|
|
`R-ENV-01`, and the resulting L0 run is `--include-scope whole-repo --requirements R-ENV-01`, which
|
|
pulls in this whole-repo check. Doc-only and `platform/**` changes never selected it, which is why
|
|
#69 (1m55s) and #72 (1m53s) stayed fast while #71 (27m44s), the #73 merge (27m46s) and #75 all ran
|
|
~28 minutes.
|
|
|
|
## What I verified
|
|
|
|
* **The hot spot, by measurement rather than reading.** `docker top` on the live CI job container for
|
|
run #299 showed `python3 -` with **19:54 of CPU time** under
|
|
`ha-control-plane-as-code/check.sh`, inside
|
|
`conformance/run.sh --layer l0 --include-scope whole-repo --requirements R-ENV-01`.
|
|
* **Reproduced locally**: the check did not finish within a 100-second budget on an unmodified
|
|
checkout.
|
|
* **Localised to the exact line** by extracting the inline Python and running it under
|
|
`faulthandler.dump_traceback_later(25)`, which named `re/__init__.py:177 in search` called from
|
|
the `join_when` line — not inferred from reading the patterns.
|
|
* **The pattern is unique in the repo**: `grep -rlnE "\{0,[0-9]+\}\?" conformance/tests/l0-static/*/check.sh`
|
|
returns only this file, so there is no sibling with the same hazard.
|
|
* **The fix is fast AND verdict-preserving**, tested against the real `K3S_TEXT` (22,179 chars, 403
|
|
lines): the new regex matches (`YES`) in **0.2 ms**. Sanity-checked that the thing it looks for
|
|
genuinely exists — one `--server https://` and three `k3s_server_mode == 'secondary'` occurrences.
|
|
* **The whole check now runs in 0.199s, rc 0, 115 assertions**, with the same success message as
|
|
before.
|
|
|
|
## What I did NOT verify
|
|
|
|
* **I did not re-time the full gate in CI after the fix.** The expectation is that `infra/**` changes
|
|
return to the ~2-minute range, but that is arithmetic from the 19m54s figure, not a measured
|
|
post-fix run. The PR's own CI run is the first real datapoint — read it.
|
|
* **I did not audit the other 23 L0 checks for different performance pathologies.** This one was
|
|
found because it was actively burning CPU in a live job; a check that is merely slow (not
|
|
exponential) would not have stood out, and `ci/gate.sh` has **no per-stage timing** at all
|
|
(`grep -nE "SECONDS|date \+|elapsed"` finds nothing), so there is no data on where the remaining
|
|
time goes.
|
|
* **Nothing prevents this class of bug returning.** No check asserts gate stage runtime, and no lint
|
|
forbids nested quantifiers under `re.S`. A timing budget per stage, or a `re` lint, would close it;
|
|
neither is written here.
|
|
* **I did not verify my claim about *why* only `infra/**` was affected beyond the path mapping and
|
|
the observed durations.** The correlation is strong (5 runs, two clear clusters) and the live
|
|
process list confirms the `--requirements R-ENV-01` scope, but I did not read
|
|
`conformance/run.sh`'s scope-selection logic end to end.
|
|
* The regex rewrite assumes the `when:` clause is indented with spaces or tabs only (`[ \t]*`). A
|
|
file using some other whitespace before `when:` would now fail to match where the old DOTALL
|
|
version would have succeeded. Unlikely in YAML, and the current input matches, but it is a
|
|
narrowing.
|