ha-control-plane-as-code/check.sh had:
re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:...'secondary'",
K3S_TEXT, re.S)
With re.S every `.` matches newlines, so `.*?` and the nested `(?:.*\n){0,10}?`
can each consume the whole document -- nested unbounded repetition, which
backtracks exponentially on a failed or late match. MEASURED in CI via
`docker top` on the live job container: python3 at 19:54 of CPU time, which is
essentially the entire ~28 minute gate.
Localised to the exact line by extracting the inline python and running it under
faulthandler.dump_traceback_later, which named re/__init__.py:177 in search from
this call -- not inferred from reading.
Replaced `.` with `[^\n]*` and dropped re.S, so the {0,10} is a real ten-line
window with nothing to backtrack into. On the real input (22,179 chars):
0.2 ms, match still found. Whole check: 0.199s, rc 0, 115 assertions, identical
success message.
WHY IT SURFACED NOW, AND IT WAS MINE: K3S_TEXT is every *.yml in
roles/k3s_server concatenated. The regex was latent for as long as it existed;
PR #71 added ~100 lines to that role (kubelet graceful-shutdown config), growing
the input past the point where exponential backtracking stopped being
survivable.
WHY ONLY infra/** WAS SLOW: touched-requirements.sh maps infra/* to R-ENV-01, so
the L0 run becomes --include-scope whole-repo --requirements R-ENV-01, which
selects this whole-repo check. Doc/platform changes never did -- #69 1m55s and
#72 1m53s stayed fast while #71 27m44s, the #73 merge 27m46s and #75 all ran
~28m.
The pattern is unique in the repo (grep over all l0-static check.sh files).
Not re-timed in CI post-fix; this PR's own run is the first datapoint.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
jakman85andClaude Opus 5