Compare commits

..
Author SHA1 Message Date
jakman85andClaude Opus 5.5 89f48e4a5f docs(decisions): handoff for parallel L0 checks; runner usage names --jobs
gate / gate (pull_request) Successful in 59s
gate / conformance (pull_request) Skipped
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:34:48 -06:00
jakman85andClaude Opus 5.5 a775082ae1 feat(ci): run L0/L1 offline checks, mutation suites and yamllint in bounded parallel pools
- conformance/lib/run_l1_offline.py: checks run in a thread pool (--jobs / CONF_OFFLINE_JOBS,
  default min(8, usable CPUs)); results consumed in submission order, so log lines, JUnit,
  step summary and the 0/1/3 aggregation are unchanged. Exports a smaller MUTATION_JOBS to
  the children when parallel, so nested pools do not oversubscribe.
- keycloak-{passkey,client-secret}-reconciler/mutations.py: anchors checked first (serial),
  then the --selftest subprocesses run in a pool (MUTATION_JOBS, default min(8, CPUs));
  verdicts reported in MUTATIONS order. Output byte-identical to the serial loop.
- ci/lint.sh: yamllint over LINT_JOBS contiguous slices concurrently, per-slice output
  buffered and printed in order; stage rc = highest slice rc.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:18:57 -06:00
jakman85 1a33814522 Merge pull request 'chore(renovate): hand-migrate docker key, track gitea/runner, keep range lower bounds' (#145) from chore/renovate-config-cleanup into main
gate / gate (push) Successful in 2m59s
gate / conformance (push) Skipped
Reviewed-on: #145
2026-11-08 06:17:59 +00:00
jakman85 4a3f6dee59 Merge pull request 'fix(keycloak): install the 26.8.2 operator's new CRDs (operator CrashLoop)' (#144) from fix/keycloak-operator-crds into main
gate / gate (push) Canceled after 0s
gate / conformance (push) Canceled after 0s
Reviewed-on: #144
2026-11-08 06:16:56 +00:00
jakman85andClaude Opus 5.5 759a0e51c8 chore(renovate): hand-migrate docker key, track gitea/runner, keep range lower bounds
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Skipped
- Replace the deprecated top-level `docker` key with the packageRule Renovate's
  own migration produced (matchCategories docker, enabled true), keeping every
  comment. Supersedes #139, which dropped all comment lines.
- Add gitea/runner (act_runner's upstream rename) to the blast-radius rule.
- rangeStrategy bump for flux/helm `>=A <B` ranges: the default `auto` resolves
  to `replace`, which turned `>=1.12.0 <1.13.0` into `<1.14.0` (#131, #130,
  #115). In-range `bump` updates are disabled; Flux adopts those unaided.
- Restore headlamp's lower bound: `<0.47.0` -> `>=0.46.0 <0.47.0` (live 0.46.0).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:05:49 -06:00
jakman85andClaude Opus 5.5 ea30ba876c fix(keycloak): install the 26.8.2 operator's new CRDs (operator CrashLoop)
gate / gate (pull_request) Successful in 3m5s
gate / conformance (pull_request) Skipped
PR #121 bumped the three keycloak-k8s-resources URLs to 26.8.2, but 26.8.2
ships two more CRDs (keycloakoidcclients, keycloaksamlclients). The new
operator fails to start its keycloaksamlclients informer (Not Found) and
CrashLoops, holding Flux `platform` in health checks.

- add both CRD URLs with the same renovate marker/depName
- patch operator memory 450Mi -> limit 768Mi / request 512Mi (26.0.6 was
  OOMKilled at 421/450Mi, 154 restarts)
- README/renovate comments: Renovate cannot add CRDs a release introduces

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:00:01 -06:00
17 changed files with 704 additions and 299 deletions

No files matched your search

+1 -2
View File
@@ -165,8 +165,7 @@ Cost: the job shares the single act_runner slot with the gate, but the build its
merge-gate.yaml the pipeline (lint → l0 → l1 → conformance → decision)
images.yaml build/verify/publish this repo's container images (see above)
scripts/
install-tools.sh installs the pinned tools per stage (from ci/versions.env) — used by the
dormant merge-gate.yaml blueprint only; gate.yml runs in the toolbox image
install-tools.sh installs the pinned tools per stage (from ci/versions.env)
decide.sh the R-TEST-03 merge decision over job results
images.sh the image pipeline's driver (what to build, base, push, digest)
README.md this file
+10 -99
View File
@@ -3,10 +3,8 @@
# This is the LIVE gate wired to the in-cluster Gitea Actions runner (ns gitea-actions,
# Docker executor + DinD). It invokes the AGGREGATE merge gate — `./ci/gate.sh`, which is
# exactly what `make -C ci gate` runs, so CI and the devcontainer gate the same way
# (R-TEST-02) — and since 2026-11-08 it runs INSIDE the same image: the job `container:` is the
# CI-built devcontainer toolbox (see "THE JOB IMAGE" below), not a generic runner image that
# downloads the pinned tools on every run. (See further below for why the script, not `make`,
# is invoked here.) It runs every stage in order —
# (R-TEST-02): the runner just installs the pinned tools from ci/versions.env first (and see
# below for why the script, not `make`, is invoked here). It runs every stage in order —
# spec-immutability, lint, Kyverno L0, render==golden, L1 offline render — and then applies the
# R-TEST-03 decision, printing
# "MERGE GATE: PASS" (0) / "BLOCKED" (1) / "UNDETERMINED" (3) / "NOT APPLICABLE" (0). It is the
@@ -19,8 +17,7 @@
# kinds of non-run must be told apart HERE, in the caller, because only the caller knows which
# environment it is:
# * a pinned tool missing or a seam unwired ON THIS RUNNER is a DEFECT — the toolchain is
# baked into the job image and checked against ci/versions.env two steps up.
# GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
# installed for us two steps up. GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
# "does this look like CI?" heuristic) and makes those skips exit 1: untolerated, job fails.
# * spec-immutability CANNOT be verified here at all: the spec repo is a separate repo and is
# not checked out into this runner (SPEC_REPO_SLUG unwired; blueprint in merge-gate.yaml). It
@@ -44,32 +41,6 @@
# same script the laptop runs; the extra pre-step is a read-only git-diff check and is
# purely additive. The full pipeline blueprint (spec-checkout, per-layer jobs, decision job)
# is retained dormant in merge-gate.yaml.
#
# THE JOB IMAGE (2026-11-08, feat/gate-toolbox-container). The gate job runs in
# gitea.home.keithjackson.dev/home-network/toolbox, pinned BY DIGEST — the image
# .gitea/workflows/images.yaml builds from infra/devcontainer/Containerfile, verifies, and
# publishes. It is the SAME image `infra/devcontainer/dev.sh` runs, so a local gate and the CI
# gate now execute the identical toolchain (R-TEST-02), not two installs that agree by intent.
# Before this, the job ran in ghcr.io/catthehacker/ubuntu:act-22.04 and spent ~11s of every run
# re-downloading yamllint, check-jsonschema, kyverno, promtool and chainsaw
# (.gitea/workflows/scripts/install-tools.sh l0 + l1; measured on runs 494/495).
# * PINNED BY DIGEST, MOVED BY RENOVATE. The `# renovate-digest:` marker above the image line
# is the repo-wide convention (renovate.json5, "This repo's own CI-published images"):
# Renovate follows the `main` tag and opens a digest-bump PR, whose OWN gate run then
# executes on the new image — the bump tests itself. infra/gitea/registry-retention.sh reads
# the same marker, so the pinned digest is never pruned. Never hand-edit it to a tag.
# * PULL AUTH: none needed. The Home-Network org is public and the registry issues an
# anonymous pull token (measured 2026-11-08: anonymous /v2/token -> manifest 200), so the
# runner's DinD pulls it like any public image. If the org is ever made private, add
# `credentials:` here with a read:package token secret — the automatic Actions token has no
# package scope at /v2/token (see images.yaml's header).
# * COLD PULL: ~467 MB compressed (1.54 GB unpacked). The DinD store is an emptyDir, so the
# first job after a runner-pod restart or a digest bump pays one pull; every other run finds
# it cached (forcePull=false), exactly as it already did for the catthehacker image.
# * NO NODE in the toolbox, so `actions/checkout` (a Node action) cannot run here. The checkout
# step is plain git instead — see its comment for how it keeps fetch-depth-0 semantics.
# * install-tools.sh is NOT deleted: merge-gate.yaml (the dormant blueprint) still calls it,
# and conformance/tests/l0-static/alert-rules-unit-tested asserts its promtool wiring.
---
name: gate
@@ -107,76 +78,16 @@ jobs:
# Single required status check: the static merge gate.
gate:
runs-on: ubuntu-latest
# The CI-built toolbox — the same image dev.sh runs. See "THE JOB IMAGE" in the header.
container:
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/toolbox tag=main
image: gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501efe34fa0b039e3f0d360738686a0f59109866c49779768f5d5ed0ef81
steps:
# Plain git, because actions/checkout is a Node action and the toolbox has no Node. It
# reproduces what actions/checkout@v7 with `fetch-depth: 0` did here (read from its logged
# commands on runs 490/495): full history of every branch + tags, so
# ci/touched-requirements.sh can diff against origin/<base>; the exact event commit
# checked out (detached for a PR, as `-B <branch>` for a push). One difference, on
# purpose: the job token is passed to the single fetch through GIT_CONFIG_* environment
# variables and is NOT persisted into .git/config — nothing after this step needs
# authenticated git (ci/ and conformance/ make no network git calls), so there is no
# reason to leave a credential in the workspace for the gate's scripts to read.
- name: Checkout (git, full history)
env:
GIT_TOKEN: ${{ github.token }}
SERVER_URL: ${{ github.server_url }}
REPOSITORY: ${{ github.repository }}
SHA: ${{ github.sha }}
REF: ${{ github.ref }}
run: |
set -euo pipefail
# act runs steps with umask 0000, so without this every checked-out directory is
# world-writable — and Ansible then IGNORES infra/ansible/ansible.cfg as a source
# ("Ansible is being run in a world writable directory"), which made the L0 check
# prod-baremetal-not-dev-surface fail on this PR's first CI run (run 500). The
# toolbox has Ansible, so that check now drives it for real. 0022 = what a
# workstation clone gets (reproduced locally: 0000 fails the check, 0022 passes).
umask 0022
git config --global --add safe.directory "$PWD"
git init -q .
git remote add origin "${SERVER_URL}/${REPOSITORY}"
auth="$(printf 'x-access-token:%s' "$GIT_TOKEN" | base64 -w0)"
GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=http.extraHeader \
GIT_CONFIG_VALUE_0="AUTHORIZATION: basic ${auth}" \
git -c protocol.version=2 fetch --quiet --no-tags --prune --no-recurse-submodules origin \
'+refs/heads/*:refs/remotes/origin/*' '+refs/tags/*:refs/tags/*' \
"+${SHA}:refs/remotes/checkout/head"
unset auth GIT_TOKEN
case "$REF" in
refs/heads/*) git checkout -q --force -B "${REF#refs/heads/}" "$SHA" ;;
*) git checkout -q --force --detach "$SHA" ;;
esac
[ "$(git rev-parse HEAD)" = "$SHA" ] || { echo "::error::HEAD is not ${SHA}" >&2; exit 1; }
echo "checked out $(git log -1 --format='%H %s') (ref ${REF}); $(git rev-list --count HEAD) commits of history"
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
fetch-depth: 0 # full history so ci/touched-requirements.sh can diff vs main
# The image is the toolchain, so prove it is the PINNED one before trusting it: every
# *_VERSION in ci/versions.env must equal what the image recorded at build time
# (/etc/dev-toolchain-versions, written by infra/devcontainer/Containerfile; its `verify`
# stage already asserted the version-sensitive binaries report those pins). This replaces
# the per-run install: a drifted toolbox fails HERE, naming the tool, instead of gating with
# a different engine. ci/versions.env and infra/devcontainer/versions.env must move together.
- name: Toolchain is the one ci/versions.env pins
- name: Install pinned CI toolchain (ci/versions.env)
run: |
set -euo pipefail
. ci/versions.env
drift=0
for var in $(grep -oE '^[A-Z_]+_VERSION' ci/versions.env); do
tool="$(printf '%s' "${var%_VERSION}" | tr 'A-Z_' 'a-z-')"
want="${!var}"
got="$(sed -n "s/^${tool}=//p" /etc/dev-toolchain-versions)"
if [ "$got" = "$want" ]; then
printf 'ok %-18s %s\n' "$tool" "$want"
else
printf '::error::DRIFT %s: ci/versions.env pins %s, the job image has %s\n' "$tool" "$want" "${got:-nothing}"
drift=1
fi
done
[ "$drift" -eq 0 ]
./.gitea/workflows/scripts/install-tools.sh l0
./.gitea/workflows/scripts/install-tools.sh l1
# R-TEST-06: the spec is read-only to agents. An EXPLICIT early surfacing of the
# spec-immutability check, so its result is a distinct, tolerant status line in the
+1 -1
View File
@@ -99,7 +99,7 @@ caller that must distinguish them invokes `./ci/gate.sh` directly.
| stage | a skip there means | verdict |
|---|---|---|
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is the job image — the CI-built toolbox, `.gitea/workflows/gate.yml` `container:` — so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
| `decisions-log` | **nothing** — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run" | a reported skip is a **defect in the stage** and BLOCKS. The stage fails when the frozen `DECISIONS-LOG.md` is modified (which silently re-creates, for every *other* open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a `decisions/` entry is malformed. There is deliberately **no** `REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard. See `decisions/README.md`. |
| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
+3 -3
View File
@@ -46,8 +46,8 @@
# WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing):
#
# lint / l0 / render-golden / l1 A skip here means a PINNED tool is absent or a wired
# seam went unwired. On a CI runner the pinned toolchain is the job image itself (the
# CI-built toolbox, .gitea/workflows/gate.yml), so a skip there is a DEFECT, not an
# seam went unwired. On a CI runner the pinned toolchain is installed for us by
# .gitea/workflows/scripts/install-tools.sh, so a skip there is a DEFECT, not an
# environment: it BLOCKS. On a workstation a missing tool is a real local condition, so
# it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips()
# in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open.
@@ -257,7 +257,7 @@ for s in lint l0 render-golden l1; do
skip)
if strict_skips; then
err "${s}: skipped (unwired seam / missing tool) under STRICT skips, where the pinned"
err " toolchain is baked into the job image (.gitea/workflows/gate.yml container:) and every"
err " toolchain is installed for us (.gitea/workflows/scripts/install-tools.sh) and every"
err " seam is on main — so this is a DEFECT, not an environment -> blocks merge"
summary "- ${s}: **SKIPPED under strict skips** (unwired seam / missing tool) — blocks merge"
BLOCK=1
+3 -3
View File
@@ -97,9 +97,9 @@ cluster_available() { truthy "${CLUSTER_AVAILABLE:-false}"; }
# --- Skip strictness: is this a CI runner, or a workstation? -------------------
# A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs:
#
# CI the pinned toolchain IS the job image (the CI-built toolbox; gate.yml `container:`),
# and every seam is on `main`. A stage that skips here is a
# DEFECT (a tool dropped from the image, a seam that went unwired) -> BLOCK.
# CI the pinned toolchain is installed by .gitea/workflows/scripts/install-tools.sh
# before the gate runs, and every seam is on `main`. A stage that skips here is a
# DEFECT (a tool dropped from the installer, a seam that went unwired) -> BLOCK.
# local a missing pinned tool is a real, ordinary local condition -> the gate reports
# UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red.
#
+38 -2
View File
@@ -25,8 +25,44 @@ if [ "${#files[@]}" -eq 0 ]; then
exit 0
fi
yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${files[@]}"
ok "yaml/lint passed (${#files[@]} files)"
# yamllint is single-threaded, so the file list is split into LINT_JOBS CONTIGUOUS slices of the
# sorted list, one yamllint per slice, run concurrently. Each slice's output goes to its own file
# and the files are printed in slice order afterwards: the log is the same text, in the same
# file order, as one serial `yamllint "${files[@]}"`, and a finding is never interleaved with
# another process's output (yamllint's standard format puts the FILE NAME on its own line above
# that file's `line:col` findings, so interleaving would misattribute them). The stage's exit
# code is the highest any slice returned (yamllint: 1 = error, 2 = warning under strict), so a
# single failing slice fails the stage exactly as the serial call did. LINT_JOBS=1 is serial.
lint_jobs="${LINT_JOBS:-$(n="$(nproc 2>/dev/null || echo 1)"; [ "$n" -gt 8 ] && n=8; echo "$n")}"
case "$lint_jobs" in ''|*[!0-9]*|0) err "LINT_JOBS must be a positive integer, got '${lint_jobs}'"; exit 2 ;; esac
[ "$lint_jobs" -gt "${#files[@]}" ] && lint_jobs="${#files[@]}"
lint_tmp="$(mktemp -d)"
trap 'rm -rf "$lint_tmp"' EXIT
per=$(( (${#files[@]} + lint_jobs - 1) / lint_jobs ))
pids=()
for (( i = 0; i < lint_jobs; i++ )); do
slice=("${files[@]:$(( i * per )):${per}}")
[ "${#slice[@]}" -gt 0 ] || break
( rc=0
yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${slice[@]}" \
>"${lint_tmp}/${i}.out" 2>&1 || rc=$?
echo "$rc" >"${lint_tmp}/${i}.rc" ) &
pids+=("$!")
done
wait "${pids[@]}" || true
lint_rc=0
for (( i = 0; i < ${#pids[@]}; i++ )); do
cat "${lint_tmp}/${i}.out"
# A slice that left no rc file died before yamllint returned: that is a failure, not a pass.
slice_rc="$(cat "${lint_tmp}/${i}.rc" 2>/dev/null || true)"
case "$slice_rc" in ''|*[!0-9]*) slice_rc=1 ;; esac
[ "$slice_rc" -gt "$lint_rc" ] && lint_rc="$slice_rc"
done
if [ "$lint_rc" -ne 0 ]; then
err "yaml/lint failed (yamllint exit ${lint_rc}; ${#files[@]} files in ${#pids[@]} parallel slice(s))"
exit "$lint_rc"
fi
ok "yaml/lint passed (${#files[@]} files, ${#pids[@]} parallel slice(s))"
# A workflow's `configurationFile:` must name a file that exists. Renovate's did not after #79
# renamed renovate.json -> renovate.json5, and every scheduled run failed with ENOENT for 11 days
+105 -14
View File
@@ -14,7 +14,7 @@ Pure standard library (matches the harness's no-pip-dependency posture); the che
themselves shell out (render checks to catalog/render.py, whose PyYAML dependency lives
there; L0 checks to git + PyYAML).
Usage: run_l1_offline.py --junit <out.xml> [--label-prefix P] <test-dir> [<test-dir> ...]
Usage: run_l1_offline.py --junit <out.xml> [--label-prefix P] [--jobs N] <test-dir> [<test-dir> ...]
Exit 0 iff every selected offline check passes.
The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory
@@ -43,13 +43,35 @@ Exit codes: 0 all selected checks passed / 1 at least one FAILED / 3 nothing fai
scope-declared check was UNWIRED. conformance/run.sh propagates 3 verbatim; ci/l0.sh turns it
into the gate's own rc 3. This is the ONE place the "unwired, not green" rule lives now,
instead of once per hand-written stage.
PARALLELISM (--jobs, env CONF_OFFLINE_JOBS)
-------------------------------------------
The checks are independent processes, so they run in a bounded thread pool (one `bash` child
per worker). Default: min(8, usable CPUs); `--jobs 1` is the old serial runner exactly. Each
check's stdout+stderr is CAPTURED and only reported after every check has finished, in the
ORDER THE DIRS WERE GIVEN, so the log, the JUnit and the step summary are byte-for-byte
independent of which check finished first (only the trailing "[N.Ns]" wall-time on each
status line varies). The 0/1/3 aggregation runs over the same ordered case list as before.
Parallel-safety is a property of the CHECKS, not of this runner: every check must keep its
scratch state in a private mktemp/mkdtemp dir and must not write into the tree or a fixed
path. The audit that established that for the current set is recorded in
decisions/2026-11-08-parallel-l0-checks.md; a new check that needs shared state must say so.
Nested pools: the keycloak reconciler checks run their own mutation suites in a pool
(mutations.py, env MUTATION_JOBS). When this runner is itself parallel it exports a SMALLER
MUTATION_JOBS (unless the caller already set one), so the two suites share the CPUs the
other jobs-2 checks are not using instead of each taking min(8, CPUs). See inner_jobs().
"""
from __future__ import annotations
import argparse
import os
import re
import subprocess
import sys
import time
from concurrent.futures import ThreadPoolExecutor
import xml.etree.ElementTree as ET
from pathlib import Path
from xml.dom import minidom
@@ -97,6 +119,46 @@ def collect_meta(test_yaml: Path, prefix: str):
return name, reqs
def usable_cpus() -> int:
"""CPUs this process may run on (the cgroup/affinity view, not the host's count)."""
try:
return max(1, len(os.sched_getaffinity(0)))
except (AttributeError, OSError):
return max(1, os.cpu_count() or 1)
def positive_int(text: str) -> int:
n = int(text)
if n < 1:
raise argparse.ArgumentTypeError(f"must be >= 1, got {n}")
return n
def inner_jobs(cpus: int, outer: int) -> int:
"""MUTATION_JOBS to hand the checks when this runner runs `outer` of them at once.
Two checks (the keycloak client-secret and passkey reconcilers) fan out internally; the
other outer-2 workers are single processes. Give the two suites an equal share of the CPUs
those others leave free, never fewer than 2 each (a 1-wide pool would make the 40-mutation
passkey suite the critical path of the whole stage), never more than 8. Peak process count
is then about (outer-2) + 2*inner, i.e. <= cpus on any machine with cpus >= outer + 2, and
at worst ~1.25-1.5x cpus on a small runner — CPU-bound oversubscription of that size costs
latency per process, not throughput.
"""
if outer <= 1:
return min(8, cpus)
return max(2, min(8, (cpus - outer + 2) // 2))
def run_check(check: Path, env: dict) -> tuple[int, str, float]:
t0 = time.monotonic()
proc = subprocess.run(
["bash", str(check)], capture_output=True, text=True,
stdin=subprocess.DEVNULL, env=env,
)
return proc.returncode, (proc.stdout + proc.stderr).strip(), time.monotonic() - t0
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--junit", type=Path, required=True)
@@ -108,26 +170,54 @@ def main() -> int:
"(makes this runner exit 3); repeatable")
ap.add_argument("--summary-out", type=Path, default=None,
help="write one Markdown line per case here, for the CI step summary")
ap.add_argument("--jobs", type=positive_int, default=None,
help="checks to run at once (default: env CONF_OFFLINE_JOBS, else "
"min(8, usable CPUs)); 1 = serial")
ap.add_argument("dirs", nargs="+", type=Path)
args = ap.parse_args()
strict_scopes = set(args.unwired_scope)
cases = [] # (name, reqs, dir, rc, output, scope)
cpus = usable_cpus()
if args.jobs is not None:
jobs = args.jobs
elif os.environ.get("CONF_OFFLINE_JOBS", "").strip():
try:
jobs = positive_int(os.environ["CONF_OFFLINE_JOBS"].strip())
except (ValueError, argparse.ArgumentTypeError) as e:
ap.error(f"CONF_OFFLINE_JOBS={os.environ['CONF_OFFLINE_JOBS']!r}: {e}")
else:
jobs = min(8, cpus)
child_env = dict(os.environ)
if not child_env.get("MUTATION_JOBS", "").strip():
child_env["MUTATION_JOBS"] = str(inner_jobs(cpus, jobs))
print(f"[offline] running {len(args.dirs)} check(s), jobs={jobs} "
f"(cpus={cpus}, MUTATION_JOBS={child_env['MUTATION_JOBS']})", file=sys.stderr)
# Metadata first, in the given order; then the checks themselves through the pool. The
# results are consumed in SUBMISSION order, so `cases` is identical to the serial runner's.
planned = [] # (name, reqs, dir, scope, check-or-None)
for d in args.dirs:
check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None)
name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix)
scope = scope_of(d / "chainsaw-test.yaml", args.scope_annotation)
if check is None:
cases.append((name, reqs, d, 3,
f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}",
scope))
continue
proc = subprocess.run(
["bash", str(check)], capture_output=True, text=True
)
out = (proc.stdout + proc.stderr).strip()
cases.append((name, reqs, d, proc.returncode, out, scope))
planned.append((name, reqs, d, scope, check))
cases = [] # (name, reqs, dir, rc, output, scope)
elapsed = [] # wall seconds per case, same order (None: nothing ran)
with ThreadPoolExecutor(max_workers=jobs) as pool:
futures = [pool.submit(run_check, check, child_env) if check is not None else None
for (_n, _r, _d, _s, check) in planned]
for (name, reqs, d, scope, check), fut in zip(planned, futures):
if fut is None:
cases.append((name, reqs, d, 3,
f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}",
scope))
elapsed.append(None)
continue
rc, out, secs = fut.result()
cases.append((name, reqs, d, rc, out, scope))
elapsed.append(secs)
failures = sum(1 for c in cases if c[3] not in (0, 3))
skipped = sum(1 for c in cases if c[3] == 3)
@@ -145,7 +235,7 @@ def main() -> int:
},
)
summary_lines = []
for name, reqs, d, rc, out, scope in cases:
for (name, reqs, d, rc, out, scope), secs in zip(cases, elapsed):
tc = ET.SubElement(
suite,
"testcase",
@@ -164,8 +254,9 @@ def main() -> int:
fail.text = out
status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL")
mark = f" scope={scope}" if scope else ""
took = f" [{secs:.1f}s]" if secs is not None else ""
print(f"[offline] {layer_of(d):3s} {status:4s} {name} "
f"({','.join(reqs) or 'untagged'}){mark}",
f"({','.join(reqs) or 'untagged'}){mark}{took}",
file=sys.stderr)
if rc not in (0, 3):
print(out, file=sys.stderr)
@@ -25,15 +25,19 @@
#
# python3 mutations.py <path-to-extracted-reconciler.py>
#
# Runtime is ~20s (one subprocess per mutation, each running the full selftest
# against an in-process stub Keycloak; no network, no cluster).
# Runtime is ~11s serial (one subprocess per mutation, each running the full selftest
# against an in-process stub Keycloak; no network, no cluster). The selftests run in a
# bounded pool (env MUTATION_JOBS, default min(8, CPUs); 1 = serial); output order and
# verdicts are independent of the pool size.
# =============================================================================
from __future__ import annotations
import os
import re
import subprocess
import sys
import tempfile
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
@@ -136,43 +140,99 @@ MUTATIONS: list[tuple[str, str, str, str]] = [
]
def mutation_jobs() -> int:
"""Selftests to run at once: env MUTATION_JOBS, else min(8, usable CPUs).
conformance/lib/run_l1_offline.py exports a smaller MUTATION_JOBS when it is itself
running several checks in parallel, so the nested pools do not oversubscribe the host.
MUTATION_JOBS=1 reproduces the old serial loop exactly.
"""
raw = os.environ.get("MUTATION_JOBS", "").strip()
if raw:
try:
n = int(raw)
except ValueError:
n = 0
if n < 1:
raise ValueError("MUTATION_JOBS must be a positive integer, got %r" % raw)
return n
try:
cpus = len(os.sched_getaffinity(0))
except (AttributeError, OSError):
cpus = os.cpu_count() or 1
return max(1, min(8, cpus))
def run_selftest(mutated_src: str) -> subprocess.CompletedProcess:
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
fh.write(mutated_src)
path = fh.name
try:
return subprocess.run([sys.executable, path, "--selftest"],
capture_output=True, text=True, timeout=240,
stdin=subprocess.DEVNULL)
finally:
Path(path).unlink(missing_ok=True)
def main() -> int:
if len(sys.argv) != 2:
print("usage: mutations.py <extracted-reconciler.py>", file=sys.stderr)
return 2
src = Path(sys.argv[1]).read_text()
problems: list[str] = []
killed = 0
try:
jobs = mutation_jobs()
except ValueError as e:
print("mutations.py: %s" % e, file=sys.stderr)
return 2
# Pass 1, serial and cheap: every anchor is checked BEFORE anything runs, exactly as the
# serial loop did, and each mutation's source is built here. None = anchor failure.
planned: list[tuple[str, str, str | None, str | None]] = []
for name, why, old, new in MUTATIONS:
old, new = dedent4(old), dedent4(new)
# An anchor that no longer matches is a FAILURE, not a skip: it means the code
# moved and this mutation silently stopped proving anything.
n = src.count(old)
if n != 1:
problems.append("%s: anchor matches %d times (expected exactly 1) -- the "
planned.append((name, why,
"%s: anchor matches %d times (expected exactly 1) -- the "
"code moved and this mutation no longer proves anything. "
"Re-anchor it against the current source." % (name, n))
"Re-anchor it against the current source." % (name, n), None))
continue
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
fh.write(src.replace(old, new, 1))
path = fh.name
try:
proc = subprocess.run([sys.executable, path, "--selftest"],
capture_output=True, text=True, timeout=240)
finally:
Path(path).unlink(missing_ok=True)
if proc.returncode == 0:
problems.append("%s SURVIVED -- --selftest still passed with this regression "
"applied, so nothing in the suite actually checks it. The "
"mistake it models: %s" % (name, why))
continue
killed += 1
out = proc.stdout + proc.stderr
m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
how = ("%s assertion(s) caught it" % m.group(1) if m
else "the selftest aborted (caught, less cleanly)")
print(" killed %-42s %s" % (name, how))
planned.append((name, why, None, src.replace(old, new, 1)))
# Pass 2: the selftests, `jobs` at a time. Each child gets its own temp file, its own
# mkdtemp dirs and its own 127.0.0.1:0 stub port, so they share nothing. Results are
# consumed in MUTATIONS order, so every line below is printed in the same order, with the
# same verdict, as the serial loop produced -- whichever child finishes first.
problems: list[str] = []
killed = 0
pool = ThreadPoolExecutor(max_workers=jobs)
try:
futures = [pool.submit(run_selftest, mutated) if mutated is not None else None
for (_n, _w, _a, mutated) in planned]
for (name, why, anchor_problem, _m), fut in zip(planned, futures):
if anchor_problem is not None:
problems.append(anchor_problem)
continue
proc = fut.result()
if proc.returncode == 0:
problems.append("%s SURVIVED -- --selftest still passed with this regression "
"applied, so nothing in the suite actually checks it. The "
"mistake it models: %s" % (name, why))
continue
killed += 1
out = proc.stdout + proc.stderr
m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
how = ("%s assertion(s) caught it" % m.group(1) if m
else "the selftest aborted (caught, less cleanly)")
print(" killed %-42s %s" % (name, how))
finally:
# A timeout (or any other exception) still propagates and fails the check, as it did
# serially; queued mutations are cancelled rather than run for nothing.
pool.shutdown(wait=True, cancel_futures=True)
print()
if problems:
@@ -40,15 +40,19 @@
#
# python3 mutations.py <path-to-extracted-reconciler.py>
#
# Runtime is ~60s (one subprocess per mutation, each running the full selftest
# against an in-process stub Keycloak; no network, no cluster).
# Runtime is ~85s serial (one subprocess per mutation, each running the full selftest
# against an in-process stub Keycloak; no network, no cluster). The selftests run in a
# bounded pool (env MUTATION_JOBS, default min(8, CPUs); 1 = serial); output order and
# verdicts are independent of the pool size.
# =============================================================================
from __future__ import annotations
import os
import re
import subprocess
import sys
import tempfile
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
# (name, the mistake being modelled, anchor, replacement)
@@ -403,42 +407,98 @@ MUTATIONS: list[tuple[str, str, str, str]] = [
]
def mutation_jobs() -> int:
"""Selftests to run at once: env MUTATION_JOBS, else min(8, usable CPUs).
conformance/lib/run_l1_offline.py exports a smaller MUTATION_JOBS when it is itself
running several checks in parallel, so the nested pools do not oversubscribe the host.
MUTATION_JOBS=1 reproduces the old serial loop exactly.
"""
raw = os.environ.get("MUTATION_JOBS", "").strip()
if raw:
try:
n = int(raw)
except ValueError:
n = 0
if n < 1:
raise ValueError("MUTATION_JOBS must be a positive integer, got %r" % raw)
return n
try:
cpus = len(os.sched_getaffinity(0))
except (AttributeError, OSError):
cpus = os.cpu_count() or 1
return max(1, min(8, cpus))
def run_selftest(mutated_src: str) -> subprocess.CompletedProcess:
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
fh.write(mutated_src)
path = fh.name
try:
return subprocess.run([sys.executable, path, "--selftest"],
capture_output=True, text=True, timeout=300,
stdin=subprocess.DEVNULL)
finally:
Path(path).unlink(missing_ok=True)
def main() -> int:
if len(sys.argv) != 2:
print("usage: mutations.py <extracted-reconciler.py>", file=sys.stderr)
return 2
src = Path(sys.argv[1]).read_text()
problems: list[str] = []
killed = 0
try:
jobs = mutation_jobs()
except ValueError as e:
print("mutations.py: %s" % e, file=sys.stderr)
return 2
# Pass 1, serial and cheap: every anchor is checked BEFORE anything runs, exactly as the
# serial loop did, and each mutation's source is built here. None = anchor failure.
planned: list[tuple[str, str, str | None, str | None]] = []
for name, why, old, new in MUTATIONS:
# An anchor that no longer matches is a FAILURE, not a skip: it means the code
# moved and this mutation silently stopped proving anything.
n = src.count(old)
if n != 1:
problems.append("%s: anchor matches %d times (expected exactly 1) -- the "
planned.append((name, why,
"%s: anchor matches %d times (expected exactly 1) -- the "
"code moved and this mutation no longer proves anything. "
"Re-anchor it against the current source." % (name, n))
"Re-anchor it against the current source." % (name, n), None))
continue
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
fh.write(src.replace(old, new, 1))
path = fh.name
try:
proc = subprocess.run([sys.executable, path, "--selftest"],
capture_output=True, text=True, timeout=300)
finally:
Path(path).unlink(missing_ok=True)
if proc.returncode == 0:
problems.append("%s SURVIVED -- --selftest still passed with this regression "
"applied, so nothing in the suite actually checks it. The "
"mistake it models: %s" % (name, why))
continue
killed += 1
out = proc.stdout + proc.stderr
m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
how = ("%s assertion(s) caught it" % m.group(1) if m
else "the selftest aborted (caught, less cleanly)")
print(" killed %-54s %s" % (name, how))
planned.append((name, why, None, src.replace(old, new, 1)))
# Pass 2: the selftests, `jobs` at a time. Each child gets its own temp file, its own
# mkdtemp dirs and its own 127.0.0.1:0 stub port, so they share nothing. Results are
# consumed in MUTATIONS order, so every line below is printed in the same order, with the
# same verdict, as the serial loop produced -- whichever child finishes first.
problems: list[str] = []
killed = 0
pool = ThreadPoolExecutor(max_workers=jobs)
try:
futures = [pool.submit(run_selftest, mutated) if mutated is not None else None
for (_n, _w, _a, mutated) in planned]
for (name, why, anchor_problem, _m), fut in zip(planned, futures):
if anchor_problem is not None:
problems.append(anchor_problem)
continue
proc = fut.result()
if proc.returncode == 0:
problems.append("%s SURVIVED -- --selftest still passed with this regression "
"applied, so nothing in the suite actually checks it. The "
"mistake it models: %s" % (name, why))
continue
killed += 1
out = proc.stdout + proc.stderr
m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
how = ("%s assertion(s) caught it" % m.group(1) if m
else "the selftest aborted (caught, less cleanly)")
print(" killed %-54s %s" % (name, how))
finally:
# A timeout (or any other exception) still propagates and fails the check, as it did
# serially; queued mutations are cancelled rather than run for nothing.
pool.shutdown(wait=True, cancel_futures=True)
print()
if problems:
@@ -1,104 +0,0 @@
---
date: 2026-11-08
branch: feat/gate-toolbox-container
title: The merge gate runs inside the CI-built toolbox image, pinned by digest — no per-run tool install
---
## What changed
`.gitea/workflows/gate.yml`, job `gate` (job name, the gate step, its env and its
`|| [ "$?" -eq 3 ]` tolerance are unchanged, so the required context `gate / gate (pull_request)`
is unchanged):
- **`container:`** `gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501e…ef81` — the
image `images.yaml` published from `dd94101` (run 789; it is today's `main` and `sha-dd941016478c`
tag). The same image `infra/devcontainer/dev.sh` runs, so local and CI gate on the identical
toolchain. A `# renovate-digest: image=… tag=main` marker sits on the line above, the repo's
existing convention; `infra/gitea/registry-retention.sh` reads the same marker, so this digest
is protected from pruning.
- **`actions/checkout` replaced by a plain-git step.** The toolbox has no Node (measured:
`command -v node` → missing), and `actions/checkout` is a Node action. Chosen over adding Node to
the toolbox because that would need two sequential PRs (Containerfile, then a publish, then the
pin) and would put a JS runtime into the operator toolbox only to run one action. The step
reproduces what checkout@v7 `fetch-depth: 0` actually ran here (read from runs 490 and 495):
every branch to `refs/remotes/origin/*`, tags, plus the event SHA; detached for a PR,
`-B <branch>` for a push; asserts `HEAD == github.sha`. The job token goes to that one fetch via
`GIT_CONFIG_*` env and is **not** persisted in `.git/config` (nothing later needs network git).
It sets **`umask 0022`** first: the PR's first CI run (run 500) failed L0
`prod-baremetal-not-dev-surface` because act runs steps with umask 0000, every checked-out
directory was world-writable, and Ansible (present in the toolbox, absent from the catthehacker
image) then ignored `infra/ansible/ansible.cfg`. Reproduced locally in the pinned image: clone
under umask 0000 → that check fails with the same message; under 0022 → `check OK`. Side
effect worth knowing: the Ansible-driven parts of that check now actually execute in CI.
- **"Install pinned CI toolchain" step removed**, replaced by **"Toolchain is the one
ci/versions.env pins"**: every `*_VERSION` in `ci/versions.env` must equal the image's
`/etc/dev-toolchain-versions`, else `::error::DRIFT` and the job fails. Without this, dropping
the installer would have made `ci/versions.env` an unenforced claim for everything except
promtool (which `alert-rules-unit-tested` already asserts).
- Header comments rewritten for the new job image (pull auth, cold-pull size, no-Node, why
`install-tools.sh` stays).
`renovate.json5`: the `renovate-digest` custom manager now also scans `.gitea/workflows/gate.yml`;
the `github-actions` manager is disabled for this repo's own images (same reason the `kubernetes`
manager already was: it would see a digest-only image and look up `latest`).
Comment/message wording only: `ci/gate.sh` (header + the strict-skip error text), `ci/lib.sh`,
`ci/README.md`, `.gitea/workflows/README.md` — they said the CI toolchain comes from
`install-tools.sh`.
`.gitea/workflows/scripts/install-tools.sh` is **kept**: `merge-gate.yaml` (dormant blueprint)
calls it, and `conformance/tests/l0-static/alert-rules-unit-tested/check.sh` asserts its promtool
wiring (not touched — another agent owns `conformance/`).
**Consequence to know:** every toolbox digest bump is now a Renovate PR touching `.gitea/**`,
i.e. a sensitive path needing an operator `/approve-override`. That is arguably correct (a new
toolchain changes what green means — and `ci/versions.env` was already sensitive), but it is a
new recurring operator touch.
## What I verified
- Registry pull auth: the Home-Network org is `public`; anonymous `/v2/token` → manifest `200`,
`docker-content-digest: sha256:83a3501e…ef81` for both `main` and `sha-dd941016478c`. So no
`container: credentials:` is needed. (The Actions token could not have been used anyway — no
package scope, per images.yaml's header.)
- Image: 466,912,274 bytes compressed across 6 layers (manifest sum), 1,541,728,125 bytes
unpacked (`podman image inspect`). Local cold pull 10.4s on the workstation.
- Image tool versions equal `ci/versions.env` for all 8 pins (kyverno 1.19.0, kyverno-json 0.0.3,
chainsaw 0.2.12, yamllint 1.35.1, kubectl 1.36.4, yq 4.44.6, promtool 3.14.0, check-jsonschema
0.29.4) — both via `/etc/dev-toolchain-versions` and by running each binary's `--version`.
`sleep`, `tail`, `git` 2.56.0, `bash`, `python3`, `jq`, `curl` present (act's entrypoint is
`/bin/sleep 10800`).
- The new checkout script, run in the pinned image against the public repo (auth header disabled
for the test only): PR ref `refs/pull/143/head` → detached at `60e3a24`, 1018 commits; push ref
`refs/heads/main` → `On branch main` at `e8d2666`; `origin/main` present in both. The
toolchain-drift step passes on the real `ci/versions.env` and exits 1 naming both tools when
yamllint/promtool pins are altered.
- Renovate marker regex (copied from renovate.json5) matches gate.yml exactly once, capturing
depName/tag/digest.
- LOCAL GATE, run **in the pinned CI image itself** (`IMAGE=<the digest above>` through
`dev.sh`, `DEV_KUBECONFIG=/dev/null`, ENGINE shim adding the main checkout's `.git` read-only),
on commit `9537ca2`: **rc 3**; spec-immutability, decisions-log (78 entries), lint, l0,
render-golden, l1, approval-policy (N/A locally — not a PR event) all pass; **gitea-settings
the only stage not run** (no `SETTINGS_API_TOKEN` locally); L2/L3 conformance skipped as
usual. rc 3 is not green.
- **CI, this PR's own runs** (Gitea API `/actions/runs/{id}/jobs` + job logs):
- run 500 (`514ed5d`, cold DinD cache: `Image exists? false`, image pulled): job 151s; set-up
incl. pull **9s**, checkout <1s, toolchain check 1s (all 8 pins `ok`), gate 140s → BLOCKED by
L0 `prod-baremetal-not-dev-surface` (umask, fixed above) + approval-policy.
- run 503 (`27d603f`, warm: `Image exists? true`): job **140s**; set-up 1s, checkout 1s,
toolchain check 0s, gate 137s. Every stage passes — l0 (incl. prod-baremetal-not-dev-surface),
lint, decisions-log, render-golden, l1, **gitea-settings** — except **approval-policy**,
the expected sensitive-path block. spec-immutability skipped (expected, no spec repo in CI).
- baseline before (runs 494/495, push on main): job 180s/179s; set-up 1–2s, checkout 1s,
**install 11s**, gate 164–165s. The install step is gone; the remaining ~25s gate-step delta
is NOT claimed as caused by this change (different base commits, one sample each).
- Re-gated locally on `27d603f` in the pinned image: rc 3, gitea-settings the only not-run stage.
## What I did NOT verify
- Renovate actually opening a digest-bump PR for gate.yml — only the regex was tested, not a
Renovate run.
- Push-to-main behaviour in CI (only the PR event runs before merge).
- Whether the gate-step speedup (165s → 137s) is real or run-to-run variance.
- That the cold-pull cost stays ~9s under a busier node or a bigger digest delta; one sample.
@@ -0,0 +1,74 @@
---
date: 2026-11-08
branch: fix/keycloak-operator-crds
title: "Keycloak 26.8.2 operator CrashLoop: install its two new CRDs, raise its memory"
---
## What changed
**Incident.** PR #121 (Renovate) bumped `platform/keycloak/kustomization.yaml`'s three
`keycloak-k8s-resources` URLs 26.0.6 -> 26.8.2. Upstream `kubernetes/` at 26.8.2 holds **four**
CRDs, not two (GitHub contents API, `?ref=26.8.2`):
| file | at 26.0.6 | at 26.8.2 |
|---|---|---|
| `keycloaks.k8s.keycloak.org-v1.yml` | yes | yes |
| `keycloakrealmimports.k8s.keycloak.org-v1.yml` | yes | yes |
| `keycloakoidcclients.k8s.keycloak.org-v1.yml` | — | **new** (v2alpha1) |
| `keycloaksamlclients.k8s.keycloak.org-v1.yml` | — | **new** (v2alpha1) |
| `kubernetes.yml` | yes | yes (now carries RBAC for both new kinds) |
(`cluster-wide/` is a separate variant; upstream's own `kubernetes/kustomization.yml` lists exactly
the five files above.) The 26.8.2 operator starts an informer for every controller it has and dies
on the first missing kind: `Couldn't start informer for keycloaksamlclients.k8s.keycloak.org/v2alpha1
... GET .../namespaces/keycloak/keycloaksamlclients ... Not Found`, exit 1, CrashLoopBackOff. Flux
`platform` (`wait: true`, 5m) sat in health checks and every dependent Kustomization reported
`dependency ... is not ready`. The old 26.0.6 ReplicaSet kept serving (RollingUpdate 25%/25%), so SSO
stayed up.
**Fix.**
- `platform/keycloak/kustomization.yaml`: added the two CRD URLs, each with the same
`# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources` marker (the repo's
custom.regex manager matches all five — checked by running its regex over the file). Header
comment updated to v26.8.2, plus a warning that Renovate cannot add a CRD a release introduces.
- Same file: a `patches:` JSON6902 patch on Deployment `keycloak-operator`: memory limit
450Mi -> **768Mi**, request 450Mi -> **512Mi**. 450Mi is upstream's default (unchanged at 26.8.2,
so not something the bump changed). Evidence it is too small here: the 26.0.6 pod had **154
restarts**, `lastState.terminated.reason: OOMKilled` (exit 137, 2026-11-07T23:09:54Z), and
`kubectl top` showed **421Mi / 450Mi**. 26.8.2 runs four controllers instead of two on Quarkus
3.40.1.1 (was 3.15.1). No variable in the operator to disable watching unused kinds was used —
installing the CRDs is upstream's supported shape.
- `platform/keycloak/README.md` "Upgrading Keycloak": the diff-the-CRD-list step; "three URLs" ->
"the release URLs". `renovate.json5`: same wording fix in a comment only.
## What I verified
- Upstream file lists at 26.0.6 and 26.8.2 (GitHub API) and `diff` of `kubernetes.yml` between the
two: the new kinds' ClusterRoles + RoleBindings ship in it, and the Deployment's resources are
still 450Mi/450Mi.
- Live, read-only (`kubectl --context k3s-prod`, 2026-11-08 ~05:57Z): only `keycloaks` and
`keycloakrealmimports` CRDs installed; Deployment strategy RollingUpdate maxSurge/maxUnavailable
25%; RS `keycloak-operator-58dff6c4` (26.8.2) 0/1 CrashLoopBackOff, RS `78dbb75f64` (26.0.6) 1/1;
operator log is the informer error above; Flux `platform` `Healthy=Unknown` "Running health
checks for revision main@sha1:e8d2666"; no NetworkPolicy and no CiliumNetworkPolicy in ns
`keycloak`.
- No sensitive path touched (`infra/gitea/approval-policy.json`), so this should merge on green.
- Gate: see the PR body for the exact rc and stages of the run against the final commit.
## What I did NOT verify
- **That the 26.8.2 operator becomes Ready after this merges.** Only the two missing kinds were
observed failing; another start-up failure behind them is possible.
- **That 768Mi is enough** — no 26.8.2 memory measurement exists yet; re-check `kubectl top` after
a week.
- **The Keycloak server upgrade 26.0.6 -> 26.8.2 itself.** Once the operator runs it will roll
keycloak-0/1 and Keycloak migrates the DB schema. The CR's `spec.update.strategy` defaults to
`RecreateOnImageChange` — expect a **full SSO outage** while both pods recreate, not a rolling
one. Back up / confirm the latest CNPG backup first if that matters.
- **The 26.8.2 Keycloak CRD's `spec.networkPolicy.enabled` defaults to `true`.** The operator will
create a NetworkPolicy selecting the Keycloak pods (first policy in that namespace). With no
`from` lists set it allows all sources on the HTTP/management ports, so Gateway, ServiceMonitor
and the reconciler Jobs should still reach it — but this was read from the CRD text, not observed.
- **The passkey reconciler's 26.0.6 Admin API assumptions against 26.8.2** (README "Upgrading
Keycloak" list) — not re-checked here; PR #121 merged without it.
- `catalog/keycloak.yaml`'s documentation-only image tag was not updated.
+116
View File
@@ -0,0 +1,116 @@
---
date: 2026-11-08
branch: feat/parallel-l0-checks
title: L0 offline checks, mutation suites and yamllint run in bounded parallel pools (gate 184s -> 32s locally)
---
## What changed
The gate's wall time was dominated by serial loops over independent subprocesses. Three of
them now run in bounded pools; none of them changes a verdict, an exit code, or the order of
anything printed.
- `conformance/lib/run_l1_offline.py` — the per-check loop (L0 and L1 offline checks) runs in
a `ThreadPoolExecutor` of `bash <check>` children. `--jobs N` / env `CONF_OFFLINE_JOBS`,
default `min(8, usable CPUs)` (`sched_getaffinity`, so a cgroup cpuset is respected);
`--jobs 1` is the old serial runner. Results are consumed in SUBMISSION order, so `cases`,
the JUnit, the step summary and the 0/1/3 aggregation are built from exactly the same
ordered list as before. Additive output only: one `[offline] running N check(s), jobs=…`
line, and a trailing `[N.Ns]` wall time on each per-check status line. Children get
`stdin=/dev/null` (parallel children must not share the runner's stdin).
- `conformance/tests/l0-static/keycloak-{passkey,client-secret}-reconciler/mutations.py` —
pass 1 (serial) checks every anchor and builds each mutated source, exactly as before;
pass 2 runs the `--selftest` children in a pool (env `MUTATION_JOBS`, default
`min(8, CPUs)`, `1` = serial). Verdicts are consumed in `MUTATIONS` order. A timeout
(`TimeoutExpired`) still propagates and fails the check; queued mutations are cancelled.
An invalid `MUTATION_JOBS` exits 2 (fails the check, never skips it).
- **Nested-pool budget.** When the outer runner is parallel it exports
`MUTATION_JOBS = max(2, min(8, (cpus - jobs + 2) // 2))` unless the caller already set
one. Reason: only those two checks fan out; the other `jobs-2` workers are single
processes, so the two suites split the CPUs those leave free. Peak processes ≈
`(jobs-2) + 2*inner` ≤ cpus whenever cpus ≥ jobs+2 (24 CPUs: 6 + 16 = 22), and ~1.25–1.5×
cpus on a small runner (4 CPUs: 2 + 4 = 6). The floor of 2 is deliberate: a 1-wide inner
pool would make the 40-mutation passkey suite (~85s serial) the critical path of the stage.
- `ci/lint.sh` — yamllint runs over `LINT_JOBS` (default `min(8, nproc)`) CONTIGUOUS slices
of the sorted file list concurrently; each slice's output is buffered to its own file and
printed in slice order, so the text is the same as one serial call and a `file` header is
never separated from its `line:col` findings. Stage rc = the highest slice rc; a slice that
leaves no rc file counts as 1. Invalid `LINT_JOBS` exits 2.
### Parallel-safety audit (all 25 `l0-static/*/check.sh`, 3 `l1-rendering/*/render-check.sh`)
Grepped every check for fixed `/tmp` paths, `_artifacts`/`ARTIFACTS`/`COVERAGE_OUT`,
`write_text`/`open(…,"w")`, `mktemp`/`mkdtemp`/`TemporaryDirectory`, sockets/ports, `git`
writes and `run.sh` invocations, and read what each writer writes to:
- **Scratch state: all private.** Every writer uses its own `mktemp -d` / `mkdtemp` /
`TemporaryDirectory` (placeholder-guard-coupling, spec-immutability-enforced,
whole-repo-invariants-wired, auditability-proofs-producible, alert-rules-unit-tested and its
`platform/monitoring/tests/run.sh`, internal-services-publish-no-public-dns,
oauth2-proxy-{api-routes-per-app,cookie-names-distinct}, both keycloak reconcilers, the three
L1 render checks). No fixed `/tmp` path anywhere.
- **Git writes:** only spec-immutability-enforced, and only `git init/add/commit` inside its
own `mktemp` repo. Everything else is `git ls-files`/`ls-tree`/`log`/`rev-parse` (read-only).
- **Ports:** the reconciler selftests bind `127.0.0.1:0` (kernel-assigned); the passkey
selftest also dials `127.0.0.1:1` as a deliberate refused connection. No fixed listener.
- **Ansible** (dev-nodes-provisioned-as-code): `--syntax-check`/`--list-hosts` only,
`retry_files_enabled = False`; its local tmp is per-process.
- **The one shared writer:** auditability-proofs-producible runs
`./run.sh --coverage-only --requirements …`, which REWRITES
`conformance/_artifacts/coverage/coverage.{json,md}` (it inherits `CONF_ARTIFACTS`). That was
already true serially. No other offline check reads or writes that directory (the only
reader is `tests/meta/harness-selfcheck`, not an offline check), and the outer runner
writes coverage BEFORE and the JUnit AFTER the pool, so the final state is the same as
serial. Not serialised; left as is and recorded here. `lib/status.py` (run by that check)
reads `_artifacts/report.junit.xml` as an advisory — written only after the pool, unchanged.
- whole-repo-invariants-wired calls `run.sh --print-selection` (writes nothing) and the
runner itself against synthetic dirs in its own `mktemp`.
Result: no check needed fixing or serialising. A new check that needs shared state must say
so; the runner's docstring now states that contract.
## What I verified
All in the pinned toolbox (`dev.sh run`, `DEV_KUBECONFIG=/dev/null`, ENGINE shim adding a ro
mount of the main checkout's `.git`; spec repo resolved as a sibling, spec-immutability
`pass`), host 24 CPUs, every `dev.sh` call under `flock`.
- **Gate, before** (origin/main `1a33814`, detached scratch worktree + spec mount):
`GATE_RC=3`, in-container wall **183.9s**; `l0` stage **168.3s**, `lint` **7.8s**.
"Stages that did not run: gitea-settings" (plus the expected on-base
touched-set-not-determinable note, since main has no branch commits).
- **Gate, after** (this branch, commit `a775082`): `GATE_RC=3`, in-container wall **32.1s**;
`l0` **21.1s**, `lint` **2.4s**, `l1` 2.3s. "Stages that did not run: gitea-settings" — the
only not-run stage. **rc 3, not green.** Caveat: the branch run selected 23 L0 checks (the
touched-requirement scoping dropped dev-nodes-provisioned-as-code and
placeholder-guard-coupling), the main run 25.
- **Same-tree L0 comparison** (all 25 l0 dirs, same tree, same container):
`CONF_OFFLINE_JOBS=1 MUTATION_JOBS=1` **164.0s / 165.3s** vs default (jobs=8,
MUTATION_JOBS=8) **19.2s / 18.7s**. JUnit `cmp`-identical, step summary `cmp`-identical,
stderr identical except the timing suffixes and the JUnit path in the last line.
- **Mutation suites, host:** passkey 83.2s → 10.5s (`MUTATION_JOBS=8`), client-secret
11.2s → 2.0s; output of `MUTATION_JOBS=1` and `=8` byte-identical to the pre-change script's,
rc 0 all round.
- **Falsification (not committed):** in a scratch worktree at `a775082`, made the passkey
mutation "the protected-subtree screen is never called" a no-op (replacement == anchor).
Host: `MUTATION_JOBS=1` and `=8` both rc 1, output byte-identical (with a second, broken
anchor also injected: both reported). Toolbox: runner rc 1 both serial and parallel,
`L0 FAIL keycloak-passkey-reconciler … 1 of 40 mutation(s) not proven … SURVIVED`; full
gate **rc 1**, `stage l0: FAIL`, `MERGE GATE: BLOCKED`. Reverted; scratch worktree removed.
- **lint.sh, host scratch repo** (31 YAML files, two broken): `LINT_JOBS=1/4/8` all rc 1,
stdout identical, findings named `y/f07.yaml` / `2:1 … key-duplicates` etc.; clean tree rc 0;
`LINT_JOBS=x` rc 2.
## What I did NOT verify
- **CI timings at the time of writing.** The PR's CI run is measured after push and reported
on the PR / handoff message, not here. The CI runner's CPU count is UNVERIFIED — the new
`[offline] running … jobs=… cpus=…` line in the CI log is what will show it. On a small
runner the speedup will be smaller than locally.
- **approval-policy** will be red on CI: `ci/**` and `conformance/**` are sensitive paths and
need the operator's review. I did not and must not override it.
- Flakiness under load: the selftests have no wall-clock assertions I found, and 4 toolbox
runs + several host runs were all stable, but that is a handful of runs, not a soak.
- The coverage files under `conformance/_artifacts/coverage/` end in the same state as before
only because a single check rewrites them; I did not change that pre-existing quirk.
- L2/L3 conformance: not run (no cluster in CI, as before).
@@ -0,0 +1,65 @@
---
date: 2026-11-08
branch: chore/renovate-config-cleanup
title: Renovate config — hand-migrate the docker key, track gitea/runner, keep HelmRelease range lower bounds
---
## What changed
- **`renovate.json5`: the deprecated top-level `"docker": {"enabled": true}` is hand-migrated** to
`packageRules[0] = {"matchCategories": ["docker"], "enabled": true}`, the exact shape Renovate's
own migration produced on `origin/renovate/migrate-config` (PR #139). Its comment moved with it.
This replaces #139, which was closed because its rewrite deleted every comment in the file. No
other comment was removed or reflowed (diff is additions plus the moved block).
- **`gitea/runner` added to the blast-radius rule** beside `gitea/act_runner`, with a comment:
act_runner was renamed upstream to gitea/runner after 0.6.1; Renovate does not follow renames,
so the actual image migration is a separate manual PR.
- **Two new packageRules for flux/helm `>=A <B` ranges:** `rangeStrategy: "bump"`, plus
`matchUpdateTypes: ["bump"] -> enabled: false`, so in-range releases (which Flux adopts on its
own) do not become PRs. Cause of the bug, from the 44.167.0 source: rangeStrategy is unset
(`auto`); the flux manager has no `getRangeStrategy`/`updateLockedDependency`, so
`lib/modules/manager/index.ts getRangeStrategy` resolves `auto` to `replace`; helm versioning is
the npm api verbatim, and npm `getNewValue` under `replace` turns `>=1.12.0 <1.13.0` into
`<1.14.0` (what PRs #131, #130 and #115 did).
- **`platform/headlamp/helmrelease.yaml`: `version: "<0.47.0"` -> `">=0.46.0 <0.47.0"`**, comment
refreshed. The live chart is 0.46.0, so the set of installable charts is unchanged.
## What I verified
- **Semantic diff of #139:** I parsed `renovate.json5` on `origin/renovate/migrate-config` and on its
parent `5b484f6` with json5 (the copy inside the renovate image). With `docker` removed from the
parent and `packageRules[0]` removed from the migrated file, the two are identical. So the only
change was `{"matchCategories":["docker"],"enabled":true}`, inserted at packageRules[0].
- **Validator, `ghcr.io/renovatebot/renovate:44.167.0`** (the tag that `.gitea/workflows/renovate.yaml`
pins as `container.image`): `renovate-config-validator` gives `Config validated successfully`
(rc 0), and so does `--strict` (rc 0), both as a file path and in repo mode. **`--strict` on
origin/main's file gives rc 1** and prints the `docker` migration diff. Nothing is left to migrate.
- **`getNewValue` called in the 44.167.0 image** (helm versioning), currentValue
`>=1.12.0 <1.13.0`, newVersion 1.13.0: replace `<1.14.0`, widen `>=1.12.0 <1.14.0`,
bump `>=1.13.0 <1.14.0`, pin/update-lockfile `<1.14.0`. On `<0.47.0`, bump returns the value
unchanged, so it would never propose anything for that range. That is why the rule is scoped to
the `>=A <B` shape.
- **Local dry run** (`--platform=local`, `RENOVATE_DRY_RUN=lookup`, 44.167.0, live chart repos)
against a fixture repo of four HelmReleases under `platform/`:
- With this branch's config: `No config migration necessary`. longhorn `>=1.9.0 <1.10.0` ->
`>=1.13.0 <1.14.0` (minor). kube-prometheus-stack `>=88.0.0 <89.0.0` -> `>=93.1.0 <94.0.0`
(major). The in-range bump `>=88.6.5 <89.0.0` was dropped (`Filtered out 1 disabled update(s)`).
headlamp `>=0.46.0 <0.47.0` and `<0.47.0`: no update.
- Control run with origin/main's config: `Config migration necessary`, longhorn -> `<1.14.0`,
kube-prometheus-stack -> `<94.0.0`. This reproduces the reported bug.
- Live headlamp HelmRelease (read-only kubectl, k3s-prod): spec `<0.47.0`, history chartVersion
`0.46.0`, Ready `True`.
## What I did NOT verify
- **I have not seen a real scheduled Renovate run against this repo with this config.** The claim that
Renovate stops reopening the migrate-config PR rests on the validator and on `No config migration
necessary` in the local dry run, not on the Gitea bot. After merge, watch the next run, and if
`renovate/migrate-config` is still on the remote, delete it.
- **Under bump, the new lower bound is the release Renovate proposes**, e.g. `>=1.13.2 <1.14.0`, not
`>=1.13.0`. Flux still installs the newest release in the window. Ranges with other shapes keep the
default `replace`: `3.9.x`, `~`, ceiling-only, and exact pins.
- **The open PRs #131, #115 and any others still carry unbounded `<X` values.** They were generated
under the old strategy, and Renovate should rebase or regenerate them under bump on its next run.
I have not checked that it does.
- The gitea/runner rename itself (a new image reference and any config changes) is not done here.
+5 -1
View File
@@ -40,7 +40,11 @@ spec:
# values (-session-ttl + the -oidc-* args from the externalSecret; useCookie stays default-false
# so `-oidc-use-cookie` is NOT emitted) is supported by the v0.41.0 backend. Renovate maintains
# within 0.41.x; the appVersion continues to pin the Headlamp image.
version: "<0.47.0"
# 2026-11-08: NOW 0.46.x. PR #130 (Renovate, default rangeStrategy `replace`) rewrote
# `>=0.41.0 <0.42.0` to the unbounded-below `<0.47.0`; the lower bound is restored here at the
# chart that was live when it merged (0.46.0). renovate.json5 now bumps `>=A <B` ranges as a window,
# so the next widen PR proposes `>=0.47.x <0.48.0` rather than `<0.48.0`.
version: ">=0.46.0 <0.47.0"
sourceRef:
kind: HelmRepository
name: headlamp
+9 -1
View File
@@ -367,9 +367,17 @@ is involved — so step 2 failing with no log entry is the expected signature of
The `Keycloak` CR has no `spec.image`, so the server image is whatever the operator's
`RELATED_IMAGE_KEYCLOAK` names — the same tag as the operator (prod, 2026-10-30: operator
`26.0.6`, `keycloak-0/1` on `quay.io/keycloak/keycloak:26.0.6`). Operator, CRDs and server are
released in lockstep from `keycloak/keycloak-k8s-resources`, and the three URLs in
released in lockstep from `keycloak/keycloak-k8s-resources`, and the release URLs in
`kustomization.yaml` are the one knob. Renovate raises them as ONE PR (`keycloak-operator` group,
labels `needs-review` + `blast-radius`), **never auto-merged**; majors wait for a dashboard tick.
**Renovate bumps the URLs that exist; it cannot add a CRD a release introduced.** Before merging a
bump, diff `kubernetes/` (and its `kustomization.yml`) at the new tag against the URL list. 26.0.6
shipped 2 CRDs, 26.8.2 ships 4 (`keycloakoidcclients`, `keycloaksamlclients` added); PR #121 bumped
only the existing URLs and the 26.8.2 operator CrashLooped on the missing `keycloaksamlclients`
informer, stalling Flux `platform` and every tier below it (2026-11-08,
`decisions/2026-11-08-keycloak-operator-crds.md`). The operator's memory is also patched in
`kustomization.yaml` (upstream's 450Mi OOMKilled 26.0.6 here); keep the patch across bumps.
`catalog/keycloak.yaml`'s image tag is documentation only (its render deploys nothing) and
Renovate is disabled for it — update it by hand to match after an operator bump merges.
+37 -3
View File
@@ -20,18 +20,32 @@ kind: Kustomization
namespace: keycloak
resources:
- namespace.yaml
# --- Keycloak Operator v26.0.6 (pinned upstream release manifests) ---
# --- Keycloak Operator v26.8.2 (pinned upstream release manifests) ---
# The operator, its CRDs and the SERVER IMAGE move in lockstep: the Keycloak CR has no
# spec.image, so the operator's RELATED_IMAGE_KEYCLOAK (quay.io/keycloak/keycloak:<same tag>)
# is what runs. These three URLs are therefore THE Keycloak version. Each carries a Renovate
# marker (custom.regex manager, renovate.json5) under ONE depName, so all three bump in ONE
# is what runs. These URLs are therefore THE Keycloak version. Each carries a Renovate
# marker (custom.regex manager, renovate.json5) under ONE depName, so all of them bump in ONE
# branch, never auto-merged (packageRule "keycloak-operator"). Before merging a bump, re-check
# passkey-reconciler.yaml's 26.0.6 Admin API assumptions — see README.md "Upgrading Keycloak".
#
# THE URL LIST MUST MATCH UPSTREAM'S kubernetes/kustomization.yml AT THE NEW TAG — Renovate only
# rewrites the version in the URLs that are here; it cannot add a CRD the new release introduced.
# 26.0.6 shipped 2 CRDs; 26.8.2 ships 4 (+ keycloakoidcclients, keycloaksamlclients, both
# v2alpha1). PR #121 bumped only the existing three URLs, and the 26.8.2 operator — which starts an
# informer for EVERY kind it controls, used or not — CrashLooped on
# `Couldn't start informer for keycloaksamlclients.k8s.keycloak.org/v2alpha1 ... Not Found`,
# stalling Flux `platform` and everything downstream (2026-11-08). On every bump, diff
# https://github.com/keycloak/keycloak-k8s-resources/tree/<tag>/kubernetes
# against this list before merging.
# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloaks.k8s.keycloak.org-v1.yml
# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloakrealmimports.k8s.keycloak.org-v1.yml
# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloakoidcclients.k8s.keycloak.org-v1.yml
# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloaksamlclients.k8s.keycloak.org-v1.yml
# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/kubernetes.yml
# --- Admin (break-glass) credential — SOPS-encrypted, referenced by name (ADR-0007, R-IDP-04) ---
- keycloak-admin-credentials.sops.yaml
@@ -51,3 +65,23 @@ resources:
- keycloak-db-backup-credentials-prod.sops.yaml
# --- Grafana dashboard for Keycloak health (plain ConfigMap; R-OBS-03) ---
- grafana-dashboard.yaml
# --- Operator memory: upstream ships requests=limits=450Mi, which this cluster outgrew ON 26.0.6 ---
# Measured on prod 2026-11-08: the 26.0.6 operator pod had 154 restarts in 64d, lastState
# `OOMKilled` (exit 137, 2026-11-07T23:09:54Z), and sat at 421Mi of its 450Mi (94%) in `kubectl top`.
# 26.8.2 keeps the same 450Mi default but runs FOUR controllers/informers instead of two (adds
# keycloakoidcclient + keycloaksamlclient) on a newer Quarkus (3.15.1 -> 3.40.1.1), so it would
# plausibly start above a ceiling 26.0.6 already crossed. Limit 768Mi = ~1.8x the observed 421Mi
# working set; request 512Mi = the observed working set plus headroom, so the scheduler reserves what
# it actually uses. Revisit with `kubectl top` once 26.8.2 has run for a week. Patched here (not in
# the remote kubernetes.yml) so a Renovate bump keeps it.
patches:
- target:
kind: Deployment
name: keycloak-operator
patch: |-
- op: replace
path: /spec/template/spec/containers/0/resources/limits/memory
value: 768Mi
- op: replace
path: /spec/template/spec/containers/0/resources/requests/memory
value: 512Mi
+67 -16
View File
@@ -37,6 +37,8 @@
// renovate-config-validator /tmp/renovate.json5
// (Not `--strict`: that also fails on a pending migration of the pre-existing top-level
// `"docker": {"enabled": true}` block at the bottom, which is a warning, not an error.)
// RESOLVED 2026-11-08: that block is now hand-migrated to the first packageRule (`matchCategories:
// ["docker"]`), the exact shape Renovate's own migration produced, so no migration is pending.
//
// `regex` vs `custom.regex` in enabledManagers, RESOLVED FROM SOURCE (Renovate 39.264.0 and
// later): both work. lib/config/migrations/custom/enabled-managers-migration.ts rewrites
@@ -142,9 +144,7 @@
"description": "This repo's own CI-published images, pinned by digest, following the tag named in a `# renovate-digest:` marker on the previous line.",
"managerFilePatterns": [
"/(^|/)versions\\.env$/",
"/(^|/)platform/pm-agent/.+\\.ya?ml$/",
// ADDED 2026-11-08: the merge gate's job `container:` is the toolbox, by digest.
"/^\\.gitea/workflows/gate\\.ya?ml$/"
"/(^|/)platform/pm-agent/.+\\.ya?ml$/"
],
"matchStrings": [
"#\\s*renovate-digest:\\s*image=(?<depName>\\S+)\\s+tag=(?<currentValue>\\S+)\\s*\\n[^\\n]*?(?<currentDigest>sha256:[a-f0-9]{64})"
@@ -198,6 +198,22 @@
}
],
"packageRules": [
// The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were
// BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation
// over a running tag. That controller was never enabled and never will be here (see _comment),
// so the only thing those two flags were achieving was leaving every plain `image:` tag in the
// repo unmanaged. `docker` is left out of enabledManagers because this repo contains no
// Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.
// MOVED HERE 2026-11-08 from a top-level `"docker": {"enabled": true}` key, which Renovate
// deprecated (the top-level language objects became `matchCategories` rules). Hand-migrated to
// the exact output of Renovate's own config migration (branch renovate/migrate-config, PR #139:
// a json5 semantic diff of that file against its parent shows THIS rule, inserted at
// packageRules[0], as the ONLY change). #139 itself was closed unmerged because its rewrite
// dropped every comment in this file. With nothing left to migrate, Renovate stops reopening it.
{
"matchCategories": ["docker"],
"enabled": true
},
{
"description": "Auto-merge the allowed low-risk classes on green CI \u2014 no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.",
"matchUpdateTypes": [
@@ -254,6 +270,11 @@
// Keycloak's (the IdP) included, now that imageName is tracked.
"gitea/gitea",
"gitea/act_runner",
// ADDED 2026-11-08. act_runner was RENAMED upstream to gitea/runner after 0.6.1
// (https://gitea.com/gitea/runner/releases). Renovate does not follow a rename — it will
// never propose gitea/act_runner -> gitea/runner — so that migration is a manual PR. Listed
// now so the new name is blast-radius from its first reference, not after an auto-merge.
"gitea/runner",
"ghcr.io/cloudnative-pg/postgresql"
],
"automerge": false,
@@ -266,7 +287,7 @@
// and the server image are released in lockstep from keycloak/keycloak-k8s-resources, and the
// Keycloak CR carries no spec.image, so the operator's RELATED_IMAGE_KEYCLOAK picks the server
// (measured on prod 2026-10-30: operator 26.0.6 -> keycloak-0/1 run quay.io/keycloak/keycloak:
// 26.0.6). The three release URLs in platform/keycloak/kustomization.yaml carry `# renovate:`
// 26.0.6). The release URLs in platform/keycloak/kustomization.yaml carry `# renovate:`
// markers under this ONE depName, so they move together in one branch (groupName makes that
// explicit). NEVER AUTO-MERGED, not even a patch: Keycloak is the IdP that gates SSO for every
// app AND kubectl OIDC, and platform/keycloak/passkey-reconciler.yaml encodes Keycloak 26.0.6
@@ -292,8 +313,8 @@
"enabled": false
},
{
"description": "ADDED 2026-10-29. This repo's own images are followed ONLY through the `# renovate-digest:` markers above. The `kubernetes` manager also sees platform/pm-agent/deployment.yaml and infra/agent/worker-job-template.yaml, where the image has a digest and no tag; left on, it would look those up as `latest` (a tag CI never pushes) and, for the template, re-open the self-referential loop the custom manager deliberately avoids. `github-actions` ADDED 2026-11-08 for the same reason: it extracts the gate job's digest-only `container: image:` in .gitea/workflows/gate.yml, which the custom manager already follows through its marker.",
"matchManagers": ["kubernetes", "github-actions"],
"description": "ADDED 2026-10-29. This repo's own images are followed ONLY through the `# renovate-digest:` markers above. The `kubernetes` manager also sees platform/pm-agent/deployment.yaml and infra/agent/worker-job-template.yaml, where the image has a digest and no tag; left on, it would look those up as `latest` (a tag CI never pushes) and, for the template, re-open the self-referential loop the custom manager deliberately avoids.",
"matchManagers": ["kubernetes"],
"matchPackageNames": [
"gitea.home.keithjackson.dev/home-network/toolbox",
"gitea.home.keithjackson.dev/home-network/agent-runner",
@@ -370,6 +391,45 @@
"automerge": false,
"addLabels": ["needs-review", "range-widen"]
},
// ADDED 2026-11-08 — KEEP THE LOWER BOUND WHEN A MINOR-LOCKED RANGE MOVES. MEASURED PROBLEM:
// PRs #131 (longhorn), #130 (headlamp) and #115 (kube-prometheus-stack) turned
// `">=1.12.0 <1.13.0"` into `"<1.14.0"` — an UNBOUNDED-below range that also admits every
// older chart (and #130 merged: platform/headlamp/helmrelease.yaml carried `<0.47.0`). WHY,
// from the 44.167.0 source this file is validated against: no rangeStrategy is set, so it is
// `auto`; the flux manager has no getRangeStrategy and no updateLockedDependency, so
// lib/modules/manager/index.ts getRangeStrategy resolves `auto` -> `replace`. The helm
// datasource's defaultVersioning is `helm`, which is the npm versioning api verbatim
// (lib/modules/versioning/helm/index.ts: `api = { ...npm }`), and npm getNewValue under
// `replace` rewrites `>=A <B` to just `<C`. Calling that getNewValue in the 44.167.0 image for
// currentValue `>=1.12.0 <1.13.0`, newVersion 1.13.0:
// replace -> `<1.14.0` widen -> `>=1.12.0 <1.14.0` bump -> `>=1.13.0 <1.14.0`
// Only `bump` keeps a one-minor window. (widen keeps the bound but grows the window by a minor
// on every PR — the opposite of minor-locking.) Under bump the new lower bound is the release
// proposed (e.g. 1.13.2 -> `>=1.13.2 <1.14.0`); Flux still picks the newest that satisfies it.
// Scoped to the bounded `>=A <B` shape only: on a ceiling-only range like `<0.47.0` bump
// returns the value unchanged, so Renovate would silently propose NOTHING for it, ever.
{
"description": "ADDED 2026-11-08. HelmRelease `version: \">=A <B\"` ranges move as a whole window (rangeStrategy bump), so a range-widen PR yields `>=1.13.0 <1.14.0`, not `<1.14.0`. See the comment above for the measured cause.",
"matchManagers": ["flux"],
"matchDatasources": ["helm"],
"matchCurrentValue": "/^>=\\S+ <\\S+$/",
"rangeStrategy": "bump"
},
// ...AND DO NOT LET bump ADD NOISE. Under bump Renovate also raises the lower bound for a
// release ALREADY INSIDE the range (lib/workers/repository/process/lookup/generate.ts sets
// isBump; e.g. 1.12.5 against `>=1.12.0 <1.13.0` -> `>=1.12.5 <1.13.0`), which `replace` never
// proposed. Flux already installs that release with no PR, and the range-widen rule above
// would hold every one for review. util/package-rules/update-types.ts matches isBump as
// updateType `bump`, so exactly those updates are switched off here; out-of-range moves are
// ordinary minor/major updates and are unaffected.
{
"description": "ADDED 2026-11-08. No PR for a chart release the HelmRelease range already admits — Flux adopts it on its own.",
"matchManagers": ["flux"],
"matchDatasources": ["helm"],
"matchCurrentValue": "/^>=\\S+ <\\S+$/",
"matchUpdateTypes": ["bump"],
"enabled": false
},
{
"description": "ADDED 2026-10-30. velero-plugin-for-aws is pinned to a RELEASE CANDIDATE (v1.14.3-rc.2) and Renovate proposed NOTHING for it, although v1.14.3 and v1.14.4 are stable on Docker Hub. Docker versioning treats everything after the first `-` as a compatibility SUFFIX, so only other `*-rc.2` tags qualify and the RC is a fossil forever. Semver versioning reads it as a prerelease, and a prerelease upgrades to the stable release.",
"matchPackageNames": ["velero/velero-plugin-for-aws"],
@@ -441,14 +501,5 @@
"conformance/tests/**",
"platform/image-automation/per-service/**",
"platform/kyverno/tests/**"
],
// The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were
// BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation
// over a running tag. That controller was never enabled and never will be here (see _comment),
// so the only thing those two flags were achieving was leaving every plain `image:` tag in the
// repo unmanaged. `docker` is left out of enabledManagers because this repo contains no
// Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.
"docker": {
"enabled": true
}
]
}