From d9571fc6d8732cd64d33c2babebfaa0d8abe5ded Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Fri, 30 Oct 2026 02:35:33 -0500 Subject: [PATCH] fix(renovate): edit catalog sources, track CNPG imageName, honour update.range, guard real-run auto-merges Audit of the first completed dry runs (353/354) plus a debug-level local reproduction: - App-image updates edited only the generated apps/wave-*. A ServiceDefinition has no apiVersion, so the kubernetes manager extracts nothing from catalog/, and every such PR would fail render-golden (reproduced: GOLDEN DRIFT). New regex managers for catalog source.image and render.py OAUTH2_PROXY_IMAGE put the source and generated edits in one branch (render-golden rc 0, reproduced). - CNPG Cluster imageName was untracked (3 Clusters), while the grant Job was tracked. New manager. - allowedVersions mirror catalog update.range: this drops gitea 28.x, qbittorrent 20.x, radarr 6, nextcloud 35 and postgresql 18. - Never auto-merge: gitea, act_runner, postgresql (blast radius), flux range widening, python minors. Majors wait for a Dependency Dashboard tick. - velero plugin RC: semver versioning (v1.14.3-rc.2 -> v1.14.4 now proposed). - flux docker duplicates disabled (the "Cannot find replaceString" line). Two false comments corrected. Local dry run: regex deps 7 -> 20, branches 33 -> 29. The validator passes on 44.167.0. Co-Authored-By: Claude Opus 5.5 --- decisions/2026-10-30-renovate-run-hygiene.md | 128 +++++++++++++++++++ renovate.json5 | 109 +++++++++++++++- 2 files changed, 232 insertions(+), 5 deletions(-) create mode 100644 decisions/2026-10-30-renovate-run-hygiene.md diff --git a/decisions/2026-10-30-renovate-run-hygiene.md b/decisions/2026-10-30-renovate-run-hygiene.md new file mode 100644 index 0000000..1425691 --- /dev/null +++ b/decisions/2026-10-30-renovate-run-hygiene.md @@ -0,0 +1,128 @@ +--- +date: 2026-10-30 +branch: fix/renovate-run-hygiene +title: Renovate's first completed dry runs, audited; it would have opened red PRs for every app and missed the databases. Config fixed +--- + +## What changed + +**`renovate.json5` only.** Each change carries a comment quoting the measurement behind it. + +1. **Three new regex managers, so updates edit the SOURCE.** + - `catalog/*.yaml` `source.image`. + - `catalog/render.py` `OAUTH2_PROXY_IMAGE`. + - CNPG `imageName:` under `platform/` and `apps/`. + + The first two match the rendered line's depName, so the catalog edit and the generated + `apps/wave-*` edit land in **one** branch. +2. **`allowedVersions` mirroring each ServiceDefinition's `update.range`:** + - gitea/gitea `<2.0.0` + - nextcloud `<32.0.0` + - qbittorrent `<6.0.0` + - radarr `<6.0.0` + - sonarr `<5.0.0` + - semaphore `<3.0.0` + + Plus CNPG postgresql `<17.0.0`, because a Postgres major is a migration, not a bump. +3. **The blast-radius never-automerge list gains** `gitea/gitea`, `gitea/act_runner` and + `ghcr.io/cloudnative-pg/postgresql`. +4. **HelmRelease range widening** (`matchManagers: flux`, ranged currentValue) is never + auto-merged and is labelled `range-widen`. +5. **Majors need a Dependency Dashboard tick** (`dependencyDashboardApproval`) before a PR opens. +6. **`flux` manager docker deps are disabled.** These are duplicates of `kubernetes` manager + extractions. +7. **`velero/velero-plugin-for-aws` uses `semver` versioning**, so its RC pin can reach a stable + release. +8. **`python` minors are not auto-merged.** +9. **Two false comments are corrected.** + - Ranges "cap Renovate at patches": false, Renovate widens them. + - `nextcloud:31-apache` and `docker:27-dind` "cannot be maintained": false, they were. + +PR #85 is merged, so nothing here overlaps an open branch. + +### The audit: dispatch runs 353 (07:12 UTC) and 354 (07:23 UTC), on 593d0fa + +Both runs ended `success`, Renovate `"result": "done"`, `exitCode 0`. Both were at `info` level, so +I reproduced the per-dependency detail with a debug-level local dry run of the same commit (see +below). + +The runs were inside the 03:00–06:00 America/New_York window, so branches were processed: +- run 353: 37 "Would commit" branches; +- run 354: 40, because the operator had added `RENOVATE_GITHUB_COM_TOKEN` 9 s before it. + +Gitea's router log for the act-runner pod during both runs shows 317 2xx responses. The only +non-2xx were all expected: +- `/v2/` 401: the registry challenge; +- `/admin/orgs` 403: the #85 scope probe, by design; +- 6× `HEAD …/manifests/sha256:000…` 404: the placeholder digests. + +| # | Class | Count | Cause | Severity | Fix | +|---|---|---|---|---|---| +| 1 | App-image PRs edit only generated `apps/wave-*` | 9 branches | A ServiceDefinition has no `apiVersion`, so the `kubernetes` manager extracts 0 deps from `catalog/`. The oauth2-proxy sidecar is a Python constant (`catalog/render.py:132`) | **Real problem.** Every such PR fails `ci/render-golden.sh` (reproduced: rc 1 "GOLDEN DRIFT"), stays red and holds a PR slot | git (1) | +| 2 | CNPG `imageName` untracked | 3 Clusters | The `kubernetes` manager reads `image:` only. The one tracked postgres line was the grant Job (`nextcloud-db.yaml:303`, "keep in step with spec.imageName") | **Real problem.** A real run bumps the Job and leaves the databases behind | git (1) | +| 3 | Bogus or out-of-policy majors | 5 branches: gitea 28.2.0, qbittorrent 20.04.1, radarr 6.x, nextcloud 35, postgresql 18.6 | Odd legacy tags, plus `update.range` in `catalog/` not being read by Renovate | Risk if run for real: unmergeable PRs, which hold slots | git (2) | +| 4 | Auto-merge of the merge gate's own host | gitea 1.24.3→1.27.3, act_runner 0.2.11→0.6.1, both "minor" | Not in the blast-radius list. A 0.x minor is breaking by semver | **Risk if run for real** | git (3) | +| 5 | Range widening auto-merged | external-dns, headlamp (minor); longhorn (blast radius); kube-prometheus-stack (major) | Renovate widens `>=a .yaml` + `apps/wave-*/.yaml`. + - **The oauth2-proxy branch** carries `catalog/render.py` plus the 5 rendered apps and + `platform/crafty`. + - **The postgresql-16.x branch** carries all three Clusters. + - The 2 flux docker duplicates report `disabled`. +- **render==golden holds for the new branch shape.** + - In the toolbox, I applied the gitea branch edit (catalog + apps) and the oauth2-proxy branch + edit (`render.py` constant + 5 apps) textually. `./ci/render-golden.sh` reported + `all 10 onboarded service golden(s) reproduce`, rc 0. + - Negative control, the old shape with only `apps/wave-0/gitea.yaml` bumped: rc 1, + `GOLDEN DRIFT`. + - Both edits were reverted afterwards. + +## What I did NOT verify + +- **The effect of the automerge, label and dashboard-approval rules (3, 4, 5, 8). UNVERIFIED in a + run.** `--platform=local` stops before branch processing, so I saw only which updates exist, not + their merged flags. The matchers are standard (`matchPackageNames`, `matchManagers`, + `matchCurrentValue`, `matchUpdateTypes`). The next dispatch's debug log shows them per branch. +- **`allowedVersions` under docker versioning** used Renovate's "Falling back to npm semver + syntax" path (16 debug lines). That it filters is shown by the five removed branches. +- **render==golden for the other app branches** (qbittorrent, radarr, sonarr, semaphore, seerr). I + proved it only for gitea and oauth2-proxy. They have the same shape: one `source.image` line ↔ + one rendered `image:` line. +- **The `allowedVersions` rules duplicate `catalog/*/update.range`** and can drift from it. Nothing + checks them against each other. A lint comparing the two is the obvious follow-up. +- **Any real (non-dry) run.** None has happened. PR creation, labels, auto-merge through Gitea's + API and the Dependency Dashboard issue are all unobserved. +- **Sensitive path?** `renovate.json5` is not in `infra/gitea/approval-policy.json`, as far as I + know. CI's `approval-policy` result on the PR is the measurement. diff --git a/renovate.json5 b/renovate.json5 index 898c75e..08c9d2d 100644 --- a/renovate.json5 +++ b/renovate.json5 @@ -149,6 +149,50 @@ ], "datasourceTemplate": "docker", "versioningTemplate": "docker" + }, + // ADDED 2026-10-30 — THE CATALOG IS THE SOURCE; apps/wave-*/*.yaml IS GENERATED FROM IT. + // First real dry run (dispatch runs 353/354): EVERY app-image update edited only the rendered + // apps/wave-*/*.yaml and NONE edited catalog/. The `kubernetes` manager lists catalog/ above + // but extracts nothing there: a ServiceDefinition has no `apiVersion`, so it is not a + // Kubernetes manifest to that manager (measured: 0 deps from catalog/). Every such PR would + // fail ci/render-golden.sh (the canonical render reverts the bump), sit red and hold a + // prConcurrentLimit slot. These two managers read the SOURCES, under the SAME depName as the + // rendered line, so both land in ONE branch and render==golden holds: `source.image` in each + // ServiceDefinition, and render.py's OAUTH2_PROXY_IMAGE (the six auth sidecars). + { + "customType": "regex", + "description": "ServiceDefinition `source.image: :` (catalog/*.yaml) — the source of the rendered apps/wave-* image.", + "managerFilePatterns": ["/(^|/)catalog/[^/]+\\.ya?ml$/"], + "matchStrings": [ + "\\n\\s+image:\\s*(?[^\\s:@#\"']+):(?[^\\s@#\"']+)" + ], + "datasourceTemplate": "docker" + }, + { + "customType": "regex", + "description": "catalog/render.py OAUTH2_PROXY_IMAGE — the per-app oauth2-proxy sidecar every forward-auth app is rendered with.", + "managerFilePatterns": ["/(^|/)catalog/render\\.py$/"], + "matchStrings": [ + "\\nOAUTH2_PROXY_IMAGE\\s*=\\s*\"(?[^\":@]+):(?[^\"@]+)\"" + ], + "datasourceTemplate": "docker" + }, + // ADDED 2026-10-30 — CNPG `Cluster.spec.imageName`. The `kubernetes` manager reads `image:` only, + // so the three database Clusters (platform/keycloak/postgres-cluster.yaml, apps/wave-0/ + // nextcloud-db.yaml, apps/wave-0/semaphore-db.yaml) were untracked, while the grant Job in + // nextcloud-db.yaml — whose comment says "keep the tag in step with spec.imageName" — WAS. A real + // run would have bumped the Job and left the database behind. Same depName, so one branch. + { + "customType": "regex", + "description": "CloudNativePG Cluster `imageName: :`.", + "managerFilePatterns": [ + "/(^|/)platform/.+\\.ya?ml$/", + "/(^|/)apps/.+\\.ya?ml$/" + ], + "matchStrings": [ + "\\n\\s+imageName:\\s*(?[^\\s:@#\"']+):(?[^\\s@#\"']+)" + ], + "datasourceTemplate": "docker" } ], "packageRules": [ @@ -200,7 +244,15 @@ "longhorn", "cilium", "cloudnative-pg", - "kube-prometheus-stack" + "kube-prometheus-stack", + // ADDED 2026-10-30. gitea and act_runner ARE the merge gate: a bad auto-merged bump takes + // down the very check that would have to approve its revert (the dry run proposed gitea + // 1.24.3 -> 1.27.3 and act_runner 0.2.11 -> 0.6.1, both as auto-mergeable "minor"; a 0.x + // minor is a breaking line by semver). The postgresql image restarts every CNPG database, + // Keycloak's (the IdP) included, now that imageName is tracked. + "gitea/gitea", + "gitea/act_runner", + "ghcr.io/cloudnative-pg/postgresql" ], "automerge": false, "addLabels": [ @@ -227,6 +279,51 @@ "gitea.home.keithjackson.dev/home-network/conformance-runner" ], "addLabels": ["own-image"] + }, + { + "description": "ADDED 2026-10-30. The `flux` manager also extracts `image:` lines inside HelmRelease values, which the `kubernetes` manager already owns. Measured: exactly two (alpine in platform/oauth2-proxy/helmrelease.yaml, velero-plugin-for-aws in platform/velero/helmrelease.yaml), each extracted twice; the second rewrite of the same line logged `Cannot find replaceString in current file content` in runs 353/354. Flux keeps its charts (datasource helm).", + "matchManagers": ["flux"], + "matchDatasources": ["docker"], + "enabled": false + }, + { + "description": "ADDED 2026-10-30. MIRRORS catalog/.yaml `update.range` — the operator's declared per-service policy (schemas/servicedefinition.schema.json), which Renovate does not read. Without it the dry run proposed gitea 28.2.0, qbittorrent 20.04.1 (both stale/odd tags, not real successors), radarr 6.x and nextcloud 35 (four majors past 31; Nextcloud cannot skip majors). KEEP IN STEP with the catalog ranges.", + "matchPackageNames": ["gitea/gitea"], + "allowedVersions": "<2.0.0" + }, + { "matchPackageNames": ["nextcloud"], "allowedVersions": "<32.0.0" }, + { "matchPackageNames": ["lscr.io/linuxserver/qbittorrent"], "allowedVersions": "<6.0.0" }, + { "matchPackageNames": ["lscr.io/linuxserver/radarr"], "allowedVersions": "<6.0.0" }, + { "matchPackageNames": ["lscr.io/linuxserver/sonarr"], "allowedVersions": "<5.0.0" }, + { "matchPackageNames": ["semaphoreui/semaphore"], "allowedVersions": "<3.0.0" }, + { + "description": "ADDED 2026-10-30. A CNPG Postgres MAJOR is a migration (new cluster + import), not an image bump; the dry run proposed 16.4 -> 18.6. Minors stay (and are blast-radius, above).", + "matchPackageNames": ["ghcr.io/cloudnative-pg/postgresql"], + "allowedVersions": "<17.0.0" + }, + { + "description": "ADDED 2026-10-30. A HelmRelease semver RANGE is a policy ceiling, and Renovate does not respect it — it WIDENS it (dry run: external-dns `<1.22.0` -> `<1.24.0`, headlamp `<0.42.0` -> `<0.47.0`), and the default rule would then auto-merge that as a 'minor', silently letting Flux adopt every new minor. Raising a ceiling is deliberate per-component work: propose it, never auto-merge it.", + "matchManagers": ["flux"], + "matchCurrentValue": "/^[<>=~^]|\\.x$/", + "automerge": false, + "addLabels": ["needs-review", "range-widen"] + }, + { + "description": "ADDED 2026-10-30. velero-plugin-for-aws is pinned to a RELEASE CANDIDATE (v1.14.3-rc.2) and Renovate proposed NOTHING for it, although v1.14.3 and v1.14.4 are stable on Docker Hub. Docker versioning treats everything after the first `-` as a compatibility SUFFIX, so only other `*-rc.2` tags qualify and the RC is a fossil forever. Semver versioning reads it as a prerelease, and a prerelease upgrades to the stable release.", + "matchPackageNames": ["velero/velero-plugin-for-aws"], + "versioning": "semver" + }, + { + "description": "ADDED 2026-10-30. A python MINOR is a language-runtime release (the dry run proposed 3.13 -> 3.15 for the Keycloak reconcilers), and the gate does not execute those scripts. Review it; patches still auto-merge.", + "matchPackageNames": ["python"], + "matchUpdateTypes": ["minor"], + "automerge": false, + "addLabels": ["needs-review"] + }, + { + "description": "ADDED 2026-10-30. MAJORS WAIT FOR A TICK ON THE DEPENDENCY DASHBOARD instead of opening a PR. They are never auto-merged anyway, and with prConcurrentLimit 5 the dry run's 10 majors would hold every PR slot open indefinitely, starving the patch/minor/digest PRs that can merge on green.", + "matchUpdateTypes": ["major"], + "dependencyDashboardApproval": true } ], // WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A @@ -235,10 +332,12 @@ // lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak // (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete // version before any automation can maintain them, and until then they drift on every pull with - // nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12, - // e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it - // has been told not to leave; raising those ceilings is deliberate per-component work and is - // NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md. + // nothing recording what is running. CORRECTED 2026-10-30: nextcloud:31-apache and docker:27-dind + // ARE maintained (the first dry run proposed 35-apache and 29-dind; the suffix is kept). + // Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on + // kube-prometheus-stack): THIS FILE USED TO SAY they cap Renovate at patches. They do not — + // Renovate WIDENS the range (runs 353/354). The `range-widen` rule above keeps that a reviewed + // PR rather than an auto-merge. Both lists are in decisions/2026-10-17-automated-upgrades.md. // platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest` // and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed // shape, not software to keep current. `conformance/**` was already ignored for the same -- 2.54.0