Merge pull request 'fix(renovate): edit catalog sources, track CNPG imageName, honour update.range, guard real-run auto-merges' (#86) from fix/renovate-run-hygiene into main

Reviewed-on: #86

decisions/2026-10-30-renovate-run-hygiene.md

@@ -0,0 +1,128 @@
+---
+date: 2026-10-30
+branch: fix/renovate-run-hygiene
+title: Renovate's first completed dry runs, audited; it would have opened red PRs for every app and missed the databases. Config fixed
+---
+
+## What changed
+
+**`renovate.json5` only.** Each change carries a comment quoting the measurement behind it.
+
+1. **Three new regex managers, so updates edit the SOURCE.**
+   - `catalog/*.yaml` `source.image`.
+   - `catalog/render.py` `OAUTH2_PROXY_IMAGE`.
+   - CNPG `imageName:` under `platform/` and `apps/`.
+
+   The first two match the rendered line's depName, so the catalog edit and the generated
+   `apps/wave-*` edit land in **one** branch.
+2. **`allowedVersions` mirroring each ServiceDefinition's `update.range`:**
+   - gitea/gitea `<2.0.0`
+   - nextcloud `<32.0.0`
+   - qbittorrent `<6.0.0`
+   - radarr `<6.0.0`
+   - sonarr `<5.0.0`
+   - semaphore `<3.0.0`
+
+   Plus CNPG postgresql `<17.0.0`, because a Postgres major is a migration, not a bump.
+3. **The blast-radius never-automerge list gains** `gitea/gitea`, `gitea/act_runner` and
+   `ghcr.io/cloudnative-pg/postgresql`.
+4. **HelmRelease range widening** (`matchManagers: flux`, ranged currentValue) is never
+   auto-merged and is labelled `range-widen`.
+5. **Majors need a Dependency Dashboard tick** (`dependencyDashboardApproval`) before a PR opens.
+6. **`flux` manager docker deps are disabled.** These are duplicates of `kubernetes` manager
+   extractions.
+7. **`velero/velero-plugin-for-aws` uses `semver` versioning**, so its RC pin can reach a stable
+   release.
+8. **`python` minors are not auto-merged.**
+9. **Two false comments are corrected.**
+   - Ranges "cap Renovate at patches": false, Renovate widens them.
+   - `nextcloud:31-apache` and `docker:27-dind` "cannot be maintained": false, they were.
+
+PR #85 is merged, so nothing here overlaps an open branch.
+
+### The audit: dispatch runs 353 (07:12 UTC) and 354 (07:23 UTC), on 593d0fa
+
+Both runs ended `success`, Renovate `"result": "done"`, `exitCode 0`. Both were at `info` level, so
+I reproduced the per-dependency detail with a debug-level local dry run of the same commit (see
+below).
+
+The runs were inside the 03:00–06:00 America/New_York window, so branches were processed:
+- run 353: 37 "Would commit" branches;
+- run 354: 40, because the operator had added `RENOVATE_GITHUB_COM_TOKEN` 9 s before it.
+
+Gitea's router log for the act-runner pod during both runs shows 317 2xx responses. The only
+non-2xx were all expected:
+- `/v2/` 401: the registry challenge;
+- `/admin/orgs` 403: the #85 scope probe, by design;
+- 6× `HEAD …/manifests/sha256:000…` 404: the placeholder digests.
+
+| # | Class | Count | Cause | Severity | Fix |
+|---|---|---|---|---|---|
+| 1 | App-image PRs edit only generated `apps/wave-*` | 9 branches | A ServiceDefinition has no `apiVersion`, so the `kubernetes` manager extracts 0 deps from `catalog/`. The oauth2-proxy sidecar is a Python constant (`catalog/render.py:132`) | **Real problem.** Every such PR fails `ci/render-golden.sh` (reproduced: rc 1 "GOLDEN DRIFT"), stays red and holds a PR slot | git (1) |
+| 2 | CNPG `imageName` untracked | 3 Clusters | The `kubernetes` manager reads `image:` only. The one tracked postgres line was the grant Job (`nextcloud-db.yaml:303`, "keep in step with spec.imageName") | **Real problem.** A real run bumps the Job and leaves the databases behind | git (1) |
+| 3 | Bogus or out-of-policy majors | 5 branches: gitea 28.2.0, qbittorrent 20.04.1, radarr 6.x, nextcloud 35, postgresql 18.6 | Odd legacy tags, plus `update.range` in `catalog/` not being read by Renovate | Risk if run for real: unmergeable PRs, which hold slots | git (2) |
+| 4 | Auto-merge of the merge gate's own host | gitea 1.24.3→1.27.3, act_runner 0.2.11→0.6.1, both "minor" | Not in the blast-radius list. A 0.x minor is breaking by semver | **Risk if run for real** | git (3) |
+| 5 | Range widening auto-merged | external-dns, headlamp (minor); longhorn (blast radius); kube-prometheus-stack (major) | Renovate widens `>=a <b`. The config's comment claimed the opposite | **Risk if run for real:** silently lets Flux adopt new minors | git (4, 9) |
+| 6 | Majors hold PR slots | 10 majors in run 354 against `prConcurrentLimit: 5` | Majors never auto-merge, so they stay open | Risk: starves auto-mergeable PRs | git (5) |
+| 7 | `Cannot find replaceString … alpine:3.22` (INFO) | 1 per run | The same HelmRelease `image:` line is extracted by `flux` and `kubernetes` (also the velero plugin) | Cosmetic. The branch is still right | git (6) |
+| 8 | velero-plugin-for-aws stuck on `v1.14.3-rc.2` | 1 dep, 0 updates | Docker versioning treats `-rc.2` as a suffix, so stable `v1.14.3` and `v1.14.4` never qualify | **Real problem** (silent) | git (7) |
+| 9 | python 3.13→3.15 as auto-mergeable "minor" | 2 files | Language runtime minor | Risk if run for real | git (8) |
+| 10 | `WARN GitHub token is required` | run 353: 5 packages, 17 deps | No github.com token | Real (they were unmaintained) | **operator, done:** run 354 has no WARN and adds `pin-dependencies`, `kubevirt-cdi`, `actions-checkout-7.x`, `major-github-artifact-actions` |
+| 11 | Floating or untagged images (`invalid-value`) | 4: jackett, plex `latest`; pihole, keycloak no tag | No version to compare | Real, already documented | none in git. Pinning by hand is an operator/PM decision |
+| 12 | `runs-on: ubuntu-latest` `invalid-version` | 11 | The github-actions manager extracts runner labels | Cosmetic, debug-only | none |
+| 13 | `Docker Manifest is unknown` (debug) | 3 | HEAD on the all-zero placeholder digests | Expected. Clears when the own-image digest PRs merge | none |
+| 14 | `Docker: error fetching data from DockerHub` 403 (debug) | 5 | Hub refuses `page=11`. Renovate already has the tags | Cosmetic. Updates were still found | none |
+| 15 | `Config migration necessary` | 1 | Top-level `"docker": {"enabled": true}` | Cosmetic, pre-existing | none (as before) |
+
+**Checked and fine:**
+- The 3 own-image digest updates were found in both runs. All 4 `# renovate-digest:` markers parse.
+- No marker failed to parse.
+- No package file is under `conformance/runs` or `tests`, nor is `DECISIONS-LOG.md`, `decisions/` or
+  any `*.sops.yaml` (77 package files checked).
+- `kubevirt-cdi` grouping works.
+- The schedule window behaved as designed.
+- Vulnerability alerts: none in either log.
+
+## What I verified
+
+- I read both job logs in full (132 and 129 lines), and the router log for 10.0.0.38 during both
+  runs.
+- **`renovate-config-validator` on 44.167.0**: `Config validated successfully`. The migration diff
+  is only the pre-existing `docker` block.
+- **Local dry runs.** Pinned image, `--platform=local`, debug, origin/main snapshot. "Before" is
+  593d0fa and "after" is this config; the GitHub-hosted deps were skipped in both.
+  - **Extraction:** `regex` went from 6 files / 7 deps to **19 files / 20 deps**. Every other
+    manager is unchanged: flux 15, github-actions 24, kubernetes 131, kustomize 2.
+  - **Branches:** 33 before, **29 after**. Removed: gitea-28.x, qbittorrent-20.x, radarr-6.x,
+    nextcloud-35.x and postgresql-18.x. Added: velero `v1.14.3-rc.2 → v1.14.4` (patch).
+  - **Gitea, qbittorrent, radarr, sonarr, semaphore and seerr branches** now each carry
+    `catalog/<svc>.yaml` + `apps/wave-*/<svc>.yaml`.
+  - **The oauth2-proxy branch** carries `catalog/render.py` plus the 5 rendered apps and
+    `platform/crafty`.
+  - **The postgresql-16.x branch** carries all three Clusters.
+  - The 2 flux docker duplicates report `disabled`.
+- **render==golden holds for the new branch shape.**
+  - In the toolbox, I applied the gitea branch edit (catalog + apps) and the oauth2-proxy branch
+    edit (`render.py` constant + 5 apps) textually. `./ci/render-golden.sh` reported
+    `all 10 onboarded service golden(s) reproduce`, rc 0.
+  - Negative control, the old shape with only `apps/wave-0/gitea.yaml` bumped: rc 1,
+    `GOLDEN DRIFT`.
+  - Both edits were reverted afterwards.
+
+## What I did NOT verify
+
+- **The effect of the automerge, label and dashboard-approval rules (3, 4, 5, 8). UNVERIFIED in a
+  run.** `--platform=local` stops before branch processing, so I saw only which updates exist, not
+  their merged flags. The matchers are standard (`matchPackageNames`, `matchManagers`,
+  `matchCurrentValue`, `matchUpdateTypes`). The next dispatch's debug log shows them per branch.
+- **`allowedVersions` under docker versioning** used Renovate's "Falling back to npm semver
+  syntax" path (16 debug lines). That it filters is shown by the five removed branches.
+- **render==golden for the other app branches** (qbittorrent, radarr, sonarr, semaphore, seerr). I
+  proved it only for gitea and oauth2-proxy. They have the same shape: one `source.image` line ↔
+  one rendered `image:` line.
+- **The `allowedVersions` rules duplicate `catalog/*/update.range`** and can drift from it. Nothing
+  checks them against each other. A lint comparing the two is the obvious follow-up.
+- **Any real (non-dry) run.** None has happened. PR creation, labels, auto-merge through Gitea's
+  API and the Dependency Dashboard issue are all unobserved.
+- **Sensitive path?** `renovate.json5` is not in `infra/gitea/approval-policy.json`, as far as I
+  know. CI's `approval-policy` result on the PR is the measurement.

renovate.json5

@@ -149,6 +149,50 @@
       ],
       "datasourceTemplate": "docker",
       "versioningTemplate": "docker"
+    },
+    // ADDED 2026-10-30 — THE CATALOG IS THE SOURCE; apps/wave-*/*.yaml IS GENERATED FROM IT.
+    // First real dry run (dispatch runs 353/354): EVERY app-image update edited only the rendered
+    // apps/wave-*/*.yaml and NONE edited catalog/. The `kubernetes` manager lists catalog/ above
+    // but extracts nothing there: a ServiceDefinition has no `apiVersion`, so it is not a
+    // Kubernetes manifest to that manager (measured: 0 deps from catalog/). Every such PR would
+    // fail ci/render-golden.sh (the canonical render reverts the bump), sit red and hold a
+    // prConcurrentLimit slot. These two managers read the SOURCES, under the SAME depName as the
+    // rendered line, so both land in ONE branch and render==golden holds: `source.image` in each
+    // ServiceDefinition, and render.py's OAUTH2_PROXY_IMAGE (the six auth sidecars).
+    {
+      "customType": "regex",
+      "description": "ServiceDefinition `source.image: <repo>:<tag>` (catalog/*.yaml) — the source of the rendered apps/wave-* image.",
+      "managerFilePatterns": ["/(^|/)catalog/[^/]+\\.ya?ml$/"],
+      "matchStrings": [
+        "\\n\\s+image:\\s*(?<depName>[^\\s:@#\"']+):(?<currentValue>[^\\s@#\"']+)"
+      ],
+      "datasourceTemplate": "docker"
+    },
+    {
+      "customType": "regex",
+      "description": "catalog/render.py OAUTH2_PROXY_IMAGE — the per-app oauth2-proxy sidecar every forward-auth app is rendered with.",
+      "managerFilePatterns": ["/(^|/)catalog/render\\.py$/"],
+      "matchStrings": [
+        "\\nOAUTH2_PROXY_IMAGE\\s*=\\s*\"(?<depName>[^\":@]+):(?<currentValue>[^\"@]+)\""
+      ],
+      "datasourceTemplate": "docker"
+    },
+    // ADDED 2026-10-30 — CNPG `Cluster.spec.imageName`. The `kubernetes` manager reads `image:` only,
+    // so the three database Clusters (platform/keycloak/postgres-cluster.yaml, apps/wave-0/
+    // nextcloud-db.yaml, apps/wave-0/semaphore-db.yaml) were untracked, while the grant Job in
+    // nextcloud-db.yaml — whose comment says "keep the tag in step with spec.imageName" — WAS. A real
+    // run would have bumped the Job and left the database behind. Same depName, so one branch.
+    {
+      "customType": "regex",
+      "description": "CloudNativePG Cluster `imageName: <repo>:<tag>`.",
+      "managerFilePatterns": [
+        "/(^|/)platform/.+\\.ya?ml$/",
+        "/(^|/)apps/.+\\.ya?ml$/"
+      ],
+      "matchStrings": [
+        "\\n\\s+imageName:\\s*(?<depName>[^\\s:@#\"']+):(?<currentValue>[^\\s@#\"']+)"
+      ],
+      "datasourceTemplate": "docker"
     }
   ],
   "packageRules": [
@@ -200,7 +244,15 @@
         "longhorn",
         "cilium",
         "cloudnative-pg",
-        "kube-prometheus-stack"
+        "kube-prometheus-stack",
+        // ADDED 2026-10-30. gitea and act_runner ARE the merge gate: a bad auto-merged bump takes
+        // down the very check that would have to approve its revert (the dry run proposed gitea
+        // 1.24.3 -> 1.27.3 and act_runner 0.2.11 -> 0.6.1, both as auto-mergeable "minor"; a 0.x
+        // minor is a breaking line by semver). The postgresql image restarts every CNPG database,
+        // Keycloak's (the IdP) included, now that imageName is tracked.
+        "gitea/gitea",
+        "gitea/act_runner",
+        "ghcr.io/cloudnative-pg/postgresql"
       ],
       "automerge": false,
       "addLabels": [
@@ -227,6 +279,51 @@
         "gitea.home.keithjackson.dev/home-network/conformance-runner"
       ],
       "addLabels": ["own-image"]
+    },
+    {
+      "description": "ADDED 2026-10-30. The `flux` manager also extracts `image:` lines inside HelmRelease values, which the `kubernetes` manager already owns. Measured: exactly two (alpine in platform/oauth2-proxy/helmrelease.yaml, velero-plugin-for-aws in platform/velero/helmrelease.yaml), each extracted twice; the second rewrite of the same line logged `Cannot find replaceString in current file content` in runs 353/354. Flux keeps its charts (datasource helm).",
+      "matchManagers": ["flux"],
+      "matchDatasources": ["docker"],
+      "enabled": false
+    },
+    {
+      "description": "ADDED 2026-10-30. MIRRORS catalog/<svc>.yaml `update.range` — the operator's declared per-service policy (schemas/servicedefinition.schema.json), which Renovate does not read. Without it the dry run proposed gitea 28.2.0, qbittorrent 20.04.1 (both stale/odd tags, not real successors), radarr 6.x and nextcloud 35 (four majors past 31; Nextcloud cannot skip majors). KEEP IN STEP with the catalog ranges.",
+      "matchPackageNames": ["gitea/gitea"],
+      "allowedVersions": "<2.0.0"
+    },
+    { "matchPackageNames": ["nextcloud"], "allowedVersions": "<32.0.0" },
+    { "matchPackageNames": ["lscr.io/linuxserver/qbittorrent"], "allowedVersions": "<6.0.0" },
+    { "matchPackageNames": ["lscr.io/linuxserver/radarr"], "allowedVersions": "<6.0.0" },
+    { "matchPackageNames": ["lscr.io/linuxserver/sonarr"], "allowedVersions": "<5.0.0" },
+    { "matchPackageNames": ["semaphoreui/semaphore"], "allowedVersions": "<3.0.0" },
+    {
+      "description": "ADDED 2026-10-30. A CNPG Postgres MAJOR is a migration (new cluster + import), not an image bump; the dry run proposed 16.4 -> 18.6. Minors stay (and are blast-radius, above).",
+      "matchPackageNames": ["ghcr.io/cloudnative-pg/postgresql"],
+      "allowedVersions": "<17.0.0"
+    },
+    {
+      "description": "ADDED 2026-10-30. A HelmRelease semver RANGE is a policy ceiling, and Renovate does not respect it — it WIDENS it (dry run: external-dns `<1.22.0` -> `<1.24.0`, headlamp `<0.42.0` -> `<0.47.0`), and the default rule would then auto-merge that as a 'minor', silently letting Flux adopt every new minor. Raising a ceiling is deliberate per-component work: propose it, never auto-merge it.",
+      "matchManagers": ["flux"],
+      "matchCurrentValue": "/^[<>=~^]|\\.x$/",
+      "automerge": false,
+      "addLabels": ["needs-review", "range-widen"]
+    },
+    {
+      "description": "ADDED 2026-10-30. velero-plugin-for-aws is pinned to a RELEASE CANDIDATE (v1.14.3-rc.2) and Renovate proposed NOTHING for it, although v1.14.3 and v1.14.4 are stable on Docker Hub. Docker versioning treats everything after the first `-` as a compatibility SUFFIX, so only other `*-rc.2` tags qualify and the RC is a fossil forever. Semver versioning reads it as a prerelease, and a prerelease upgrades to the stable release.",
+      "matchPackageNames": ["velero/velero-plugin-for-aws"],
+      "versioning": "semver"
+    },
+    {
+      "description": "ADDED 2026-10-30. A python MINOR is a language-runtime release (the dry run proposed 3.13 -> 3.15 for the Keycloak reconcilers), and the gate does not execute those scripts. Review it; patches still auto-merge.",
+      "matchPackageNames": ["python"],
+      "matchUpdateTypes": ["minor"],
+      "automerge": false,
+      "addLabels": ["needs-review"]
+    },
+    {
+      "description": "ADDED 2026-10-30. MAJORS WAIT FOR A TICK ON THE DEPENDENCY DASHBOARD instead of opening a PR. They are never auto-merged anyway, and with prConcurrentLimit 5 the dry run's 10 majors would hold every PR slot open indefinitely, starving the patch/minor/digest PRs that can merge on green.",
+      "matchUpdateTypes": ["major"],
+      "dependencyDashboardApproval": true
     }
   ],
   // WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A
@@ -235,10 +332,12 @@
   // lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak
   // (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete
   // version before any automation can maintain them, and until then they drift on every pull with
-  // nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12,
-  // e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it
-  // has been told not to leave; raising those ceilings is deliberate per-component work and is
-  // NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.
+  // nothing recording what is running. CORRECTED 2026-10-30: nextcloud:31-apache and docker:27-dind
+  // ARE maintained (the first dry run proposed 35-apache and 29-dind; the suffix is kept).
+  // Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on
+  // kube-prometheus-stack): THIS FILE USED TO SAY they cap Renovate at patches. They do not —
+  // Renovate WIDENS the range (runs 353/354). The `range-widen` rule above keeps that a reviewed
+  // PR rather than an auto-merge. Both lists are in decisions/2026-10-17-automated-upgrades.md.
   // platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest`
   // and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed
   // shape, not software to keep current. `conformance/**` was already ignored for the same