diff --git a/decisions/2026-10-18-renovate-invalid-config.md b/decisions/2026-10-18-renovate-invalid-config.md new file mode 100644 index 0000000..5fcba76 --- /dev/null +++ b/decisions/2026-10-18-renovate-invalid-config.md @@ -0,0 +1,78 @@ +--- +date: 2026-10-18 +branch: fix/renovate-invalid-config +title: "renovate.json was invalid, so the first manual run errored out" +--- + +## What changed + +The operator ran the Renovate workflow manually and it **failed**. The cause is not the workflow: +**`renovate.json` was never a valid Renovate config.** + +It carried five pseudo-comment keys — `_comment`, `_kubernetes_comment`, `_unmanageable_comment`, +`_ignorePaths_comment`, `_docker_comment` — because JSON has no comments. Renovate **rejects unknown +top-level options**, so the entire config failed validation. Proven with the official validator +rather than inferred: + +``` +$ podman run --rm -v "$PWD/renovate.json:/tmp/renovate.json:ro,Z" \ + ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json +ERROR: Found errors in configuration + Invalid configuration option: _comment + Invalid configuration option: _docker_comment + Invalid configuration option: _ignorePaths_comment + Invalid configuration option: _kubernetes_comment + Invalid configuration option: _unmanageable_comment +``` + +This is the second independent reason Renovate had never worked, on top of PR #77's cron firing +outside `renovate.json`'s own schedule window. Either alone was sufficient to produce "nothing ever +happens"; #73 shipped both. + +**Converted `renovate.json` → `renovate.json5`.** JSON5 permits real `//` comments, and Renovate +discovers `renovate.json5` natively — nothing in `.gitea/` or `ci/` pins the old filename (only +prose inside comments), and no `RENOVATE_CONFIG_FILE` is set. Every word of the original rationale +is preserved as comments positioned next to what it explains, which a flattened `description` +string would have lost. + +A header on the new file records the failure, the validator command, and the instruction to run it +before committing — this class of bug is invisible to the repo's own gate. + +## What I verified + +* **The exact failure**, from the official validator, quoted above. Not read off a log, not inferred + from the workflow. +* **The fix validates**: the same validator against `renovate.json5` returns + `INFO: Config validated successfully against 1 file(s)`. +* **Nothing was dropped in the conversion**, checked programmatically against `git show + HEAD:renovate.json`: 18 non-comment keys before, 18 after, **no missing keys, no extra keys, and + no key whose value changed**. `docker: {enabled: true}` confirmed still present — the validator + prints a normalisation diff that mentions it, which is its own migration suggestion, not a + removal. +* **74 comment lines survive** in the new file, so the rationale is intact. +* **`RENOVATE_TOKEN` exists** as a repository Actions secret (listed by name via the Gitea API), so + the workflow's fail-hard preflight is satisfied. +* **A suspicion was DISPROVED:** `enabledManagers` containing bare `regex` was thought to need to be + `custom.regex`, which would have silently disabled every inline `# renovate:` marker. The + validator accepts `regex`, so that concern was unfounded and nothing here changes it. +* Nothing functionally references `renovate.json` by name — `grep -rn "renovate\.json" .gitea/ ci/ + conformance/` returns only comment prose. + +## What I did NOT verify + +* **Renovate has still never completed a run.** This removes one of two blockers; PR #77 removes the + other. Neither is merged, so **no successful Renovate run has been observed** and I cannot say the + bot works — only that these two specific defects are fixed. +* **I did not re-run the workflow after the fix.** The honest test is a `workflow_dispatch` with + `dry_run: true` once both this and #77 are merged. Expect ~30 dependencies behind, rate-limited to + `prHourlyLimit: 2` / `prConcurrentLimit: 5`. +* **Whether the managers actually extract anything.** A valid config that matches no files is still + a no-op. The dry-run log's per-manager dependency counts are the only thing that proves the + `managerFilePatterns` are right, and I have not seen them. +* **Whether `renovate.json5` is honoured by the pinned Renovate version in the workflow.** JSON5 + discovery is long-standing and the validator accepted the file, but the workflow's image tag was + not cross-checked against the version that added it. +* **Nothing in the gate catches an invalid Renovate config.** This bug would have recurred silently. + A one-line L0 check shelling out to `renovate-config-validator` would close it; it is not written + here, and `conformance/**` is a sensitive path. +* I did not re-measure branch protection or runner state (CLAUDE.md §2b). diff --git a/renovate.json b/renovate.json deleted file mode 100644 index 4402e14..0000000 --- a/renovate.json +++ /dev/null @@ -1,99 +0,0 @@ -{ - "$schema": "https://docs.renovatebot.com/renovate-schema.json", - "_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17 RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags. It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009 split that gave image tags to Flux image-automation is formally closed — see .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write anyway; and enabling it would need an in-cluster git WRITE credential for the repo that defines the cluster). image-automation has been commented out at platform/kustomization.yaml:52 since it was authored and the cluster holds zero ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down. CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag` line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER: .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.", - "extends": [ - "config:recommended", - ":dependencyDashboard", - ":semanticCommits", - "helpers:pinGitHubActionDigests" - ], - "timezone": "America/New_York", - "schedule": ["after 3am and before 6am"], - "prConcurrentLimit": 5, - "prHourlyLimit": 2, - "commitMessagePrefix": "chore(deps):", - "labels": ["dependencies", "automated-update"], - "ignoreUnstable": true, - "respectLatest": true, - "enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"], - "flux": { - "managerFilePatterns": ["/(^|/)platform/.+\\.ya?ml$/", "/(^|/)apps/.+\\.ya?ml$/"] - }, - "kustomize": { - "managerFilePatterns": ["/(^|/)kustomization\\.ya?ml$/"] - }, - "_kubernetes_comment": "ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes manifests, and it is NOT auto-detected — it only looks at files matched here, which is why this block is as broad as it is. Scope covers every place this repo writes a container image: platform components, the rendered app waves, the ServiceDefinition catalog that generates them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the rendered output is reverted by the next render and would also fail ci/render-golden.sh. Renovate will raise BOTH, and both must land together.", - "kubernetes": { - "managerFilePatterns": [ - "/(^|/)platform/.+\\.ya?ml$/", - "/(^|/)platform-config/.+\\.ya?ml$/", - "/(^|/)apps/.+\\.ya?ml$/", - "/(^|/)catalog/.+\\.ya?ml$/", - "/(^|/)runbooks/serving/.+\\.ya?ml$/", - "/(^|/)infra/agent/.+\\.ya?ml$/" - ] - }, - "customManagers": [ - { - "customType": "regex", - "description": "Inline `# renovate: datasource=... depName=...` markers on pinned URLs / version literals (e.g. platform/kubevirt/kustomization.yaml release URLs, ci/versions.env, infra/devcontainer/versions.env).", - "managerFilePatterns": [ - "/(^|/)kustomization\\.ya?ml$/", - "/(^|/)versions\\.env$/" - ], - "matchStrings": [ - "#\\s*renovate:\\s*datasource=(?\\S+)\\s+depName=(?\\S+)(?:\\s+versioning=(?\\S+))?\\s*\\n.*?(?v?[0-9]+\\.[0-9]+\\.[0-9]+)" - ], - "versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}" - } - ], - "packageRules": [ - { - "description": "Auto-merge the allowed low-risk classes on green CI — no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.", - "matchUpdateTypes": ["minor", "patch", "digest"], - "automerge": true, - "automergeType": "pr", - "platformAutomerge": true - }, - { - "description": "MAJOR / potentially-breaking bumps require human review (docs/11 safety rail). Never auto-merged.", - "matchUpdateTypes": ["major"], - "automerge": false, - "addLabels": ["needs-review", "breaking"] - }, - { - "description": "Keep the KubeVirt + CDI operator pair on a compatible track — grouped so they bump together (platform/kubevirt/kustomization.yaml).", - "matchDepNames": ["kubevirt/kubevirt", "kubevirt/containerized-data-importer"], - "groupName": "kubevirt-cdi" - }, - { - "description": "Pinned tool versions (ci/versions.env, infra/devcontainer/versions.env) are a deliberate change surface — label but do not auto-merge; a version bump there changes CI reproducibility.", - "matchFileNames": ["ci/versions.env", "infra/devcontainer/versions.env"], - "automerge": false, - "addLabels": ["ci-tooling"] - }, - { - "description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check — it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.", - "matchPackageNames": [ - "kured", - "longhorn", - "cilium", - "cloudnative-pg", - "kube-prometheus-stack" - ], - "automerge": false, - "addLabels": ["needs-review", "blast-radius"] - } - ], - "_unmanageable_comment": "WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest, lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete version before any automation can maintain them, and until then they drift on every pull with nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it has been told not to leave; raising those ceilings is deliberate per-component work and is NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.", - "ignorePaths": [ - "**/*.example", - "**/*.sops.yaml", - "conformance/**", - "platform/image-automation/per-service/**", - "platform/kyverno/tests/**" - ], - "_ignorePaths_comment": "platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest` and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed shape, not software to keep current. `conformance/**` was already ignored for the same reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise have opened.", - "_docker_comment": "The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation over a running tag. That controller was never enabled and never will be here (see _comment), so the only thing those two flags were achieving was leaving every plain `image:` tag in the repo unmanaged. `docker` is left out of enabledManagers because this repo contains no Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.", - "docker": { "enabled": true } -} diff --git a/renovate.json5 b/renovate.json5 new file mode 100644 index 0000000..f6decdf --- /dev/null +++ b/renovate.json5 @@ -0,0 +1,187 @@ +// ================================================================================================= +// Renovate configuration — JSON5, so the rationale can live NEXT TO what it explains. +// ================================================================================================= +// THIS FILE WAS `renovate.json` AND IT WAS INVALID. It carried five `_comment` / `_*_comment` +// pseudo-comment keys, because JSON has no comments. Renovate REJECTS unknown top-level options, so +// the whole config failed validation and the first manual run of the workflow ERRORED OUT. Proven +// with the official validator (`ghcr.io/renovatebot/renovate renovate-config-validator`): +// +// ERROR: Found errors in configuration +// Invalid configuration option: _comment +// Invalid configuration option: _docker_comment +// Invalid configuration option: _ignorePaths_comment +// Invalid configuration option: _kubernetes_comment +// Invalid configuration option: _unmanageable_comment +// +// Renovate discovers `renovate.json5` natively (no RENOVATE_CONFIG_FILE needed, and nothing in +// .gitea/ or ci/ pins the old filename — only prose in comments), and JSON5 permits real `//` +// comments. So the fix keeps every word of the original rationale, positioned, instead of +// flattening it into a `description` string. +// +// VALIDATE BEFORE COMMITTING — it is one command and it is the only thing that catches this class: +// podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \ +// ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json5 +// +// NOTE on what was NOT wrong: `enabledManagers` containing bare `regex` was suspected of needing to +// be `custom.regex`. The validator accepts `regex`, so that suspicion was unfounded and nothing +// here changes it. +{ "$schema": "https://docs.renovatebot.com/renovate-schema.json", + // Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17 + // RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags. + // It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009 + // split that gave image tags to Flux image-automation is formally closed — see + // .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than + // opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with + // user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write + // anyway; and enabling it would need an in-cluster git WRITE credential for the repo that + // defines the cluster). image-automation has been commented out at + // platform/kustomization.yaml:52 since it was authored and the cluster holds zero + // ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down. + // CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid + // fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag` + // line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth + // sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER: + // .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here. + "extends": ["config:recommended", ":dependencyDashboard", ":semanticCommits", "helpers:pinGitHubActionDigests"], + "timezone": "America/New_York", + "schedule": ["after 3am and before 6am"], + "prConcurrentLimit": 5, + "prHourlyLimit": 2, + "commitMessagePrefix": "chore(deps):", + "labels": ["dependencies", "automated-update"], + "ignoreUnstable": true, + "respectLatest": true, + "enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"], + "flux": { + "managerFilePatterns": [ + "/(^|/)platform/.+\\.ya?ml$/", + "/(^|/)apps/.+\\.ya?ml$/" + ] + }, + "kustomize": { + "managerFilePatterns": [ + "/(^|/)kustomization\\.ya?ml$/" + ] + }, + // ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes + // manifests, and it is NOT auto-detected — it only looks at files matched here, which is why + // this block is as broad as it is. Scope covers every place this repo writes a container image: + // platform components, the rendered app waves, the ServiceDefinition catalog that generates + // them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as + // `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the + // rendered output is reverted by the next render and would also fail ci/render-golden.sh. + // Renovate will raise BOTH, and both must land together. + "kubernetes": { + "managerFilePatterns": [ + "/(^|/)platform/.+\\.ya?ml$/", + "/(^|/)platform-config/.+\\.ya?ml$/", + "/(^|/)apps/.+\\.ya?ml$/", + "/(^|/)catalog/.+\\.ya?ml$/", + "/(^|/)runbooks/serving/.+\\.ya?ml$/", + "/(^|/)infra/agent/.+\\.ya?ml$/" + ] + }, + "customManagers": [ + { + "customType": "regex", + "description": "Inline `# renovate: datasource=... depName=...` markers on pinned URLs / version literals (e.g. platform/kubevirt/kustomization.yaml release URLs, ci/versions.env, infra/devcontainer/versions.env).", + "managerFilePatterns": [ + "/(^|/)kustomization\\.ya?ml$/", + "/(^|/)versions\\.env$/" + ], + "matchStrings": [ + "#\\s*renovate:\\s*datasource=(?\\S+)\\s+depName=(?\\S+)(?:\\s+versioning=(?\\S+))?\\s*\\n.*?(?v?[0-9]+\\.[0-9]+\\.[0-9]+)" + ], + "versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}" + } + ], + "packageRules": [ + { + "description": "Auto-merge the allowed low-risk classes on green CI \u2014 no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.", + "matchUpdateTypes": [ + "minor", + "patch", + "digest" + ], + "automerge": true, + "automergeType": "pr", + "platformAutomerge": true + }, + { + "description": "MAJOR / potentially-breaking bumps require human review (docs/11 safety rail). Never auto-merged.", + "matchUpdateTypes": [ + "major" + ], + "automerge": false, + "addLabels": [ + "needs-review", + "breaking" + ] + }, + { + "description": "Keep the KubeVirt + CDI operator pair on a compatible track \u2014 grouped so they bump together (platform/kubevirt/kustomization.yaml).", + "matchDepNames": [ + "kubevirt/kubevirt", + "kubevirt/containerized-data-importer" + ], + "groupName": "kubevirt-cdi" + }, + { + "description": "Pinned tool versions (ci/versions.env, infra/devcontainer/versions.env) are a deliberate change surface \u2014 label but do not auto-merge; a version bump there changes CI reproducibility.", + "matchFileNames": [ + "ci/versions.env", + "infra/devcontainer/versions.env" + ], + "automerge": false, + "addLabels": [ + "ci-tooling" + ] + }, + { + "description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check \u2014 it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.", + "matchPackageNames": [ + "kured", + "longhorn", + "cilium", + "cloudnative-pg", + "kube-prometheus-stack" + ], + "automerge": false, + "addLabels": [ + "needs-review", + "blast-radius" + ] + } + ], + // WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A + // FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it + // and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest, + // lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak + // (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete + // version before any automation can maintain them, and until then they drift on every pull with + // nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12, + // e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it + // has been told not to leave; raising those ceilings is deliberate per-component work and is + // NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md. + // platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest` + // and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed + // shape, not software to keep current. `conformance/**` was already ignored for the same + // reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise + // have opened. + "ignorePaths": [ + "**/*.example", + "**/*.sops.yaml", + "conformance/**", + "platform/image-automation/per-service/**", + "platform/kyverno/tests/**" + ], + // The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were + // BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation + // over a running tag. That controller was never enabled and never will be here (see _comment), + // so the only thing those two flags were achieving was leaving every plain `image:` tag in the + // repo unmanaged. `docker` is left out of enabledManagers because this repo contains no + // Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters. + "docker": { + "enabled": true + } +}