Compare commits

...
Author SHA1 Message Date
jakman85andClaude Opus 5 713b0c3dba fix(renovate): the config was INVALID, so the first manual run errored out
gate / gate (pull_request) Successful in 1m55s
gate / conformance (pull_request) Has been skipped
The operator ran the Renovate workflow manually and it failed. The cause is not
the workflow: renovate.json was never a valid Renovate config. It carried five
pseudo-comment keys (_comment, _kubernetes_comment, _unmanageable_comment,
_ignorePaths_comment, _docker_comment) because JSON has no comments, and
Renovate REJECTS unknown top-level options. Proven with the official validator:

  ERROR: Found errors in configuration
    Invalid configuration option: _comment
    Invalid configuration option: _docker_comment
    Invalid configuration option: _ignorePaths_comment
    Invalid configuration option: _kubernetes_comment
    Invalid configuration option: _unmanageable_comment

This is the SECOND independent reason Renovate never worked, on top of PR #77's
cron firing outside renovate.json's own schedule window. Either alone was enough
to produce "nothing ever happens"; #73 shipped both.

Converted renovate.json -> renovate.json5. JSON5 permits real // comments and
Renovate discovers renovate.json5 natively; nothing in .gitea/ or ci/ pins the
old filename and no RENOVATE_CONFIG_FILE is set. Every word of the original
rationale is preserved as positioned comments (74 comment lines) rather than
flattened into a description string.

Verified the conversion dropped nothing: 18 non-comment keys before and after,
no missing/extra keys, no changed values, docker: {enabled: true} still present.
The validator now returns "Config validated successfully".

DISPROVED while here: `enabledManagers` containing bare `regex` was suspected of
needing to be `custom.regex` (which would silently disable every inline
`# renovate:` marker). The validator accepts `regex`; unfounded, unchanged.

Renovate has still never completed a run -- this fixes one of two blockers and
nothing in the gate catches an invalid config. See the handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 03:10:25 -05:00
jakman85 c340bbfc4b Merge pull request 'fix(conformance): one regex made the merge gate take 28 minutes instead of 2' (#76) from fix/ha-check-catastrophic-regex into main
gate / gate (push) Successful in 1m56s
gate / conformance (push) Has been skipped
Reviewed-on: #76
Reviewed-by: Keith Jackson <[email protected]>
2026-10-18 07:51:51 +00:00
jakman85 a9a529750d Merge pull request 'fix(unattended-upgrades): Debian has no reboot-sentinel package, so ship the writer' (#75) from fix/unattended-upgrades-debian-sentinel into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #75
2026-10-18 07:40:20 +00:00
jakman85andClaude Opus 5 793c3940a8 fix(conformance): one regex made the merge gate take 28 minutes instead of 2
gate / gate (pull_request) Successful in 1m55s
gate / conformance (pull_request) Has been skipped
ha-control-plane-as-code/check.sh had:

  re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:...'secondary'",
            K3S_TEXT, re.S)

With re.S every `.` matches newlines, so `.*?` and the nested `(?:.*\n){0,10}?`
can each consume the whole document -- nested unbounded repetition, which
backtracks exponentially on a failed or late match. MEASURED in CI via
`docker top` on the live job container: python3 at 19:54 of CPU time, which is
essentially the entire ~28 minute gate.

Localised to the exact line by extracting the inline python and running it under
faulthandler.dump_traceback_later, which named re/__init__.py:177 in search from
this call -- not inferred from reading.

Replaced `.` with `[^\n]*` and dropped re.S, so the {0,10} is a real ten-line
window with nothing to backtrack into. On the real input (22,179 chars):
0.2 ms, match still found. Whole check: 0.199s, rc 0, 115 assertions, identical
success message.

WHY IT SURFACED NOW, AND IT WAS MINE: K3S_TEXT is every *.yml in
roles/k3s_server concatenated. The regex was latent for as long as it existed;
PR #71 added ~100 lines to that role (kubelet graceful-shutdown config), growing
the input past the point where exponential backtracking stopped being
survivable.

WHY ONLY infra/** WAS SLOW: touched-requirements.sh maps infra/* to R-ENV-01, so
the L0 run becomes --include-scope whole-repo --requirements R-ENV-01, which
selects this whole-repo check. Doc/platform changes never did -- #69 1m55s and
#72 1m53s stayed fast while #71 27m44s, the #73 merge 27m46s and #75 all ran
~28m.

The pattern is unique in the repo (grep over all l0-static check.sh files).
Not re-timed in CI post-fix; this PR's own run is the first datapoint.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 02:38:23 -05:00
jakman85andClaude Opus 5 ef8ef5ac29 fix(unattended-upgrades): Debian has no reboot-sentinel package, so ship the writer
gate / gate (pull_request) Successful in 27m51s
gate / conformance (pull_request) Has been skipped
The first real run failed on all three nodes:

  fatal: [k3s-prod-1]: "No package matching 'update-notifier-common' is available"

update-notifier-common is an UBUNTU package and does not exist in Debian --
`apt-cache policy` returns Candidate: (none) on trixie. The role asserted it
"owns the hook that creates /var/run/reboot-required", which is true on Ubuntu
and false here, so the role could never have worked.

Three further measurements settled the replacement: notify-reboot-required is
absent; NO linux-image-* postinst references it (invoking it is an Ubuntu patch,
so dropping a script at that path would never be called); and no hook under
apt.conf.d or /etc/kernel mentions reboot-required. Debian trixie ships no
mechanism at all.

So the role now provides it:
* /usr/local/sbin/k3s-notify-reboot-required -- touches the sentinel when the
  running kernel is not the newest installed, using `linux-version` (from
  linux-base, a dependency of every linux-image, measured present) because
  kernel version ordering is not string ordering.
* /etc/apt/apt.conf.d/99-k3s-reboot-required -- DPkg::Post-Invoke, `|| true` so
  it can never abort an apt transaction.
/run is tmpfs (measured), so the sentinel self-clears on boot; no cleanup task.

The verify assert was ALSO weaker than it looked: a `find` glob for
*update-notifier* / *unattended-upgrades* under apt.conf.d would have matched
Debian's own unattended-upgrades conffile and reported "writer present" while
none existed. It now stats the two exact paths this role owns and requires the
script to be executable.

Newly verified: the script ran on prod-1 as unprivileged k3s under sh -x --
running=newest=7.1.8+deb13-amd64, took the equal branch, exited 0, wrote
nothing. Correct. sh -n and bash -n clean.

Still unverified: the WRITING half never executes while running == newest, so
the sentinel has not been seen created and kured has not been seen react. The
honest test is the next kernel upgrade.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 01:43:43 -05:00
9 changed files with 544 additions and 142 deletions

No files matched your search

@@ -445,8 +445,21 @@ join = re.search(r"--server https://\{\{\s*k3s_api_host", K3S_TEXT)
check(join is not None, check(join is not None,
"roles/k3s_server does not join secondaries with `--server https://{{ k3s_api_host }}` " "roles/k3s_server does not join secondaries with `--server https://{{ k3s_api_host }}` "
"— without a join target the servers are separate single-node clusters") "— without a join target the servers are separate single-node clusters")
join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'", # LINE-BOUNDED ON PURPOSE -- DO NOT "SIMPLIFY" BACK TO `.*` + re.S.
K3S_TEXT, re.S) # This was `--server https://.*?\n(?:.*\n){0,10}?\s*when:...` with re.S, which made every `.`
# match newlines too. That is nested unbounded repetition (a `{0,10}` of a quantifier that can each
# swallow the whole document), so a FAILED or late match backtracks exponentially. It cost
# **19m54s of CPU in CI** (measured 2026-10-18 via `docker top` on the live job container) and made
# the whole merge gate take ~28 minutes instead of ~2 on any change touching `infra/**` -- because
# `ci/touched-requirements.sh` maps `infra/*` to R-ENV-01, which pulls in this whole-repo check.
# It was latent for as long as the regex existed and only crossed the threshold when
# roles/k3s_server grew by ~100 lines (the kubelet graceful-shutdown config, PR #71): K3S_TEXT is
# every *.yml in that role concatenated, so the role growing is the input growing.
# `[^\n]*` cannot cross a line, so the `{0,10}` is a genuine ten-line window with nothing to
# backtrack into. Measured on the same input: 0.2 ms, same verdict (match found).
join_when = re.search(
r"--server https://[^\n]*\n(?:[^\n]*\n){0,10}?[ \t]*when:[ \t]*k3s_server_mode[ \t]*==[ \t]*'secondary'",
K3S_TEXT)
check(join_when is not None, check(join_when is not None,
"the join install task is not gated on k3s_server_mode == 'secondary' — the founder " "the join install task is not gated on k3s_server_mode == 'secondary' — the founder "
"would also try to join itself") "would also try to join itself")
@@ -185,6 +185,55 @@ Measured against the live cluster and nodes, read-only, on 2026-10-17.
- **`renovate.json` parses** as valid JSON after my edits (`json.load`). - **`renovate.json` parses** as valid JSON after my edits (`json.load`).
- **kured is not installed** and the string `kured` appears nowhere in the repo — greenfield. - **kured is not installed** and the string `kured` appears nowhere in the repo — greenfield.
### CORRECTION 2026-10-18 — the first real run FAILED, and the sentinel design was wrong
`ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod` failed on all three
nodes at the package task:
```
fatal: [k3s-prod-1]: FAILED! => "No package matching 'update-notifier-common' is available"
```
**`update-notifier-common` is an UBUNTU package and does not exist in Debian.** Measured on trixie:
`apt-cache policy update-notifier-common` → `Candidate: (none)`. The original entry asserted this
package "owns the hook that creates /var/run/reboot-required" — true on Ubuntu, false here, and the
role could never have worked. This was the exact risk the first version of this entry flagged ("the
`find` patterns are a guess at which package ships what") landing in a worse place than predicted:
not a wrong glob, a nonexistent package.
Three further measurements settled what to do instead:
* `/usr/share/update-notifier/notify-reboot-required` is **absent**.
* **No `linux-image-*` postinst references `notify-reboot-required`** — invoking it is an Ubuntu
patch, so merely placing a script at that path would never be called.
* No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
So Debian trixie has **no stock mechanism at all**, and the role now ships the writer itself:
`/usr/local/sbin/k3s-notify-reboot-required` (compares the running kernel against the newest
installed via `linux-version`, from `linux-base`, which every `linux-image` depends on — measured
`linux-base 4.12.1` present) plus a `DPkg::Post-Invoke` hook at
`/etc/apt/apt.conf.d/99-k3s-reboot-required`. `/run` is a tmpfs (measured), so the sentinel
self-clears on boot and needs no cleanup task.
The verify assert changed shape too: it was a `find` glob over `/etc/kernel/postinst.d` and
`/etc/apt/apt.conf.d` for `*update-notifier*` / `*unattended-upgrades*`. That glob was **weaker
than it looked** — it would have matched Debian's own `unattended-upgrades` apt conffile and
reported "sentinel writer present" while no writer existed. It now stats the two exact paths this
role owns and requires the script to be executable.
**Newly verified (this correction):** the sentinel script was executed on prod-1 as the
unprivileged `k3s` user under `sh -x`. It resolved `running=7.1.8+deb13-amd64`,
`newest=7.1.8+deb13-amd64`, took the equal branch, exited 0 and wrote nothing — correct, since the
running kernel *is* the newest. `sh -n` and `bash -n` clean.
**Still NOT verified after this correction:** the *writing* half. Because running == newest on all
three nodes, the `touch` path has never executed, so I have not seen the sentinel actually created,
nor seen kured react to it. The honest test is the operator's next kernel upgrade. Also unverified:
whether a `DPkg::Post-Invoke` hook fires in the `unattended-upgrades` code path specifically (it
runs dpkg, so it should, but that is inference) — and whether `linux-version sort --reverse` orders
a mixed backports/stable kernel set the way I assume when a 6.12 and a 7.1 are both installed. It
produced the right answer with the current set; a future set could differ.
## What I did NOT verify ## What I did NOT verify
The honest list. Most of this needs an actual upgrade cycle, which I did not and could not run. The honest list. Most of this needs an actual upgrade cycle, which I did not and could not run.
@@ -0,0 +1,76 @@
---
date: 2026-10-18
branch: fix/ha-check-catastrophic-regex
title: "One regex made the merge gate take 28 minutes instead of 2"
---
## What changed
The merge gate started taking **~28 minutes** instead of its usual **<2 minutes**, on every change
touching `infra/**`. One regex was responsible for essentially all of it.
`conformance/tests/l0-static/ha-control-plane-as-code/check.sh` contained:
```python
join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
K3S_TEXT, re.S)
```
With `re.S`, every `.` also matches newlines, so `.*?` and the nested `(?:.*\n){0,10}?` can each
consume the entire document. That is **nested unbounded repetition** — ten levels of a quantifier
that can each swallow everything — and a failed or late match backtracks exponentially.
Replaced `.` with `[^\n]*` and dropped `re.S`, so the `{0,10}` is a genuine ten-line window with
nothing to backtrack into. Same intent, same verdict, no DOTALL.
**Why it surfaced now, and it was mine.** `K3S_TEXT` is every `*.yml` in `roles/k3s_server`
concatenated. The regex was latent for as long as it existed; **PR #71 added ~100 lines to that role
(the kubelet graceful-shutdown config), which grew the input past the point where exponential
backtracking stopped being survivable.** Nothing about the check changed — my change grew its input.
**Why only `infra/**` changes were slow.** `ci/touched-requirements.sh:268` maps `infra/*` to
`R-ENV-01`, and the resulting L0 run is `--include-scope whole-repo --requirements R-ENV-01`, which
pulls in this whole-repo check. Doc-only and `platform/**` changes never selected it, which is why
#69 (1m55s) and #72 (1m53s) stayed fast while #71 (27m44s), the #73 merge (27m46s) and #75 all ran
~28 minutes.
## What I verified
* **The hot spot, by measurement rather than reading.** `docker top` on the live CI job container for
run #299 showed `python3 -` with **19:54 of CPU time** under
`ha-control-plane-as-code/check.sh`, inside
`conformance/run.sh --layer l0 --include-scope whole-repo --requirements R-ENV-01`.
* **Reproduced locally**: the check did not finish within a 100-second budget on an unmodified
checkout.
* **Localised to the exact line** by extracting the inline Python and running it under
`faulthandler.dump_traceback_later(25)`, which named `re/__init__.py:177 in search` called from
the `join_when` line — not inferred from reading the patterns.
* **The pattern is unique in the repo**: `grep -rlnE "\{0,[0-9]+\}\?" conformance/tests/l0-static/*/check.sh`
returns only this file, so there is no sibling with the same hazard.
* **The fix is fast AND verdict-preserving**, tested against the real `K3S_TEXT` (22,179 chars, 403
lines): the new regex matches (`YES`) in **0.2 ms**. Sanity-checked that the thing it looks for
genuinely exists — one `--server https://` and three `k3s_server_mode == 'secondary'` occurrences.
* **The whole check now runs in 0.199s, rc 0, 115 assertions**, with the same success message as
before.
## What I did NOT verify
* **I did not re-time the full gate in CI after the fix.** The expectation is that `infra/**` changes
return to the ~2-minute range, but that is arithmetic from the 19m54s figure, not a measured
post-fix run. The PR's own CI run is the first real datapoint — read it.
* **I did not audit the other 23 L0 checks for different performance pathologies.** This one was
found because it was actively burning CPU in a live job; a check that is merely slow (not
exponential) would not have stood out, and `ci/gate.sh` has **no per-stage timing** at all
(`grep -nE "SECONDS|date \+|elapsed"` finds nothing), so there is no data on where the remaining
time goes.
* **Nothing prevents this class of bug returning.** No check asserts gate stage runtime, and no lint
forbids nested quantifiers under `re.S`. A timing budget per stage, or a `re` lint, would close it;
neither is written here.
* **I did not verify my claim about *why* only `infra/**` was affected beyond the path mapping and
the observed durations.** The correlation is strong (5 runs, two clear clusters) and the live
process list confirms the `--requirements R-ENV-01` scope, but I did not read
`conformance/run.sh`'s scope-selection logic end to end.
* The regex rewrite assumes the `when:` clause is indented with spaces or tabs only (`[ \t]*`). A
file using some other whitespace before `when:` would now fail to match where the old DOTALL
version would have succeeded. Unlikely in YAML, and the current input matches, but it is a
narrowing.
@@ -0,0 +1,78 @@
---
date: 2026-10-18
branch: fix/renovate-invalid-config
title: "renovate.json was invalid, so the first manual run errored out"
---
## What changed
The operator ran the Renovate workflow manually and it **failed**. The cause is not the workflow:
**`renovate.json` was never a valid Renovate config.**
It carried five pseudo-comment keys — `_comment`, `_kubernetes_comment`, `_unmanageable_comment`,
`_ignorePaths_comment`, `_docker_comment` — because JSON has no comments. Renovate **rejects unknown
top-level options**, so the entire config failed validation. Proven with the official validator
rather than inferred:
```
$ podman run --rm -v "$PWD/renovate.json:/tmp/renovate.json:ro,Z" \
ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json
ERROR: Found errors in configuration
Invalid configuration option: _comment
Invalid configuration option: _docker_comment
Invalid configuration option: _ignorePaths_comment
Invalid configuration option: _kubernetes_comment
Invalid configuration option: _unmanageable_comment
```
This is the second independent reason Renovate had never worked, on top of PR #77's cron firing
outside `renovate.json`'s own schedule window. Either alone was sufficient to produce "nothing ever
happens"; #73 shipped both.
**Converted `renovate.json` → `renovate.json5`.** JSON5 permits real `//` comments, and Renovate
discovers `renovate.json5` natively — nothing in `.gitea/` or `ci/` pins the old filename (only
prose inside comments), and no `RENOVATE_CONFIG_FILE` is set. Every word of the original rationale
is preserved as comments positioned next to what it explains, which a flattened `description`
string would have lost.
A header on the new file records the failure, the validator command, and the instruction to run it
before committing — this class of bug is invisible to the repo's own gate.
## What I verified
* **The exact failure**, from the official validator, quoted above. Not read off a log, not inferred
from the workflow.
* **The fix validates**: the same validator against `renovate.json5` returns
`INFO: Config validated successfully against 1 file(s)`.
* **Nothing was dropped in the conversion**, checked programmatically against `git show
HEAD:renovate.json`: 18 non-comment keys before, 18 after, **no missing keys, no extra keys, and
no key whose value changed**. `docker: {enabled: true}` confirmed still present — the validator
prints a normalisation diff that mentions it, which is its own migration suggestion, not a
removal.
* **74 comment lines survive** in the new file, so the rationale is intact.
* **`RENOVATE_TOKEN` exists** as a repository Actions secret (listed by name via the Gitea API), so
the workflow's fail-hard preflight is satisfied.
* **A suspicion was DISPROVED:** `enabledManagers` containing bare `regex` was thought to need to be
`custom.regex`, which would have silently disabled every inline `# renovate:` marker. The
validator accepts `regex`, so that concern was unfounded and nothing here changes it.
* Nothing functionally references `renovate.json` by name — `grep -rn "renovate\.json" .gitea/ ci/
conformance/` returns only comment prose.
## What I did NOT verify
* **Renovate has still never completed a run.** This removes one of two blockers; PR #77 removes the
other. Neither is merged, so **no successful Renovate run has been observed** and I cannot say the
bot works — only that these two specific defects are fixed.
* **I did not re-run the workflow after the fix.** The honest test is a `workflow_dispatch` with
`dry_run: true` once both this and #77 are merged. Expect ~30 dependencies behind, rate-limited to
`prHourlyLimit: 2` / `prConcurrentLimit: 5`.
* **Whether the managers actually extract anything.** A valid config that matches no files is still
a no-op. The dry-run log's per-manager dependency counts are the only thing that proves the
`managerFilePatterns` are right, and I have not seen them.
* **Whether `renovate.json5` is honoured by the pinned Renovate version in the workflow.** JSON5
discovery is long-standing and the validator accepted the file, but the workflow's image tag was
not cross-checked against the version that added it.
* **Nothing in the gate catches an invalid Renovate config.** This bug would have recurred silently.
A one-line L0 check shelling out to `renovate-config-validator` would close it; it is not written
here, and `conformance/**` is a sensitive path.
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
@@ -20,10 +20,19 @@ restarted, and reboots are delegated to **kured** (`platform/kured`).
Deploying either alone produces a mechanism that looks healthy and does nothing: Deploying either alone produces a mechanism that looks healthy and does nothing:
- **kured alone** watches a file that nothing creates. Measured 2026-10-17: - **kured alone** watches a file that **nothing on Debian ever creates**. Measured 2026-10-18,
`update-notifier-common` — the package that owns the hook which writes after a first attempt to install Ubuntu's `update-notifier-common` failed on all three nodes
`/var/run/reboot-required` — was **not installed on any node**. kured would sit at 1/1 Running with *"No package matching 'update-notifier-common' is available"*:
reporting `kured_reboot_required=0` forever. - `apt-cache policy update-notifier-common` → `Candidate: (none)` — the package does not exist
in Debian; it is Ubuntu-only.
- `/usr/share/update-notifier/notify-reboot-required` → absent.
- **No `linux-image-*` postinst references `notify-reboot-required`** — calling it is an Ubuntu
patch, so even dropping a script at that path would never be invoked.
- No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
So **this role ships the writer itself** (`/usr/local/sbin/k3s-notify-reboot-required` plus a
`DPkg::Post-Invoke` hook) rather than installing a package. Without it kured would sit at 1/1
Running reporting `kured_reboot_required=0` forever.
- **this role alone** installs patches that a long-running process never picks up, and when a - **this role alone** installs patches that a long-running process never picks up, and when a
reboot *is* needed there is nothing safe to take it. `Automatic-Reboot "true"` is not the reboot *is* needed there is nothing safe to take it. `Automatic-Reboot "true"` is not the
answer: three control-plane nodes with etcd quorum 2/3, each rebooting at its own time, is how answer: three control-plane nodes with etcd quorum 2/3, each rebooting at its own time, is how
@@ -36,7 +45,7 @@ Measured on all three prod nodes, 2026-10-17:
| | | | | |
|---|---| |---|---|
| `unattended-upgrades` | **not installed** | | `unattended-upgrades` | **not installed** |
| `update-notifier-common` | **not installed** | | `update-notifier-common` | **not installable** — Ubuntu-only, `Candidate: (none)` on trixie |
| `needrestart` | **not installed** | | `needrestart` | **not installed** |
| `apt-mark showhold` | **empty** — no package held on any node | | `apt-mark showhold` | **empty** — no package held on any node |
| `/etc/apt/preferences.d/` | **empty** — no apt pin declares the backports-kernel intent | | `/etc/apt/preferences.d/` | **empty** — no apt pin declares the backports-kernel intent |
@@ -157,7 +166,9 @@ first two):
```bash ```bash
ssh [email protected] 'apt-config dump | grep -iE "unattended-upgrade::(automatic-reboot|package-blacklist|origins)"' ssh [email protected] 'apt-config dump | grep -iE "unattended-upgrade::(automatic-reboot|package-blacklist|origins)"'
ssh [email protected] 'systemctl list-timers apt-daily-upgrade.timer --no-pager' ssh [email protected] 'systemctl list-timers apt-daily-upgrade.timer --no-pager'
ssh [email protected] 'ls -la /var/run/reboot-required 2>&1; dpkg -l update-notifier-common needrestart unattended-upgrades | tail -4' ssh [email protected] 'ls -la /var/run/reboot-required* 2>&1 | head -2
ls -l /usr/local/sbin/k3s-notify-reboot-required /etc/apt/apt.conf.d/99-k3s-reboot-required 2>&1
dpkg -l needrestart unattended-upgrades | tail -3'
``` ```
Expect: `Automatic-Reboot "false"`, a blacklist containing `^linux-`, an `Origins-Pattern` Expect: `Automatic-Reboot "false"`, a blacklist containing `^linux-`, an `Origins-Pattern`
@@ -92,13 +92,11 @@ unattended_upgrades_enabled: false
unattended_upgrades_packages: unattended_upgrades_packages:
# The upgrader itself. # The upgrader itself.
- unattended-upgrades - unattended-upgrades
# THE SENTINEL WRITER, AND THE WHOLE REASON kured CAN WORK. On Debian, /var/run/reboot-required is # NOTE: `update-notifier-common` was listed here and REMOVED (2026-10-18). It is an UBUNTU
# not created by the kernel, by apt, or by unattended-upgrades — it is created by a hook shipped in # package and does not exist in Debian at all -- `apt-cache policy update-notifier-common` on
# update-notifier-common. Measured ABSENT on all three nodes. Without this package the file is # trixie returns `Candidate: (none)`, so the play failed outright with "No package matching
# never written, kured correctly never reboots, every dashboard stays green, and the reboot half of # 'update-notifier-common' is available" on all three nodes. The sentinel writer is now provided
# this PR is an elaborate no-op. The role asserts the hook exists after install rather than # by this role itself; see unattended_upgrades_sentinel_* below and tasks/main.yml.
# assuming the package shipped one (see tasks/main.yml).
- update-notifier-common
# Restarts services still mapped to a deleted library after a security upgrade. Without it a # Restarts services still mapped to a deleted library after a security upgrade. Without it a
# security-only policy patches bytes on disk that no running process reads — see DECISION 3. # security-only policy patches bytes on disk that no running process reads — see DECISION 3.
- needrestart - needrestart
@@ -170,3 +168,32 @@ unattended_upgrades_needrestart_blacklist:
unattended_upgrades_download_time: "03:30" unattended_upgrades_download_time: "03:30"
unattended_upgrades_upgrade_time: "04:00" unattended_upgrades_upgrade_time: "04:00"
unattended_upgrades_randomised_delay: 60m unattended_upgrades_randomised_delay: 60m
# ── THE REBOOT SENTINEL, PROVIDED BY THIS ROLE ──────────────────────────────────────────────────
# /var/run/reboot-required is the ONLY thing kured watches, and on Debian trixie **nothing creates
# it**. Measured on all three nodes 2026-10-18:
# * `update-notifier-common` (which owns that hook on Ubuntu) has `Candidate: (none)` — the
# package does not exist in Debian. Installing it is what broke the first run of this play.
# * `/usr/share/update-notifier/notify-reboot-required` is absent.
# * NO `linux-image-*` postinst references `notify-reboot-required` — calling it is an Ubuntu
# patch, not Debian behaviour, so merely dropping a script at that path would never be invoked.
# * No hook under /etc/apt/apt.conf.d or /etc/kernel/ mentions reboot-required.
# So the writer has to be ours. Without it kured is a permanent, INVISIBLE no-op: it stays 1/1
# Running and reports kured_reboot_required=0 forever, which is the "failure mode is a green tick"
# shape this repo keeps getting burned by.
#
# MECHANISM: a dpkg Post-Invoke hook runs a script that compares the RUNNING kernel against the
# newest INSTALLED one and touches the sentinel when they differ. Deliberately kernel-only, which
# matches the design: unattended-upgrades is security-only with `^linux-` blacklisted, so it will
# essentially never set this — the sentinel appears when the OPERATOR deliberately runs the
# node_baseline kernel play, and kured then automates only the mechanical reboot.
#
# It uses `linux-version` (from `linux-base`, a dependency of every linux-image package, so
# guaranteed present — measured `linux-base 4.12.1` installed) rather than parsing `dpkg -l`
# output, because kernel version ordering is not string ordering.
#
# /run is a tmpfs (measured), so the sentinel self-clears on boot: after a reboot the running
# kernel IS the newest and the script writes nothing. No cleanup task is needed, and none exists.
unattended_upgrades_sentinel_script: /usr/local/sbin/k3s-notify-reboot-required
unattended_upgrades_sentinel_hook: /etc/apt/apt.conf.d/99-k3s-reboot-required
@@ -35,9 +35,9 @@
tags: [packages, config, verify] tags: [packages, config, verify]
# ── PACKAGES ──────────────────────────────────────────────────────────────────────────────── # ── PACKAGES ────────────────────────────────────────────────────────────────────────────────
- name: Install unattended-upgrades, the reboot-sentinel writer and needrestart - name: Install unattended-upgrades and needrestart
# update-notifier-common is NOT optional padding: it owns the hook that creates # The reboot-sentinel writer is NOT a package here. On Debian nothing ships one -- see
# /var/run/reboot-required, which is the only thing kured watches. See defaults/main.yml. # defaults/main.yml -- so this role installs its own script and dpkg hook, two tasks below.
ansible.builtin.apt: ansible.builtin.apt:
name: "{{ unattended_upgrades_packages }}" name: "{{ unattended_upgrades_packages }}"
state: present state: present
@@ -45,6 +45,60 @@
cache_valid_time: 3600 cache_valid_time: 3600
tags: [packages] tags: [packages]
# ── THE REBOOT SENTINEL WRITER (ours, because Debian ships none) ────────────────────────────
# Full rationale and the four measurements behind it: defaults/main.yml. Short version: kured
# watches /var/run/reboot-required and nothing on Debian creates it, so without these two files
# the entire reboot half is an invisible no-op.
- name: Install the reboot-sentinel writer
ansible.builtin.copy:
dest: "{{ unattended_upgrades_sentinel_script }}"
owner: root
group: root
mode: "0755"
content: |
#!/bin/sh
# MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits are reverted.
#
# Touch /var/run/reboot-required when the RUNNING kernel is not the newest INSTALLED one.
# kured watches that path; see the role's defaults/main.yml for why this is ours to ship.
#
# `linux-version` comes from linux-base (a dependency of every linux-image package), and
# is used rather than sorting `dpkg -l` output because kernel version ordering is not
# string ordering. If it is somehow missing, exit 0 and write nothing: a dpkg hook must
# never fail an apt transaction, and a missing sentinel is caught by the verify assert.
set -eu
command -v linux-version >/dev/null 2>&1 || exit 0
running="$(uname -r)"
newest="$(linux-version list 2>/dev/null | linux-version sort --reverse 2>/dev/null | head -n1)"
# No newest (nothing installed we can read) -> nothing to say.
[ -n "$newest" ] || exit 0
if [ "$running" != "$newest" ]; then
touch /var/run/reboot-required 2>/dev/null || exit 0
# kured only needs the file, but the .pkgs companion is what an operator reads to find
# out WHY a node is waiting, so write a reason rather than an empty marker.
printf 'linux-image-%s\n' "$newest" > /var/run/reboot-required.pkgs 2>/dev/null || true
fi
exit 0
tags: [packages, config]
- name: Install the dpkg hook that runs the sentinel writer
# Post-Invoke, so it runs after dpkg has finished and a newly installed kernel is already
# visible to `linux-version list`. `|| true` so a failure here can never abort an apt
# transaction -- an unwritten sentinel is a missed reboot, but a broken apt is a broken node.
ansible.builtin.copy:
dest: "{{ unattended_upgrades_sentinel_hook }}"
owner: root
group: root
mode: "0644"
content: |
// MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits reverted.
// Writes /var/run/reboot-required when the running kernel is not the newest installed.
DPkg::Post-Invoke { "{{ unattended_upgrades_sentinel_script }} || true"; };
tags: [packages, config]
# ── CONFIG ────────────────────────────────────────────────────────────────────────────────── # ── CONFIG ──────────────────────────────────────────────────────────────────────────────────
# Written as a whole file rather than patched in place, for the reason node_baseline gives for # Written as a whole file rather than patched in place, for the reason node_baseline gives for
# its own apt source: an idempotent full-file write with a managed-by header is reviewable and # its own apt source: an idempotent full-file write with a managed-by header is reviewable and
@@ -246,21 +300,24 @@
Effective apt config confirmed: kernel blacklisted, automatic reboot disabled. Effective apt config confirmed: kernel blacklisted, automatic reboot disabled.
tags: [verify] tags: [verify]
- name: Find the hooks that can create /var/run/reboot-required # THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN.
# THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN, and # This was originally a `find` over /etc/kernel/postinst.d and /etc/apt/apt.conf.d, on the
# it is the one part that cannot be asserted from the repo — it depends on what # theory that the writer came from a package and the role could only discover whatever that
# update-notifier-common actually ships on this Debian release. Rather than hard-code a path # package happened to ship. That theory was wrong: no such package exists on Debian (see
# that may not exist, discover the candidates and assert that at least one is present. # defaults/main.yml), so the writer is now OURS and lives at two known paths. Checking two
ansible.builtin.find: # exact paths is strictly stronger than a glob -- a glob would have matched Debian's own
paths: # unattended-upgrades apt conffile and passed while no sentinel writer existed at all.
- /etc/kernel/postinst.d - name: Stat the sentinel writer script
- /etc/apt/apt.conf.d ansible.builtin.stat:
patterns: path: "{{ unattended_upgrades_sentinel_script }}"
- "*update-notifier*" register: unattended_upgrades_sentinel_script_stat
- "*unattended-upgrades*" changed_when: false
- "99update-notifier" tags: [verify]
file_type: any
register: unattended_upgrades_sentinel_hooks - name: Stat the sentinel dpkg hook
ansible.builtin.stat:
path: "{{ unattended_upgrades_sentinel_hook }}"
register: unattended_upgrades_sentinel_hook_stat
changed_when: false changed_when: false
tags: [verify] tags: [verify]
@@ -272,18 +329,21 @@
# before, so it is an assert and not a debug message. # before, so it is an assert and not a debug message.
ansible.builtin.assert: ansible.builtin.assert:
that: that:
- unattended_upgrades_sentinel_hooks.matched | int > 0 - unattended_upgrades_sentinel_script_stat.stat.exists
- unattended_upgrades_sentinel_script_stat.stat.executable | default(false)
- unattended_upgrades_sentinel_hook_stat.stat.exists
fail_msg: >- fail_msg: >-
NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required. No NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required.
update-notifier / unattended-upgrades hook was found under /etc/kernel/postinst.d or {{ unattended_upgrades_sentinel_script }} (executable) and
/etc/apt/apt.conf.d. kured watches that path and reboots only when it exists, so with no {{ unattended_upgrades_sentinel_hook }} must both exist -- this role ships both, because
writer the reboot automation is permanently and INVISIBLY idle -- kured stays 1/1 Running Debian provides no sentinel writer of its own (update-notifier-common is an UBUNTU
and reports kured_reboot_required=0 on every node forever. Confirm package and is not installable here). kured watches that path and reboots only when it
update-notifier-common is installed (`dpkg -l update-notifier-common`) and inspect exists, so with no writer the reboot automation is permanently and INVISIBLY idle --
`dpkg -L update-notifier-common | grep -E 'postinst.d|apt.conf.d'`. kured stays 1/1 Running and reports kured_reboot_required=0 on every node forever.
Re-run this role without `--tags verify` to install them.
success_msg: >- success_msg: >-
Reboot sentinel writer present ({{ unattended_upgrades_sentinel_hooks.matched }} hook(s) Reboot sentinel writer present ({{ unattended_upgrades_sentinel_script }} + dpkg hook)
found) -- a kernel or flagged package upgrade will signal kured. -- a kernel change will signal kured on the next apt transaction.
tags: [verify] tags: [verify]
- name: Report the pending-reboot state of this node - name: Report the pending-reboot state of this node
-99
View File
@@ -1,99 +0,0 @@
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17 RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags. It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009 split that gave image tags to Flux image-automation is formally closed — see .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write anyway; and enabling it would need an in-cluster git WRITE credential for the repo that defines the cluster). image-automation has been commented out at platform/kustomization.yaml:52 since it was authored and the cluster holds zero ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down. CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag` line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER: .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.",
"extends": [
"config:recommended",
":dependencyDashboard",
":semanticCommits",
"helpers:pinGitHubActionDigests"
],
"timezone": "America/New_York",
"schedule": ["after 3am and before 6am"],
"prConcurrentLimit": 5,
"prHourlyLimit": 2,
"commitMessagePrefix": "chore(deps):",
"labels": ["dependencies", "automated-update"],
"ignoreUnstable": true,
"respectLatest": true,
"enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
"flux": {
"managerFilePatterns": ["/(^|/)platform/.+\\.ya?ml$/", "/(^|/)apps/.+\\.ya?ml$/"]
},
"kustomize": {
"managerFilePatterns": ["/(^|/)kustomization\\.ya?ml$/"]
},
"_kubernetes_comment": "ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes manifests, and it is NOT auto-detected — it only looks at files matched here, which is why this block is as broad as it is. Scope covers every place this repo writes a container image: platform components, the rendered app waves, the ServiceDefinition catalog that generates them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the rendered output is reverted by the next render and would also fail ci/render-golden.sh. Renovate will raise BOTH, and both must land together.",
"kubernetes": {
"managerFilePatterns": [
"/(^|/)platform/.+\\.ya?ml$/",
"/(^|/)platform-config/.+\\.ya?ml$/",
"/(^|/)apps/.+\\.ya?ml$/",
"/(^|/)catalog/.+\\.ya?ml$/",
"/(^|/)runbooks/serving/.+\\.ya?ml$/",
"/(^|/)infra/agent/.+\\.ya?ml$/"
]
},
"customManagers": [
{
"customType": "regex",
"description": "Inline `# renovate: datasource=... depName=...` markers on pinned URLs / version literals (e.g. platform/kubevirt/kustomization.yaml release URLs, ci/versions.env, infra/devcontainer/versions.env).",
"managerFilePatterns": [
"/(^|/)kustomization\\.ya?ml$/",
"/(^|/)versions\\.env$/"
],
"matchStrings": [
"#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n.*?(?<currentValue>v?[0-9]+\\.[0-9]+\\.[0-9]+)"
],
"versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
}
],
"packageRules": [
{
"description": "Auto-merge the allowed low-risk classes on green CI — no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.",
"matchUpdateTypes": ["minor", "patch", "digest"],
"automerge": true,
"automergeType": "pr",
"platformAutomerge": true
},
{
"description": "MAJOR / potentially-breaking bumps require human review (docs/11 safety rail). Never auto-merged.",
"matchUpdateTypes": ["major"],
"automerge": false,
"addLabels": ["needs-review", "breaking"]
},
{
"description": "Keep the KubeVirt + CDI operator pair on a compatible track — grouped so they bump together (platform/kubevirt/kustomization.yaml).",
"matchDepNames": ["kubevirt/kubevirt", "kubevirt/containerized-data-importer"],
"groupName": "kubevirt-cdi"
},
{
"description": "Pinned tool versions (ci/versions.env, infra/devcontainer/versions.env) are a deliberate change surface — label but do not auto-merge; a version bump there changes CI reproducibility.",
"matchFileNames": ["ci/versions.env", "infra/devcontainer/versions.env"],
"automerge": false,
"addLabels": ["ci-tooling"]
},
{
"description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check — it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
"matchPackageNames": [
"kured",
"longhorn",
"cilium",
"cloudnative-pg",
"kube-prometheus-stack"
],
"automerge": false,
"addLabels": ["needs-review", "blast-radius"]
}
],
"_unmanageable_comment": "WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest, lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete version before any automation can maintain them, and until then they drift on every pull with nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it has been told not to leave; raising those ceilings is deliberate per-component work and is NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.",
"ignorePaths": [
"**/*.example",
"**/*.sops.yaml",
"conformance/**",
"platform/image-automation/per-service/**",
"platform/kyverno/tests/**"
],
"_ignorePaths_comment": "platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest` and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed shape, not software to keep current. `conformance/**` was already ignored for the same reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise have opened.",
"_docker_comment": "The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation over a running tag. That controller was never enabled and never will be here (see _comment), so the only thing those two flags were achieving was leaving every plain `image:` tag in the repo unmanaged. `docker` is left out of enabledManagers because this repo contains no Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.",
"docker": { "enabled": true }
}
+187
View File
@@ -0,0 +1,187 @@
// =================================================================================================
// Renovate configuration — JSON5, so the rationale can live NEXT TO what it explains.
// =================================================================================================
// THIS FILE WAS `renovate.json` AND IT WAS INVALID. It carried five `_comment` / `_*_comment`
// pseudo-comment keys, because JSON has no comments. Renovate REJECTS unknown top-level options, so
// the whole config failed validation and the first manual run of the workflow ERRORED OUT. Proven
// with the official validator (`ghcr.io/renovatebot/renovate renovate-config-validator`):
//
// ERROR: Found errors in configuration
// Invalid configuration option: _comment
// Invalid configuration option: _docker_comment
// Invalid configuration option: _ignorePaths_comment
// Invalid configuration option: _kubernetes_comment
// Invalid configuration option: _unmanageable_comment
//
// Renovate discovers `renovate.json5` natively (no RENOVATE_CONFIG_FILE needed, and nothing in
// .gitea/ or ci/ pins the old filename — only prose in comments), and JSON5 permits real `//`
// comments. So the fix keeps every word of the original rationale, positioned, instead of
// flattening it into a `description` string.
//
// VALIDATE BEFORE COMMITTING — it is one command and it is the only thing that catches this class:
// podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \
// ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json5
//
// NOTE on what was NOT wrong: `enabledManagers` containing bare `regex` was suspected of needing to
// be `custom.regex`. The validator accepts `regex`, so that suspicion was unfounded and nothing
// here changes it.
{ "$schema": "https://docs.renovatebot.com/renovate-schema.json",
// Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17
// RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags.
// It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009
// split that gave image tags to Flux image-automation is formally closed — see
// .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than
// opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with
// user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write
// anyway; and enabling it would need an in-cluster git WRITE credential for the repo that
// defines the cluster). image-automation has been commented out at
// platform/kustomization.yaml:52 since it was authored and the cluster holds zero
// ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down.
// CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid
// fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag`
// line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth
// sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER:
// .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.
"extends": ["config:recommended", ":dependencyDashboard", ":semanticCommits", "helpers:pinGitHubActionDigests"],
"timezone": "America/New_York",
"schedule": ["after 3am and before 6am"],
"prConcurrentLimit": 5,
"prHourlyLimit": 2,
"commitMessagePrefix": "chore(deps):",
"labels": ["dependencies", "automated-update"],
"ignoreUnstable": true,
"respectLatest": true,
"enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
"flux": {
"managerFilePatterns": [
"/(^|/)platform/.+\\.ya?ml$/",
"/(^|/)apps/.+\\.ya?ml$/"
]
},
"kustomize": {
"managerFilePatterns": [
"/(^|/)kustomization\\.ya?ml$/"
]
},
// ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes
// manifests, and it is NOT auto-detected — it only looks at files matched here, which is why
// this block is as broad as it is. Scope covers every place this repo writes a container image:
// platform components, the rendered app waves, the ServiceDefinition catalog that generates
// them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as
// `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the
// rendered output is reverted by the next render and would also fail ci/render-golden.sh.
// Renovate will raise BOTH, and both must land together.
"kubernetes": {
"managerFilePatterns": [
"/(^|/)platform/.+\\.ya?ml$/",
"/(^|/)platform-config/.+\\.ya?ml$/",
"/(^|/)apps/.+\\.ya?ml$/",
"/(^|/)catalog/.+\\.ya?ml$/",
"/(^|/)runbooks/serving/.+\\.ya?ml$/",
"/(^|/)infra/agent/.+\\.ya?ml$/"
]
},
"customManagers": [
{
"customType": "regex",
"description": "Inline `# renovate: datasource=... depName=...` markers on pinned URLs / version literals (e.g. platform/kubevirt/kustomization.yaml release URLs, ci/versions.env, infra/devcontainer/versions.env).",
"managerFilePatterns": [
"/(^|/)kustomization\\.ya?ml$/",
"/(^|/)versions\\.env$/"
],
"matchStrings": [
"#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n.*?(?<currentValue>v?[0-9]+\\.[0-9]+\\.[0-9]+)"
],
"versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
}
],
"packageRules": [
{
"description": "Auto-merge the allowed low-risk classes on green CI \u2014 no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.",
"matchUpdateTypes": [
"minor",
"patch",
"digest"
],
"automerge": true,
"automergeType": "pr",
"platformAutomerge": true
},
{
"description": "MAJOR / potentially-breaking bumps require human review (docs/11 safety rail). Never auto-merged.",
"matchUpdateTypes": [
"major"
],
"automerge": false,
"addLabels": [
"needs-review",
"breaking"
]
},
{
"description": "Keep the KubeVirt + CDI operator pair on a compatible track \u2014 grouped so they bump together (platform/kubevirt/kustomization.yaml).",
"matchDepNames": [
"kubevirt/kubevirt",
"kubevirt/containerized-data-importer"
],
"groupName": "kubevirt-cdi"
},
{
"description": "Pinned tool versions (ci/versions.env, infra/devcontainer/versions.env) are a deliberate change surface \u2014 label but do not auto-merge; a version bump there changes CI reproducibility.",
"matchFileNames": [
"ci/versions.env",
"infra/devcontainer/versions.env"
],
"automerge": false,
"addLabels": [
"ci-tooling"
]
},
{
"description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check \u2014 it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
"matchPackageNames": [
"kured",
"longhorn",
"cilium",
"cloudnative-pg",
"kube-prometheus-stack"
],
"automerge": false,
"addLabels": [
"needs-review",
"blast-radius"
]
}
],
// WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A
// FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it
// and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest,
// lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak
// (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete
// version before any automation can maintain them, and until then they drift on every pull with
// nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12,
// e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it
// has been told not to leave; raising those ceilings is deliberate per-component work and is
// NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.
// platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest`
// and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed
// shape, not software to keep current. `conformance/**` was already ignored for the same
// reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise
// have opened.
"ignorePaths": [
"**/*.example",
"**/*.sops.yaml",
"conformance/**",
"platform/image-automation/per-service/**",
"platform/kyverno/tests/**"
],
// The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were
// BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation
// over a running tag. That controller was never enabled and never will be here (see _comment),
// so the only thing those two flags were achieving was leaving every plain `image:` tag in the
// repo unmanaged. `docker` is left out of enabledManagers because this repo contains no
// Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.
"docker": {
"enabled": true
}
}