Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5
|
713b0c3dba |
No files matched your search
@@ -73,7 +73,7 @@ spec:
|
||||
- name: FTLCONF_dns_rateLimit_count
|
||||
value: '0'
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
|
||||
value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
|
||||
- name: FTLCONF_webserver_api_password
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
|
||||
+1
-32
@@ -143,39 +143,8 @@ env:
|
||||
# resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
|
||||
# failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
|
||||
# every client that takes the ADVERTISED connection cannot reach the server at all.
|
||||
#
|
||||
# ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
|
||||
# MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
|
||||
#
|
||||
# dig home.keithjackson.dev @192.168.2.23 (pihole) -> 192.168.2.20 TTL=0
|
||||
# dig home.keithjackson.dev @1.1.1.1 (public) -> wan.home.keithjackson.dev TTL=300
|
||||
# dig home.keithjackson.dev @192.168.2.1 (router) -> wan.... TTL=153 -> 148 -> 143
|
||||
#
|
||||
# dnsmasq answers `address=` entries with **TTL 0** unless told otherwise, and that is the whole
|
||||
# bug. The SOHO router forwards LAN queries to PiHole and CACHES the result, so:
|
||||
# * the CORRECT answer is UNCACHEABLE (TTL 0) -- the router must re-query PiHole on every
|
||||
# single lookup, which is maximum exposure to any momentary gap; while
|
||||
# * the WRONG answer is CACHEABLE FOR 5 MINUTES -- the instant the router falls back to its
|
||||
# upstream (standard forwarder behaviour when the forward target does not answer) it latches
|
||||
# the public WAN address and serves it to EVERY LAN CLIENT for the public record's 300s TTL.
|
||||
# The decrementing TTL on the router's reply above is the proof it caches rather than forwarding
|
||||
# live. So ONE sub-second PiHole gap -- a pod restart, an L2 announcement lease moving, an ARP
|
||||
# flap, a kured node reboot -- becomes FIVE MINUTES of the whole house resolving to the WAN IP.
|
||||
# That asymmetry, not the gap, is why this is visible as "flapping" rather than a blip.
|
||||
#
|
||||
# 60s is a deliberate floor, not a tuning knob: long enough to ride out a pod restart and a lease
|
||||
# re-election (both seconds), short enough that a genuine VIP change still propagates inside a
|
||||
# minute. These VIPs are pinned by name (`lbipam.cilium.io/ips`) and effectively static, so the
|
||||
# propagation cost is theoretical while the flap it prevents is measured. If flaps persist across
|
||||
# LONGER outages, raising this toward the public record's 300 would cover them -- at the cost of
|
||||
# the same delay on a real change. It does NOT help when PiHole is down longer than the TTL: the
|
||||
# cache expires, the router falls back, and the public answer latches again. That is a real
|
||||
# outage, and the fix for it is PiHole availability, not DNS caching.
|
||||
#
|
||||
# Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
|
||||
# configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
|
||||
value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
|
||||
# THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
|
||||
# apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
|
||||
# carries the full rationale and the measurement that found it.
|
||||
|
||||
@@ -1,82 +0,0 @@
|
||||
---
|
||||
date: 2026-10-18
|
||||
branch: fix/pihole-local-ttl
|
||||
title: "The split-horizon answer was uncacheable while the wrong answer cached for 5 minutes"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The operator reported DNS "flapping between the cluster ingress and the router (public IP)".
|
||||
PiHole was not at fault — both replicas answer correctly and consistently. The bug is an
|
||||
**asymmetry in cacheability** that we created.
|
||||
|
||||
Measured on prod 2026-10-18:
|
||||
|
||||
```
|
||||
dig home.keithjackson.dev @192.168.2.23 (pihole) -> 192.168.2.20 TTL=0
|
||||
dig home.keithjackson.dev @1.1.1.1 (public) -> wan.home.keithjackson.dev TTL=300
|
||||
dig home.keithjackson.dev @192.168.2.1 (router) -> wan.... TTL=153 -> 148 -> 143
|
||||
```
|
||||
|
||||
dnsmasq answers `address=` entries with **TTL 0** unless told otherwise. The SOHO router forwards
|
||||
LAN queries to PiHole and **caches** the result — the decrementing TTL above is the proof it caches
|
||||
rather than forwarding live. So:
|
||||
|
||||
* the **correct** answer is **uncacheable** (TTL 0): the router must re-query PiHole on *every*
|
||||
lookup, which is maximum exposure to any momentary gap;
|
||||
* the **wrong** answer is **cacheable for 300s**: the instant the router falls back to its upstream
|
||||
(standard forwarder behaviour when the forward target does not answer) it latches the public WAN
|
||||
address and serves it to **every LAN client** for five minutes.
|
||||
|
||||
So **one sub-second PiHole gap becomes five minutes of the whole house resolving to the WAN IP** —
|
||||
then it expires, the router re-queries PiHole, gets TTL 0 again, and is immediately re-exposed.
|
||||
That asymmetry, not the gap itself, is why this is visible as flapping rather than a blip.
|
||||
|
||||
Fix: `local-ttl=60` prepended to `FTLCONF_misc_dnsmasq_lines` in `catalog/pihole.yaml`, so the
|
||||
private answer is cacheable and a short gap is invisible to clients. `apps/wave-0/pihole.yaml`
|
||||
regenerated with the canonical render command.
|
||||
|
||||
## What I verified
|
||||
|
||||
* **PiHole is NOT the fault.** Both replicas, queried directly by pod IP, answer `192.168.2.20`;
|
||||
the VIP returned it 6/6; this workstation 8/8. DHCP hands out only `192.168.2.23`.
|
||||
* **The router caches rather than forwards live** — three successive queries returned TTL 153, 148,
|
||||
143 (decrementing), which a pure forwarder could not do.
|
||||
* **The TTL asymmetry**, as quoted above: 0 from PiHole, 300 from public DNS.
|
||||
* **The router does answer DNS and returns the public chain** (`wan.home.keithjackson.dev` → the WAN
|
||||
address), so a fallback always has a wrong answer available.
|
||||
* **`local-ttl` is the right directive**: dnsmasq applies it to replies made from configuration,
|
||||
`/etc/hosts` and DHCP leases, which covers all five `address=` names with one directive. It is a
|
||||
standalone option, not a sub-field of `address=`, so prepending it to the `;`-separated
|
||||
`FTLCONF_misc_dnsmasq_lines` list is syntactically correct.
|
||||
* **The render is clean and scoped**: `catalog/render.py --all catalog -o apps --wave-layout
|
||||
--no-image-automation` (the canonical command `ci/render-golden.sh` asserts against, and
|
||||
deliberately **not** `--emit-waves`) changed exactly two files — `catalog/pihole.yaml` and
|
||||
`apps/wave-0/pihole.yaml` — and the rendered diff is the single `local-ttl=60;` prefix.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **UNAPPLIED.** Nothing has been reconciled, so I have not observed PiHole serving a non-zero TTL,
|
||||
nor the router caching the private answer, nor the flap stopping. **Verify after merge with
|
||||
`dig home.keithjackson.dev @192.168.2.23` (expect `TTL=60`, not 0) and then the same query at the
|
||||
router (expect `192.168.2.20` with a decrementing TTL, not `wan.…`).**
|
||||
* **I did not confirm that PiHole/FTL accepts `local-ttl` via `FTLCONF_misc_dnsmasq_lines`.** The
|
||||
directive is valid dnsmasq and the field is a passthrough, but `catalog/pihole.yaml`'s own header
|
||||
records that FTL has previously written a bogus line into `dnsmasq.conf` and reported "0 invalid"
|
||||
— so a malformed passthrough fails *silently*. If the TTL is still 0 after merge, that is the
|
||||
cause, not this reasoning. Check the pod's `/etc/pihole/dnsmasq.conf`.
|
||||
* **60s is chosen, not derived.** Long enough for a pod restart and a lease re-election (both
|
||||
seconds); it does **nothing** for an outage longer than the TTL, where the cache expires and the
|
||||
public answer latches again. That is a real availability problem, not a caching one.
|
||||
* **This does not remove the trigger.** `192.168.2.23` is served by **two** LoadBalancer Services
|
||||
(`pihole-dns-tcp`, `pihole-dns-udp`) with **two independently-elected L2 announcement leases**, so
|
||||
two nodes can ARP for the same address at once — they sat on different nodes earlier the same day
|
||||
(tcp on prod-2, udp on prod-1). The same pattern applies to crafty (`.22`) and unifi (`.21`).
|
||||
Merging each pair into one mixed-protocol Service would remove the class; the operator has parked
|
||||
that decision, and it is deliberately **not** in this change.
|
||||
* **Client-side bypasses are unexamined.** A device with its own resolver, or a browser with
|
||||
DNS-over-HTTPS, bypasses PiHole entirely and always gets the public answer; this fix cannot help
|
||||
there. Only this one workstation's resolver chain was inspected.
|
||||
* **No alert fires if the split-horizon answer regresses to the public address.** Nothing probes
|
||||
`home.keithjackson.dev` from a LAN vantage point and compares it against `${GATEWAY_VIP}`, so the
|
||||
next recurrence would again be found by a human noticing. That gap is not closed here.
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
date: 2026-10-18
|
||||
branch: fix/renovate-invalid-config
|
||||
title: "renovate.json was invalid, so the first manual run errored out"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The operator ran the Renovate workflow manually and it **failed**. The cause is not the workflow:
|
||||
**`renovate.json` was never a valid Renovate config.**
|
||||
|
||||
It carried five pseudo-comment keys — `_comment`, `_kubernetes_comment`, `_unmanageable_comment`,
|
||||
`_ignorePaths_comment`, `_docker_comment` — because JSON has no comments. Renovate **rejects unknown
|
||||
top-level options**, so the entire config failed validation. Proven with the official validator
|
||||
rather than inferred:
|
||||
|
||||
```
|
||||
$ podman run --rm -v "$PWD/renovate.json:/tmp/renovate.json:ro,Z" \
|
||||
ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json
|
||||
ERROR: Found errors in configuration
|
||||
Invalid configuration option: _comment
|
||||
Invalid configuration option: _docker_comment
|
||||
Invalid configuration option: _ignorePaths_comment
|
||||
Invalid configuration option: _kubernetes_comment
|
||||
Invalid configuration option: _unmanageable_comment
|
||||
```
|
||||
|
||||
This is the second independent reason Renovate had never worked, on top of PR #77's cron firing
|
||||
outside `renovate.json`'s own schedule window. Either alone was sufficient to produce "nothing ever
|
||||
happens"; #73 shipped both.
|
||||
|
||||
**Converted `renovate.json` → `renovate.json5`.** JSON5 permits real `//` comments, and Renovate
|
||||
discovers `renovate.json5` natively — nothing in `.gitea/` or `ci/` pins the old filename (only
|
||||
prose inside comments), and no `RENOVATE_CONFIG_FILE` is set. Every word of the original rationale
|
||||
is preserved as comments positioned next to what it explains, which a flattened `description`
|
||||
string would have lost.
|
||||
|
||||
A header on the new file records the failure, the validator command, and the instruction to run it
|
||||
before committing — this class of bug is invisible to the repo's own gate.
|
||||
|
||||
## What I verified
|
||||
|
||||
* **The exact failure**, from the official validator, quoted above. Not read off a log, not inferred
|
||||
from the workflow.
|
||||
* **The fix validates**: the same validator against `renovate.json5` returns
|
||||
`INFO: Config validated successfully against 1 file(s)`.
|
||||
* **Nothing was dropped in the conversion**, checked programmatically against `git show
|
||||
HEAD:renovate.json`: 18 non-comment keys before, 18 after, **no missing keys, no extra keys, and
|
||||
no key whose value changed**. `docker: {enabled: true}` confirmed still present — the validator
|
||||
prints a normalisation diff that mentions it, which is its own migration suggestion, not a
|
||||
removal.
|
||||
* **74 comment lines survive** in the new file, so the rationale is intact.
|
||||
* **`RENOVATE_TOKEN` exists** as a repository Actions secret (listed by name via the Gitea API), so
|
||||
the workflow's fail-hard preflight is satisfied.
|
||||
* **A suspicion was DISPROVED:** `enabledManagers` containing bare `regex` was thought to need to be
|
||||
`custom.regex`, which would have silently disabled every inline `# renovate:` marker. The
|
||||
validator accepts `regex`, so that concern was unfounded and nothing here changes it.
|
||||
* Nothing functionally references `renovate.json` by name — `grep -rn "renovate\.json" .gitea/ ci/
|
||||
conformance/` returns only comment prose.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **Renovate has still never completed a run.** This removes one of two blockers; PR #77 removes the
|
||||
other. Neither is merged, so **no successful Renovate run has been observed** and I cannot say the
|
||||
bot works — only that these two specific defects are fixed.
|
||||
* **I did not re-run the workflow after the fix.** The honest test is a `workflow_dispatch` with
|
||||
`dry_run: true` once both this and #77 are merged. Expect ~30 dependencies behind, rate-limited to
|
||||
`prHourlyLimit: 2` / `prConcurrentLimit: 5`.
|
||||
* **Whether the managers actually extract anything.** A valid config that matches no files is still
|
||||
a no-op. The dry-run log's per-manager dependency counts are the only thing that proves the
|
||||
`managerFilePatterns` are right, and I have not seen them.
|
||||
* **Whether `renovate.json5` is honoured by the pinned Renovate version in the workflow.** JSON5
|
||||
discovery is long-standing and the validator accepted the file, but the workflow's image tag was
|
||||
not cross-checked against the version that added it.
|
||||
* **Nothing in the gate catches an invalid Renovate config.** This bug would have recurred silently.
|
||||
A one-line L0 check shelling out to `renovate-config-validator` would close it; it is not written
|
||||
here, and `conformance/**` is a sensitive path.
|
||||
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
|
||||
@@ -1,99 +0,0 @@
|
||||
{
|
||||
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
|
||||
"_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17 RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags. It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009 split that gave image tags to Flux image-automation is formally closed — see .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write anyway; and enabling it would need an in-cluster git WRITE credential for the repo that defines the cluster). image-automation has been commented out at platform/kustomization.yaml:52 since it was authored and the cluster holds zero ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down. CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag` line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER: .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.",
|
||||
"extends": [
|
||||
"config:recommended",
|
||||
":dependencyDashboard",
|
||||
":semanticCommits",
|
||||
"helpers:pinGitHubActionDigests"
|
||||
],
|
||||
"timezone": "America/New_York",
|
||||
"schedule": ["after 3am and before 6am"],
|
||||
"prConcurrentLimit": 5,
|
||||
"prHourlyLimit": 2,
|
||||
"commitMessagePrefix": "chore(deps):",
|
||||
"labels": ["dependencies", "automated-update"],
|
||||
"ignoreUnstable": true,
|
||||
"respectLatest": true,
|
||||
"enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
|
||||
"flux": {
|
||||
"managerFilePatterns": ["/(^|/)platform/.+\\.ya?ml$/", "/(^|/)apps/.+\\.ya?ml$/"]
|
||||
},
|
||||
"kustomize": {
|
||||
"managerFilePatterns": ["/(^|/)kustomization\\.ya?ml$/"]
|
||||
},
|
||||
"_kubernetes_comment": "ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes manifests, and it is NOT auto-detected — it only looks at files matched here, which is why this block is as broad as it is. Scope covers every place this repo writes a container image: platform components, the rendered app waves, the ServiceDefinition catalog that generates them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the rendered output is reverted by the next render and would also fail ci/render-golden.sh. Renovate will raise BOTH, and both must land together.",
|
||||
"kubernetes": {
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)platform/.+\\.ya?ml$/",
|
||||
"/(^|/)platform-config/.+\\.ya?ml$/",
|
||||
"/(^|/)apps/.+\\.ya?ml$/",
|
||||
"/(^|/)catalog/.+\\.ya?ml$/",
|
||||
"/(^|/)runbooks/serving/.+\\.ya?ml$/",
|
||||
"/(^|/)infra/agent/.+\\.ya?ml$/"
|
||||
]
|
||||
},
|
||||
"customManagers": [
|
||||
{
|
||||
"customType": "regex",
|
||||
"description": "Inline `# renovate: datasource=... depName=...` markers on pinned URLs / version literals (e.g. platform/kubevirt/kustomization.yaml release URLs, ci/versions.env, infra/devcontainer/versions.env).",
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)kustomization\\.ya?ml$/",
|
||||
"/(^|/)versions\\.env$/"
|
||||
],
|
||||
"matchStrings": [
|
||||
"#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n.*?(?<currentValue>v?[0-9]+\\.[0-9]+\\.[0-9]+)"
|
||||
],
|
||||
"versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
|
||||
}
|
||||
],
|
||||
"packageRules": [
|
||||
{
|
||||
"description": "Auto-merge the allowed low-risk classes on green CI — no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.",
|
||||
"matchUpdateTypes": ["minor", "patch", "digest"],
|
||||
"automerge": true,
|
||||
"automergeType": "pr",
|
||||
"platformAutomerge": true
|
||||
},
|
||||
{
|
||||
"description": "MAJOR / potentially-breaking bumps require human review (docs/11 safety rail). Never auto-merged.",
|
||||
"matchUpdateTypes": ["major"],
|
||||
"automerge": false,
|
||||
"addLabels": ["needs-review", "breaking"]
|
||||
},
|
||||
{
|
||||
"description": "Keep the KubeVirt + CDI operator pair on a compatible track — grouped so they bump together (platform/kubevirt/kustomization.yaml).",
|
||||
"matchDepNames": ["kubevirt/kubevirt", "kubevirt/containerized-data-importer"],
|
||||
"groupName": "kubevirt-cdi"
|
||||
},
|
||||
{
|
||||
"description": "Pinned tool versions (ci/versions.env, infra/devcontainer/versions.env) are a deliberate change surface — label but do not auto-merge; a version bump there changes CI reproducibility.",
|
||||
"matchFileNames": ["ci/versions.env", "infra/devcontainer/versions.env"],
|
||||
"automerge": false,
|
||||
"addLabels": ["ci-tooling"]
|
||||
},
|
||||
{
|
||||
"description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check — it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
|
||||
"matchPackageNames": [
|
||||
"kured",
|
||||
"longhorn",
|
||||
"cilium",
|
||||
"cloudnative-pg",
|
||||
"kube-prometheus-stack"
|
||||
],
|
||||
"automerge": false,
|
||||
"addLabels": ["needs-review", "blast-radius"]
|
||||
}
|
||||
],
|
||||
"_unmanageable_comment": "WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest, lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete version before any automation can maintain them, and until then they drift on every pull with nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it has been told not to leave; raising those ceilings is deliberate per-component work and is NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.",
|
||||
"ignorePaths": [
|
||||
"**/*.example",
|
||||
"**/*.sops.yaml",
|
||||
"conformance/**",
|
||||
"platform/image-automation/per-service/**",
|
||||
"platform/kyverno/tests/**"
|
||||
],
|
||||
"_ignorePaths_comment": "platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest` and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed shape, not software to keep current. `conformance/**` was already ignored for the same reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise have opened.",
|
||||
"_docker_comment": "The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation over a running tag. That controller was never enabled and never will be here (see _comment), so the only thing those two flags were achieving was leaving every plain `image:` tag in the repo unmanaged. `docker` is left out of enabledManagers because this repo contains no Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.",
|
||||
"docker": { "enabled": true }
|
||||
}
|
||||
+187
@@ -0,0 +1,187 @@
|
||||
// =================================================================================================
|
||||
// Renovate configuration — JSON5, so the rationale can live NEXT TO what it explains.
|
||||
// =================================================================================================
|
||||
// THIS FILE WAS `renovate.json` AND IT WAS INVALID. It carried five `_comment` / `_*_comment`
|
||||
// pseudo-comment keys, because JSON has no comments. Renovate REJECTS unknown top-level options, so
|
||||
// the whole config failed validation and the first manual run of the workflow ERRORED OUT. Proven
|
||||
// with the official validator (`ghcr.io/renovatebot/renovate renovate-config-validator`):
|
||||
//
|
||||
// ERROR: Found errors in configuration
|
||||
// Invalid configuration option: _comment
|
||||
// Invalid configuration option: _docker_comment
|
||||
// Invalid configuration option: _ignorePaths_comment
|
||||
// Invalid configuration option: _kubernetes_comment
|
||||
// Invalid configuration option: _unmanageable_comment
|
||||
//
|
||||
// Renovate discovers `renovate.json5` natively (no RENOVATE_CONFIG_FILE needed, and nothing in
|
||||
// .gitea/ or ci/ pins the old filename — only prose in comments), and JSON5 permits real `//`
|
||||
// comments. So the fix keeps every word of the original rationale, positioned, instead of
|
||||
// flattening it into a `description` string.
|
||||
//
|
||||
// VALIDATE BEFORE COMMITTING — it is one command and it is the only thing that catches this class:
|
||||
// podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \
|
||||
// ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json5
|
||||
//
|
||||
// NOTE on what was NOT wrong: `enabledManagers` containing bare `regex` was suspected of needing to
|
||||
// be `custom.regex`. The validator accepts `regex`, so that suspicion was unfounded and nothing
|
||||
// here changes it.
|
||||
{ "$schema": "https://docs.renovatebot.com/renovate-schema.json",
|
||||
// Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17
|
||||
// RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags.
|
||||
// It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009
|
||||
// split that gave image tags to Flux image-automation is formally closed — see
|
||||
// .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than
|
||||
// opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with
|
||||
// user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write
|
||||
// anyway; and enabling it would need an in-cluster git WRITE credential for the repo that
|
||||
// defines the cluster). image-automation has been commented out at
|
||||
// platform/kustomization.yaml:52 since it was authored and the cluster holds zero
|
||||
// ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down.
|
||||
// CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid
|
||||
// fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag`
|
||||
// line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth
|
||||
// sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER:
|
||||
// .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.
|
||||
"extends": ["config:recommended", ":dependencyDashboard", ":semanticCommits", "helpers:pinGitHubActionDigests"],
|
||||
"timezone": "America/New_York",
|
||||
"schedule": ["after 3am and before 6am"],
|
||||
"prConcurrentLimit": 5,
|
||||
"prHourlyLimit": 2,
|
||||
"commitMessagePrefix": "chore(deps):",
|
||||
"labels": ["dependencies", "automated-update"],
|
||||
"ignoreUnstable": true,
|
||||
"respectLatest": true,
|
||||
"enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
|
||||
"flux": {
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)platform/.+\\.ya?ml$/",
|
||||
"/(^|/)apps/.+\\.ya?ml$/"
|
||||
]
|
||||
},
|
||||
"kustomize": {
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)kustomization\\.ya?ml$/"
|
||||
]
|
||||
},
|
||||
// ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes
|
||||
// manifests, and it is NOT auto-detected — it only looks at files matched here, which is why
|
||||
// this block is as broad as it is. Scope covers every place this repo writes a container image:
|
||||
// platform components, the rendered app waves, the ServiceDefinition catalog that generates
|
||||
// them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as
|
||||
// `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the
|
||||
// rendered output is reverted by the next render and would also fail ci/render-golden.sh.
|
||||
// Renovate will raise BOTH, and both must land together.
|
||||
"kubernetes": {
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)platform/.+\\.ya?ml$/",
|
||||
"/(^|/)platform-config/.+\\.ya?ml$/",
|
||||
"/(^|/)apps/.+\\.ya?ml$/",
|
||||
"/(^|/)catalog/.+\\.ya?ml$/",
|
||||
"/(^|/)runbooks/serving/.+\\.ya?ml$/",
|
||||
"/(^|/)infra/agent/.+\\.ya?ml$/"
|
||||
]
|
||||
},
|
||||
"customManagers": [
|
||||
{
|
||||
"customType": "regex",
|
||||
"description": "Inline `# renovate: datasource=... depName=...` markers on pinned URLs / version literals (e.g. platform/kubevirt/kustomization.yaml release URLs, ci/versions.env, infra/devcontainer/versions.env).",
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)kustomization\\.ya?ml$/",
|
||||
"/(^|/)versions\\.env$/"
|
||||
],
|
||||
"matchStrings": [
|
||||
"#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n.*?(?<currentValue>v?[0-9]+\\.[0-9]+\\.[0-9]+)"
|
||||
],
|
||||
"versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
|
||||
}
|
||||
],
|
||||
"packageRules": [
|
||||
{
|
||||
"description": "Auto-merge the allowed low-risk classes on green CI \u2014 no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.",
|
||||
"matchUpdateTypes": [
|
||||
"minor",
|
||||
"patch",
|
||||
"digest"
|
||||
],
|
||||
"automerge": true,
|
||||
"automergeType": "pr",
|
||||
"platformAutomerge": true
|
||||
},
|
||||
{
|
||||
"description": "MAJOR / potentially-breaking bumps require human review (docs/11 safety rail). Never auto-merged.",
|
||||
"matchUpdateTypes": [
|
||||
"major"
|
||||
],
|
||||
"automerge": false,
|
||||
"addLabels": [
|
||||
"needs-review",
|
||||
"breaking"
|
||||
]
|
||||
},
|
||||
{
|
||||
"description": "Keep the KubeVirt + CDI operator pair on a compatible track \u2014 grouped so they bump together (platform/kubevirt/kustomization.yaml).",
|
||||
"matchDepNames": [
|
||||
"kubevirt/kubevirt",
|
||||
"kubevirt/containerized-data-importer"
|
||||
],
|
||||
"groupName": "kubevirt-cdi"
|
||||
},
|
||||
{
|
||||
"description": "Pinned tool versions (ci/versions.env, infra/devcontainer/versions.env) are a deliberate change surface \u2014 label but do not auto-merge; a version bump there changes CI reproducibility.",
|
||||
"matchFileNames": [
|
||||
"ci/versions.env",
|
||||
"infra/devcontainer/versions.env"
|
||||
],
|
||||
"automerge": false,
|
||||
"addLabels": [
|
||||
"ci-tooling"
|
||||
]
|
||||
},
|
||||
{
|
||||
"description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check \u2014 it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
|
||||
"matchPackageNames": [
|
||||
"kured",
|
||||
"longhorn",
|
||||
"cilium",
|
||||
"cloudnative-pg",
|
||||
"kube-prometheus-stack"
|
||||
],
|
||||
"automerge": false,
|
||||
"addLabels": [
|
||||
"needs-review",
|
||||
"blast-radius"
|
||||
]
|
||||
}
|
||||
],
|
||||
// WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A
|
||||
// FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it
|
||||
// and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest,
|
||||
// lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak
|
||||
// (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete
|
||||
// version before any automation can maintain them, and until then they drift on every pull with
|
||||
// nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12,
|
||||
// e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it
|
||||
// has been told not to leave; raising those ceilings is deliberate per-component work and is
|
||||
// NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.
|
||||
// platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest`
|
||||
// and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed
|
||||
// shape, not software to keep current. `conformance/**` was already ignored for the same
|
||||
// reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise
|
||||
// have opened.
|
||||
"ignorePaths": [
|
||||
"**/*.example",
|
||||
"**/*.sops.yaml",
|
||||
"conformance/**",
|
||||
"platform/image-automation/per-service/**",
|
||||
"platform/kyverno/tests/**"
|
||||
],
|
||||
// The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were
|
||||
// BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation
|
||||
// over a running tag. That controller was never enabled and never will be here (see _comment),
|
||||
// so the only thing those two flags were achieving was leaving every plain `image:` tag in the
|
||||
// repo unmanaged. `docker` is left out of enabledManagers because this repo contains no
|
||||
// Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.
|
||||
"docker": {
|
||||
"enabled": true
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user