Run 350 cleared platform init (/user 200, /version 200, /repos 200), then stopped at
GET /api/v1/orgs/Home-Network with Gitea's own 403:
token does not have at least one of required scope(s), required=[read:organization],
token scope=read:package,write:repository,read:user
The workflow's header listed only those three scopes, and the operator minted exactly that.
Renovate's initRepo -> isOrg needs read:organization, and Gitea files /issues and /labels under
the issue scope, which the dashboard and PR labels need: write:issue.
- renovate.yaml: corrects the scope list (read:user, read:organization, write:repository,
write:issue, read:package), with the call each serves. A new preflight step probes them with
GETs, the token passed to curl on stdin, and names a missing scope instead of Renovate's bare
"Authentication failure".
- ci/lint.sh: rejects workflow `uses:` lines that need a Docker daemon (docker://,
renovatebot/github-action, docker/*-action), the class fixed in #84.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
renovatebot/github-action is a wrapper around `docker run`. On this runner a job has no Docker
daemon by decision (decisions/2026-10-29-image-pipeline.md), so every run, including dispatch
run 343, died before Renovate started:
failed to connect to the docker API at unix:///var/run/docker.sock ... no such file or directory
The pinned image ghcr.io/renovatebot/renovate:44.167.0 is now the job container, and a run step
execs renovate-entrypoint.sh. actions/checkout is dropped, since Renovate clones the repo itself.
An optional RENOVATE_GITHUB_COM_TOKEN secret feeds GITHUB_COM_TOKEN.
Local dry run (--platform=local, uid 12021, the step body verbatim): 179 deps extracted, 42
updates, and all 4 own-image digest lookups against the Gitea registry succeed.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a
workstation build/push. Nothing had ever been pushed to the Gitea registry.
- .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main
builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as
sha-<commit> + main, reading the digest back from the registry. Only changed images build.
- Each image's own build-image.sh does the build and in-image verification, now engine-neutral
(DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh.
- Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex
manager for them. The worker Job template is deliberately untracked (it would loop).
- renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the
current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION.
- Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in
Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
#79 renamed the config to renovate.json5 and left configurationFile pointing at the old
name. Runs 315-322 (2026-10-22..29) all failed in ~1s before Renovate started.
Also add a ci/lint.sh assertion that every workflow configurationFile: resolves to a real
path — valid YAML, so yamllint could not catch it, and the gate stayed green throughout.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
.gitea/workflows/renovate.yaml shipped with `cron: "15 3 * * *"`. Gitea Actions
cron is UTC, and renovate.json carries schedule ["after 3am and before 6am"]
with timezone America/New_York -- and THAT inner schedule is authoritative:
Renovate re-checks it and declines to open PRs outside it.
03:15 UTC -> 23:15 EDT, the PREVIOUS EVENING in New York
window -> 03:00-06:00 local => NOT INSIDE (3h45m early)
The original comment claimed the cron sat "slightly ahead" of the window. It
sat 3h45m behind it, on the previous day. Every nightly run would have started,
logged "not within schedule", opened ZERO PRs and exited SUCCESSFULLY -- a green
tick forever, indistinguishable from "nothing needed updating", hiding the ~30
pending updates PR #73 existed to surface.
Changed to `15 8 * * *`, chosen to survive DST rather than to look tidy:
EDT (UTC-4): local 03:00-06:00 == 07:00-10:00 UTC
EST (UTC-5): local 03:00-06:00 == 08:00-11:00 UTC
08:15 UTC is inside BOTH (04:15 EDT / 03:15 EST), verified by computation at
both extremes, so neither switchover needs an edit. A cron cannot follow DST.
Also confirmed while here: RENOVATE_TOKEN now exists as a repo Actions secret,
so the workflow's fail-hard preflight will pass.
STILL UNRESOLVED and the likelier remaining bug: renovate.json's
`enabledManagers` lists bare `regex`, where modern Renovate names the custom
manager `custom.regex`. If the bare name is ignored, every inline `# renovate:`
marker in the tree does nothing. Deliberately NOT changed -- guessing at a key
that could fail validation and take the whole run down is worse than a question
one dry run answers. See the handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two halves of one repeatable upgrade process. Neither works alone.
(a) OS upgrades and reboots
platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own
Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml:
that tier gates platform-config and every apps wave, so a wedged reboot daemon there
could stall every application. A component added to make maintenance safer must not be
able to cause an outage.
infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other
open branches, and this is a different lifecycle), enabled for prod in group_vars.
The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT
REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's
rejection of automating away the gate between nodes: u-u is security-only with the
kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file
appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then
does only the mechanical half, including the ordering rule node_baseline has to shout.
Quorum: concurrent reboots are structurally impossible, not merely unlikely.
concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency),
and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node
holds it forever and nothing cascades.
Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait,
serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate
must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m
(unconditional) plus kured's Prometheus alert gate (semantic).
alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and
under the chart default kured would have been blocked on day one by the five alerts
measured firing with no end in sight, and never rebooted anything.
Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes
from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel
upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it.
Asserted against apt-config dump, not against the file just written. 6.12 kernels are
still installed and bootable on all three nodes, and nothing was held.
Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING
could create the sentinel. kured alone would have idled forever at 1/1 Running.
platform/kured/alerts.yaml — three rules over kured's only metric, with a companion
promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md.
Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this
cluster's 7d Prometheus retention.
(b) Image/dependency automation
Renovate was aspirational: renovate.json present and complete, with no workflow, no
in-cluster workload, and its docker+kubernetes managers disabled — so every plain
image: tag was maintained by nothing while seven files claimed otherwise.
.gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe
first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a
skipped Gitea job reports success and that is how this stayed inert.
Renovate now owns both halves; Flux image-automation is formally closed: it pushes
commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here
anyway — main is protected user_can_push:false and the flux-system GitRepository has
no secretRef. Its controllers are installed and have idled 43 days with zero CRs.
conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing;
tightens the check for this component (requires service.create+metrics.create, both
chart-default false, plus a companion NetworkPolicy requirement). Required because
assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is
shrink-only.
Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author
approving review; /approve-override deliberately NOT posted.
NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role
never executed, and whether CNPG switches its primary on a drain-induced eviction (the
most likely way the first real reboot fails). Full list in the handoff entry.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
jakman85andClaude Opus 5.5