Files
home-network-k3s-impl/.gitea/workflows/renovate.yaml
T
jakman85andClaude Opus 5 a2440a3125
gate / gate (pull_request) Successful in 1m57s
gate / conformance (pull_request) Has been skipped
feat(upgrades): automated OS reboots (kured) + security-only unattended-upgrades, and give Renovate a runner
Two halves of one repeatable upgrade process. Neither works alone.

(a) OS upgrades and reboots

platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own
Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml:
that tier gates platform-config and every apps wave, so a wedged reboot daemon there
could stall every application. A component added to make maintenance safer must not be
able to cause an outage.

infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other
open branches, and this is a different lifecycle), enabled for prod in group_vars.

The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT
REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's
rejection of automating away the gate between nodes: u-u is security-only with the
kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file
appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then
does only the mechanical half, including the ordering rule node_baseline has to shout.

Quorum: concurrent reboots are structurally impossible, not merely unlikely.
concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency),
and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node
holds it forever and nothing cascades.

Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait,
serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate
must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m
(unconditional) plus kured's Prometheus alert gate (semantic).

alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and
under the chart default kured would have been blocked on day one by the five alerts
measured firing with no end in sight, and never rebooted anything.

Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes
from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel
upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it.
Asserted against apt-config dump, not against the file just written. 6.12 kernels are
still installed and bootable on all three nodes, and nothing was held.

Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING
could create the sentinel. kured alone would have idled forever at 1/1 Running.

platform/kured/alerts.yaml — three rules over kured's only metric, with a companion
promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md.
Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this
cluster's 7d Prometheus retention.

(b) Image/dependency automation

Renovate was aspirational: renovate.json present and complete, with no workflow, no
in-cluster workload, and its docker+kubernetes managers disabled — so every plain
image: tag was maintained by nothing while seven files claimed otherwise.

.gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe
first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a
skipped Gitea job reports success and that is how this stayed inert.

Renovate now owns both halves; Flux image-automation is formally closed: it pushes
commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here
anyway — main is protected user_can_push:false and the flux-system GitRepository has
no secretRef. Its controllers are installed and have idled 43 days with zero CRs.

conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing;
tightens the check for this component (requires service.create+metrics.create, both
chart-default false, plus a companion NetworkPolicy requirement). Required because
assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is
shrink-only.

Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author
approving review; /approve-override deliberately NOT posted.

NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role
never executed, and whether CNPG switches its primary on a drain-induced eviction (the
most likely way the first real reboot fails). Full list in the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 00:10:29 -05:00

174 lines
11 KiB
YAML

# Renovate — the dependency/image update automation. SCHEDULED, self-hosted against this Gitea.
#
# ═══════════════════════════════════════════════════════════════════════════════════════════════
# WHY THIS FILE EXISTS: renovate.json HAS BEEN IN THIS REPO AND HAS NEVER ONCE RUN.
# ═══════════════════════════════════════════════════════════════════════════════════════════════
# Measured 2026-10-17: `renovate.json` is present, complete and thoughtfully written — it declares
# managers, schedules, automerge classes and grouping. Nothing executes it. There is no Renovate
# workflow (this is the first), no Renovate CronJob or Deployment in the cluster
# (`kubectl get cronjob,deploy,job -A | grep -i renovate` -> nothing), and no bot account doing it
# from outside. Meanwhile SEVEN files carry comments asserting "Renovate maintains this pin"
# (platform/homepage/deployment.yaml:40, runbooks/serving/deployment.yaml:87,
# platform/crafty/deployment.yaml:63, platform/vpn-ingress/daemonset.yaml:259,
# platform/keycloak/postgres-cluster.yaml:75, apps/wave-0/nextcloud-db.yaml:302, and
# platform/headlamp/helmrepository.yaml:2). Those comments have been false for the life of the file.
#
# The cost is measurable. With nothing bumping anything: the six per-app oauth2-proxy auth sidecars
# sit on v7.8.1 while the platform oauth2-proxy in the SAME cluster runs v7.15.5; Keycloak (the IdP)
# runs 26.0.6, a November-2024 release, 8 minors behind; Velero runs a RELEASE CANDIDATE
# (velero-plugin-for-aws v1.14.3-rc.2) with a stable available; the CNPG Postgres image is 16.4,
# missing every 16.5+ security release. A pin nothing maintains is not a pin, it is a fossil.
#
# ═══════════════════════════════════════════════════════════════════════════════════════════════
# WHY RENOVATE AND NOT FLUX IMAGE-AUTOMATION (the decision, and it reverses ADR-0009's split)
# ═══════════════════════════════════════════════════════════════════════════════════════════════
# ADR-0009 divided the work: Renovate owns Helm chart/dependency bumps, Flux image-automation owns
# per-service container image TAGS, deliberately non-overlapping. The second half of that split
# cannot be delivered on this repo's terms, for three measured reasons:
#
# 1. IT WOULD BYPASS THE MERGE GATE, WHICH CLAUDE.md §1 FORBIDS. ImageUpdateAutomation does not
# open pull requests; it COMMITS AND PUSHES. Its default target is the source's own branch —
# `main` — so it would write straight past the PR flow. It can be pointed at a side branch
# instead, but then nothing opens the PR and bumps silently accumulate on a branch nobody
# merges. Either way a human or a second bot has to close the loop.
# 2. ON THIS CLUSTER IT CANNOT EVEN DO THAT. Measured 2026-10-17: `main` is protected with
# `user_can_push: false`, and the `flux-system` GitRepository has NO `spec.secretRef` at all —
# the source is anonymous and read-only. Enabling write-back means minting an in-cluster git
# WRITE credential for the repository that defines the cluster, which is a materially larger
# blast radius than a CI-scoped token.
# 3. RENOVATE ALREADY OPENS PRs NATIVELY, so every change it proposes goes through the gate with
# no extra machinery.
#
# So: Renovate owns BOTH halves, and Flux image-automation stays disabled. Note this is a change of
# plan, not a change of state — image-automation has been commented out at
# platform/kustomization.yaml:52 since it was authored, and the cluster has ZERO
# ImageRepository/ImagePolicy/ImageUpdateAutomation objects (measured). Its two controllers ARE
# installed and have idled for 43 days. Nothing is being torn down here; a path that was never
# taken is being formally closed, and renovate.json's `docker`/`kubernetes` managers — disabled
# purely to avoid fighting a controller that never ran — are switched on to fill the gap.
#
# ═══════════════════════════════════════════════════════════════════════════════════════════════
# THIS WORKFLOW MUST NOT BE ABLE TO SUCCEED SILENTLY
# ═══════════════════════════════════════════════════════════════════════════════════════════════
# A SKIPPED Gitea job reports `success` (CLAUDE.md §2), so "green" is not evidence this ran. Two
# guards:
# * The preflight step FAILS HARD (exit 1) if RENOVATE_TOKEN is absent, rather than skipping.
# A missing credential on a dependency bot is indistinguishable from "nothing to update", and
# that is precisely how renovate.json stayed inert for months.
# * The summary step records what ran into the job summary unconditionally.
# What this still does NOT give you is an alert when the SCHEDULE stops firing. See the handoff
# entry's "What I did NOT verify" — that gap is stated, not papered over.
#
# OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE:
# `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository` and
# `read:user`. Set it as an Actions secret on this repository. Nothing in this repo mints,
# stores or references its value; it is consumed by name only.
# Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a
# visible red is the correct state for "automation configured but not credentialed".
#
# THE FIRST RUN WILL WANT TO OPEN A LOT OF PRs. ~30 dependencies are behind. renovate.json caps the
# blast radius (`prConcurrentLimit: 5`, `prHourlyLimit: 2`), but run it manually with
# `dry_run: true` FIRST and read the log before letting the schedule loose. That is what
# workflow_dispatch's default is for.
name: renovate
on:
schedule:
# 03:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` in
# America/New_York, and that inner schedule is the authoritative one — Renovate re-checks it and
# will decline to open PRs outside it. The two are deliberately both present: this cron decides
# when the process RUNS, renovate.json decides when it may ACT. Keeping the cron slightly ahead
# of the repo window means a run is always in flight when the window opens.
- cron: "15 3 * * *"
workflow_dispatch:
inputs:
dry_run:
description: "Dry run (no PRs, no branches). Leave true for the first run."
required: false
default: "true"
log_level:
description: "Renovate LOG_LEVEL (info | debug)"
required: false
default: "info"
concurrency:
# Never two Renovate runs at once: concurrent runs race on the same branches and can open
# duplicate PRs. Not cancel-in-progress — a half-finished Renovate run can leave a branch pushed
# with no PR, so let the first one finish.
group: renovate
cancel-in-progress: false
jobs:
renovate:
runs-on: ubuntu-latest
# Renovate scans every manager across the whole repo; 40m is generous for this repo's size and
# well under a runner's patience. Without a timeout a wedged run holds the `renovate`
# concurrency group and silently blocks every subsequent scheduled run.
timeout-minutes: 40
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Preflight - the token must exist, and its absence must be LOUD
env:
RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
run: |
set -euo pipefail
if [ -z "${RENOVATE_TOKEN:-}" ]; then
echo "::error::RENOVATE_TOKEN is not set on this repository." >&2
cat >&2 <<'MSG'
Renovate cannot run without a Gitea access token, and this step fails rather than
skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would make
"dependency automation is not running" look exactly like "everything is up to date".
That is the failure mode that let renovate.json sit inert in this repo for months while
seven files claimed "Renovate maintains this pin".
TO FIX: create a Gitea access token for the bot account with scopes
`write:repository` and `read:user`, then add it to this repository as the Actions
secret RENOVATE_TOKEN. Do not commit it anywhere.
MSG
exit 1
fi
echo "RENOVATE_TOKEN is present (value not printed)."
- name: Run Renovate
uses: renovatebot/[email protected]
with:
# Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers
# includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`.
# This is the one dependency in the repo whose updater is its own updater.
configurationFile: renovate.json
token: ${{ secrets.RENOVATE_TOKEN }}
env:
# Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
# header said were "set at runtime by the operator's renovate runner" — this is that
# runner, so they are set here rather than in the repo config, which keeps renovate.json
# portable and free of this instance's URL.
RENOVATE_PLATFORM: gitea
RENOVATE_ENDPOINT: https://gitea.home.keithjackson.dev/api/v1
RENOVATE_REPOSITORIES: Home-Network/home-network-k3s-impl
# Renovate must never push to a protected branch; it works on its own branches and opens
# PRs against this base. Stated explicitly rather than inherited, because this is the
# property that keeps the whole mechanism inside CLAUDE.md §1's branch -> PR -> gate path.
RENOVATE_BASE_BRANCHES: main
RENOVATE_ONBOARDING: "false"
RENOVATE_REQUIRE_CONFIG: "required"
RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }}
LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }}
- name: Summary
if: always()
run: |
{
echo "## Renovate"
echo
echo "- trigger: \`${{ github.event_name }}\`"
echo "- dry run: \`${{ github.event_name == 'workflow_dispatch' && inputs.dry_run || 'false' }}\`"
echo "- outcome: \`${{ job.status }}\`"
echo
echo "Renovate owns BOTH Helm chart bumps and container image tags here; Flux"
echo "image-automation is deliberately disabled (platform/kustomization.yaml)."
echo "Every change it proposes arrives as a pull request and must pass the merge gate."
} >> "$GITHUB_STEP_SUMMARY"