fix(images): name the measured runner blocker; the job container gets no Docker daemon

PR #81's first images run failed in 6s: act_runner 0.2.11 forces docker_host '-' for a tcp:// DOCKER_HOST, and gitea/act drops every bind when valid_volumes is empty. Giving jobs the privileged DinD daemon is node-root for any workflow author, so it is left to the operator; options are in the handoff entry. The driver's failure now names the cause. Co-Authored-By: Claude Opus 5.5 <[email protected]>
feat(images): build, verify and publish the three images in CI; Renovate carries the digests
2026-10-29 21:44:38 -05:00 · 2026-10-29 21:34:06 -05:00 · 2026-10-18 14:29:34 +00:00 · 2026-10-18 14:28:59 +00:00 · 2026-10-18 08:19:28 +00:00 · 2026-10-18 03:12:28 -05:00
34 changed files with 2016 additions and 353 deletions
@@ -115,14 +115,47 @@ CI. That is a real coverage gap, not a phase.
 | variable | `REQUIRE_CLUSTER_TESTS` | `'true'` (Phase 1+) makes unverified cluster-needing touched reqs a hard block |
 | secret | `CONFORMANCE_TRIGGER_KUBECONFIG` | narrow creds to create/await the in-cluster Chainsaw Job (not cluster-admin) |

+## How images are built
+
+This repo's three images — `toolbox` (`infra/devcontainer/`), `agent-runner` (`infra/agent/`,
+built `FROM` the toolbox) and `conformance-runner` (`conformance/incluster/`) — are built,
+verified and published by [`images.yaml`](images.yaml), never from a workstation
+(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh); each
+image is built by **its own** `build-image.sh`, the same script a developer runs locally, so the
+in-image verification (non-root uid, read-only rootfs, tool presence, `PM_SELFTEST=1
+pm-dispatch`, the `conformance-clone` refusals) exists once.
+
+| Trigger | What happens |
+|---|---|
+| pull request touching an image's inputs | build + verify only. Nothing is pushed and **no registry credential is referenced** |
+| push to `main` touching an image's inputs | build + verify + publish to `gitea.home.keithjackson.dev/home-network/<image>` as `sha-<commit>` (immutable) and `main` (moving); the **registry-returned** digest is in the job summary |
+| `workflow_dispatch` | the same as a push when run on `main` (all images by default); verify-only elsewhere |
+
+Only images whose inputs changed are built. The publish job **never writes to git**: Renovate
+follows each image's `main` tag through a `# renovate-digest:` marker next to every digest
+reference (`platform/pm-agent/`, `infra/agent/versions.env`, `conformance/versions.env`) and
+opens the bump PR, which goes through the gate like anything else. A PR opened with the Actions
+token would not trigger the required gate, which is why it is Renovate's job and not this one's.
+
+Credential: the repository secret **`REGISTRY_PUSH_TOKEN`** (scope `write:package`, owned by an
+account with package write on the org). The automatic Actions token cannot push in Gitea 1.24.3
+— the reasoning, with source citations, is in the header of `images.yaml`. Retention:
+[`infra/gitea/registry-retention.sh`](../../infra/gitea/registry-retention.sh), an operator
+action, keeps every digest git pins plus the newest tags.
+
+Cost on this runner: the job shares the single act_runner slot with the gate, and the DinD store
+is a node `emptyDir`; the job prunes it on exit.
+
 ## Files

 ```
 .gitea/workflows/
  merge-gate.yaml            the pipeline (lint → l0 → l1 → conformance → decision)
+  images.yaml                build/verify/publish this repo's container images (see above)
  scripts/
    install-tools.sh         installs the pinned tools per stage (from ci/versions.env)
    decide.sh                the R-TEST-03 merge decision over job results
+    images.sh                the image pipeline's driver (what to build, base, push, digest)
  README.md                  this file
 ```

@@ -0,0 +1,183 @@
+# Container images — built, verified and published by CI, never from a workstation.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# WHY THIS FILE EXISTS (operator rule, 2026-10-29)
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# "Container image creation, hosting and lifecycle must be part of CI/CD and the codebase, never a
+# one-off build/push from a workstation." Before this workflow, three images existed only as
+# workstation builds and NOTHING had ever been pushed to the Gitea registry: the Home-Network org's
+# package list was empty, platform/pm-agent/ pinned an all-zero digest and sat in ImagePullBackOff,
+# and conformance/versions.env carried an empty digest. Both build scripts said "publishing is an
+# operator action" — a step with no log, no recorded inputs and no verification anyone could read.
+#
+# WHAT IT DOES — the logic is in scripts/images.sh; this file only wires triggers and secrets.
+#   pull_request -> build + VERIFY only. No push, and no registry credential is referenced.
+#   push to main -> build + verify + PUBLISH to gitea.home.keithjackson.dev/home-network/<image>
+#   dispatch     -> same as a push when run on main; verify-only on any other ref.
+# Images: toolbox (infra/devcontainer), agent-runner (infra/agent, FROM toolbox) and
+# conformance-runner (conformance/incluster). Only images whose inputs changed are built.
+#
+# THE DIGEST LIFECYCLE — this workflow NEVER writes to git.
+#   1. publish pushes `sha-<commit>` (immutable) and `main` (moving), and prints the digest the
+#      REGISTRY returns in the job summary.
+#   2. Renovate (renovate.yaml) follows the `main` tag of each image through the
+#      `# renovate-digest:` markers next to every digest reference, and opens the bump PR.
+#   3. That PR goes through the gate like any other change, and Flux deploys it.
+# The bump PR is Renovate's and not this job's because a PR opened with the Actions token does not
+# trigger workflows, so it could never pass the REQUIRED gate check; Renovate's token is a real
+# user's.
+#
+# THE PUSH CREDENTIAL — `REGISTRY_PUSH_TOKEN`, a repository Actions secret, referenced BY NAME.
+# The automatic Actions token CANNOT be used. Gitea 1.24.3 source, verified:
+#   * services/auth/basic.go (Basic.Verify) turns a task token into the pseudo-user
+#     `gitea-actions` (ActionsUserID -2) and sets NO token scope;
+#   * routers/api/packages/container/container.go (Authenticate, the /v2/token exchange every
+#     docker client uses) then requires read:package / write:package / all scope and answers 401;
+#   * services/context/package.go (determineAccessMode) has `// TODO: ActionUser permission check`
+#     and no branch for that user, so even past login it would get at most READ on a public org.
+# So: a personal access token with scope `write:package`, minted by an account that has package
+# WRITE on the Home-Network org (org owner, or a team with the Packages unit at write). A dedicated
+# bot account in a packages-only team is the least-privilege choice. The username sent with it is
+# not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER`
+# is optional and only labels the login.
+#
+# ┌─ BLOCKER, MEASURED ON PR #81 (run 325): THIS RUNNER CANNOT BUILD IMAGES YET ───────────────┐
+# │ The job container gets no Docker daemon. act_runner 0.2.11 forces container.docker_host to  │
+# │ "-" whenever DOCKER_HOST is tcp:// (internal/app/cmd/daemon.go), and with valid_volumes: []  │
+# │ gitea/act strips every bind including the socket (pkg/container/docker_run.go             │
+# │ sanitizeConfig). So this job fails at its first check, loudly, until the operator chooses   │
+# │ how workflow code may reach a daemon — see decisions/2026-10-29-image-pipeline.md. It is    │
+# │ not a required check, so it does not block merges; it is red, which is the honest state.   │
+# └──────────────────────────────────────────────────────────────────────────────────────────────┘
+#
+# THE RUNNER (apps/wave-1/gitea-actions.yaml): act_runner 0.2.11, one job at a time, Docker
+# executor on a privileged `docker:27-dind` sidecar whose image store is an emptyDir on the node
+# (no sizeLimit) and whose memory limit is 2Gi — shared by the job container AND every build
+# container. The toolbox is ~1.4 GB uncompressed and agent-runner adds ~0.3 GB on top of it, so the
+# cleanup step below is not optional: without it each run would leave ~3 GB on a node that had
+# 7 GB free when this was written, and the next eviction would take the runner with it.
+name: images
+
+on:
+  pull_request:
+    branches: [main]
+    paths:
+      - infra/devcontainer/Containerfile
+      - infra/devcontainer/versions.env
+      - infra/devcontainer/build-image.sh
+      - infra/agent/**
+      - conformance/incluster/Containerfile
+      - conformance/incluster/conformance-clone
+      - conformance/incluster/build-image.sh
+      - .gitea/workflows/images.yaml
+      - .gitea/workflows/scripts/images.sh
+  push:
+    branches: [main]
+    paths:
+      - infra/devcontainer/Containerfile
+      - infra/devcontainer/versions.env
+      - infra/devcontainer/build-image.sh
+      - infra/agent/**
+      - conformance/incluster/Containerfile
+      - conformance/incluster/conformance-clone
+      - conformance/incluster/build-image.sh
+      - .gitea/workflows/images.yaml
+      - .gitea/workflows/scripts/images.sh
+  workflow_dispatch:
+    inputs:
+      images:
+        description: "Which images: 'all', or a space-separated subset of: toolbox agent-runner conformance-runner"
+        required: false
+        default: "all"
+
+concurrency:
+  # Never cancel a publish half-way: a pushed `sha-` tag with no matching `main` move is
+  # recoverable (a rerun skips what is already published) but pointless to provoke.
+  group: images-${{ github.ref }}
+  cancel-in-progress: false
+
+jobs:
+  images:
+    runs-on: ubuntu-latest
+    # Measured on the PR that introduced this file — see its decisions entry. Generous because a
+    # cold DinD store downloads the Fedora base and every pinned tool on each run.
+    timeout-minutes: 75
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0   # the per-image change detection diffs against the base commit
+
+      # ── pull requests: build + verify, NO credential ───────────────────────────────────────
+      - name: Build and verify (pull request — nothing is pushed)
+        if: github.event_name == 'pull_request'
+        env:
+          MODE: verify
+          IMAGES: auto
+          PR_BASE_REF: ${{ github.base_ref }}
+        run: |
+          set -euo pipefail
+          BASE_SHA="$(git merge-base "origin/${PR_BASE_REF}" HEAD)"
+          export BASE_SHA
+          ./.gitea/workflows/scripts/images.sh
+
+      # ── non-main dispatch: verify only ──────────────────────────────────────────────────────
+      - name: Build and verify (dispatch off main — nothing is pushed)
+        if: github.event_name == 'workflow_dispatch' && github.ref != 'refs/heads/main'
+        env:
+          MODE: verify
+          IMAGES: ${{ inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images }}
+        run: ./.gitea/workflows/scripts/images.sh
+
+      # ── main: publish ───────────────────────────────────────────────────────────────────────
+      - name: Preflight - the push token must exist, and its absence must be LOUD
+        if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
+        env:
+          REGISTRY_PUSH_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
+        run: |
+          set -euo pipefail
+          if [ -z "${REGISTRY_PUSH_TOKEN:-}" ]; then
+            echo "::error::REGISTRY_PUSH_TOKEN is not set on this repository." >&2
+            cat >&2 <<'MSG'
+
+          Images cannot be published without a registry credential, and this step fails rather
+          than skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would
+          make "nothing was published" look exactly like "published". That is how this repo went
+          a month with images that existed only on one workstation.
+
+          WHY NOT THE BUILT-IN ACTIONS TOKEN: Gitea 1.24.3 gives it no package scope at the
+          registry's /v2/token exchange (routers/api/packages/container/container.go,
+          Authenticate) — see the header of .gitea/workflows/images.yaml.
+
+          TO FIX: as an account with package WRITE on the Home-Network org (ideally a dedicated
+          bot in a packages-only team), create an access token with scope `write:package` and
+          nothing else, then add it to this repository as the Actions secret
+          REGISTRY_PUSH_TOKEN. Optionally set the Actions variable REGISTRY_PUSH_USER to that
+          account's name. Do not commit the token anywhere. Then re-run this workflow
+          (workflow_dispatch on main, images: all).
+          MSG
+            exit 1
+          fi
+          echo "REGISTRY_PUSH_TOKEN is present (value not printed)."
+
+      - name: Build, verify and publish (main)
+        if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
+        env:
+          MODE: publish
+          IMAGES: ${{ github.event_name == 'workflow_dispatch' && (inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images) || 'auto' }}
+          BASE_SHA: ${{ github.event.before }}
+          REGISTRY_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
+          REGISTRY_USER: ${{ vars.REGISTRY_PUSH_USER }}
+        run: ./.gitea/workflows/scripts/images.sh
+
+      # ── always: give the DinD store back ────────────────────────────────────────────────────
+      # The runner's image store is a node emptyDir with no sizeLimit (see header). Everything this
+      # job built or pulled is removed; `prune -a` cannot remove the job's own image, because the
+      # job container is still running on it.
+      - name: Clean the DinD image store
+        if: always()
+        run: |
+          docker logout gitea.home.keithjackson.dev >/dev/null 2>&1 || true
+          docker image prune -af >/dev/null || true
+          docker builder prune -af >/dev/null || true
+          docker system df || true
@@ -60,8 +60,9 @@
 # entry's "What I did NOT verify" — that gap is stated, not papered over.
 #
 # OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE:
-#   `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository` and
-#   `read:user`. Set it as an Actions secret on this repository. Nothing in this repo mints,
+#   `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository`,
+#   `read:user` and (since 2026-10-29) `read:package`. Set it as an Actions secret on this
+#   repository. Nothing in this repo mints,
 #   stores or references its value; it is consumed by name only.
 #   Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a
 #   visible red is the correct state for "automation configured but not credentialed".
@@ -74,12 +75,26 @@ name: renovate

 on:
  schedule:
-    # 03:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` in
-    # America/New_York, and that inner schedule is the authoritative one — Renovate re-checks it and
-    # will decline to open PRs outside it. The two are deliberately both present: this cron decides
-    # when the process RUNS, renovate.json decides when it may ACT. Keeping the cron slightly ahead
-    # of the repo window means a run is always in flight when the window opens.
-    - cron: "15 3 * * *"
+    # 08:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` with
+    # `"timezone": "America/New_York"`, and THAT inner schedule is authoritative — Renovate re-checks
+    # it and DECLINES TO OPEN PRs outside it. So this cron must fire INSIDE the local window, not
+    # merely near it.
+    #
+    # WAS `15 3 * * *`, WHICH NEVER WOULD HAVE WORKED (fixed 2026-10-18, before the first scheduled
+    # run). Gitea Actions cron is UTC, so 03:15 UTC is 23:15 the PREVIOUS EVENING in New York --
+    # 3h45m BEFORE the window opens, not "slightly ahead" of it as the original comment claimed.
+    # Every nightly run would have started, logged "not within schedule", opened zero PRs and
+    # exited SUCCESSFULLY. A green tick forever, and no updates ever: the exact "failure mode is a
+    # green tick" shape this repo keeps getting burned by, and indistinguishable from "nothing
+    # needed updating".
+    #
+    # THE WINDOW IN UTC MOVES WITH US DST, and a cron cannot follow it:
+    #   EDT (UTC-4, Mar-Nov): local 03:00-06:00 == 07:00-10:00 UTC
+    #   EST (UTC-5, Nov-Mar): local 03:00-06:00 == 08:00-11:00 UTC
+    # 08:15 UTC is the only quarter-hour comfortably inside BOTH (04:15 EDT / 03:15 EST), so the
+    # schedule keeps working across both switchovers without an edit. Do not "tidy" this back to an
+    # early-morning-UTC value without redoing this arithmetic.
+    - cron: "15 8 * * *"
  workflow_dispatch:
    inputs:
      dry_run:
@@ -125,8 +140,9 @@ jobs:
          seven files claimed "Renovate maintains this pin".

          TO FIX: create a Gitea access token for the bot account with scopes
-          `write:repository` and `read:user`, then add it to this repository as the Actions
-          secret RENOVATE_TOKEN. Do not commit it anywhere.
+          `write:repository`, `read:user` and `read:package` (the last one lets Renovate read
+          this repo's own images in the Gitea registry), then add it to this repository as the
+          Actions secret RENOVATE_TOKEN. Do not commit it anywhere.
          MSG
            exit 1
          fi
@@ -140,6 +156,12 @@ jobs:
          # This is the one dependency in the repo whose updater is its own updater.
          configurationFile: renovate.json
          token: ${{ secrets.RENOVATE_TOKEN }}
+          # PINNED 2026-10-29. Unset, this action version runs `ghcr.io/renovatebot/renovate:39`
+          # (its action.yml default), i.e. 39.264.0 — and renovate.json5 is written for, and
+          # validated against, a release that knows `managerFilePatterns` (>= 40.2.0). Under 39.x
+          # the config fails validation with five errors and the run stops at CONFIG_VALIDATION.
+          # Validate renovate.json5 with THIS tag; see its header for the command.
+          renovate-version: 44.167.0
        env:
          # Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
          # header said were "set at runtime by the operator's renovate runner" — this is that
@@ -156,6 +178,14 @@ jobs:
          RENOVATE_REQUIRE_CONFIG: "required"
          RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }}
          LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }}
+          # ADDED 2026-10-29: registry read access for this repo's OWN images (toolbox,
+          # agent-runner, conformance-runner — published by .gitea/workflows/images.yaml and
+          # followed through `# renovate-digest:` markers). The Gitea registry answers 401 to
+          # anonymous reads, and Renovate scopes the platform token to hostType `gitea`, which a
+          # docker lookup never matches — so without this rule every digest lookup fails. Same
+          # token; it therefore needs scope `read:package` in addition to the ones listed in the
+          # preflight above. Gitea ignores the username for token auth (services/auth/basic.go).
+          RENOVATE_HOST_RULES: '[{"hostType":"docker","matchHost":"gitea.home.keithjackson.dev","username":"renovate","password":"${{ secrets.RENOVATE_TOKEN }}"}]'

      - name: Summary
        if: always()
@@ -0,0 +1,296 @@
+#!/usr/bin/env bash
+###############################################################################
+# .gitea/workflows/scripts/images.sh — the image pipeline's driver (images.yaml calls this).
+#
+# Builds, verifies and (on main) PUBLISHES the three images this repo ships:
+#
+#   toolbox             infra/devcontainer/   the pinned dev/gate toolchain
+#   agent-runner        infra/agent/          FROM toolbox; the PM dispatcher + worker
+#   conformance-runner  conformance/incluster/ the in-cluster L2/L3 Chainsaw runner
+#
+# Each image is built by ITS OWN build-image.sh — the same script a developer runs locally —
+# so the in-image verification (non-root uid, read-only rootfs, tool presence, pm-dispatch
+# selftest, conformance-clone refusals, ...) is written once and runs in both places. This
+# file only decides WHAT to build, in which order, FROM which base, and where it goes.
+#
+# MODES (env MODE):
+#   verify   build + verify, never push, holds NO credential. Pull requests.
+#   publish  build + verify + push to ${REGISTRY}/${NAMESPACE}/<image> with tags
+#            `sha-<12 hex>` (immutable: never re-pushed) and `main` (moving; Renovate follows
+#            it). The digest recorded is the one the REGISTRY returns, read back after the push.
+#
+# WHAT GETS BUILT (env IMAGES): `auto` (default) diffs BASE_SHA..HEAD_SHA against each image's
+# inputs; otherwise a space-separated list. A change to the pipeline itself (images.yaml, this
+# file) builds everything in verify mode, and nothing extra in publish mode: re-publishing
+# unchanged inputs would only mint new digests and churn Renovate PRs.
+#
+# THE agent-runner BASE — the one ordering rule, and the reason this is not three jobs:
+#   * toolbox built in this run  -> agent-runner builds FROM it (in publish mode, FROM the
+#     digest the registry just returned for it).
+#   * verify mode, toolbox not changed -> the toolbox is built anyway as a base. A pull request
+#     holds no registry credential, so it cannot pull the published one.
+#   * publish mode, toolbox not changed -> AGENT_BASE_IMAGE from infra/agent/versions.env,
+#     i.e. the published toolbox digest that Renovate keeps current. If that is still the
+#     all-zero placeholder the toolbox has never been published, so it is built and published
+#     first (this is how the very first run bootstraps).
+#
+# LOOP-FREEDOM. Renovate turns each publish into a digest-bump PR. Those bumps land in
+# platform/pm-agent/ (no image input), conformance/versions.env (no image input) and
+# infra/agent/versions.env (agent-runner input -> ONE more agent-runner build, whose own bump
+# lands in platform/pm-agent/ and stops). The worker Job template is deliberately NOT tracked:
+# it is baked INTO agent-runner, so bumping it would rebuild agent-runner, mint a new digest,
+# bump it again, forever. pm-dispatch overwrites that field at runtime anyway.
+###############################################################################
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
+cd "${REPO_ROOT}"
+
+MODE="${MODE:-verify}"
+IMAGES="${IMAGES:-auto}"
+ENGINE="${ENGINE:-docker}"
+REGISTRY="${REGISTRY:-gitea.home.keithjackson.dev}"
+NAMESPACE="${NAMESPACE:-home-network}"
+HEAD_SHA="${HEAD_SHA:-$(git rev-parse HEAD)}"
+BASE_SHA="${BASE_SHA:-}"
+SUMMARY="${GITHUB_STEP_SUMMARY:-/dev/null}"
+ZERO_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000"
+
+ALL_IMAGES=(toolbox agent-runner conformance-runner)
+SHA_TAG="sha-${HEAD_SHA:0:12}"
+MOVING_TAG="main"
+
+log()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
+info() { printf '    %s\n' "$*"; }
+die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; printf -- '- **ERROR**: %s\n' "$*" >>"${SUMMARY}"; exit 1; }
+
+case "${MODE}" in verify|publish) ;; *) die "MODE must be verify or publish, got '${MODE}'" ;; esac
+
+# --- inputs per image: a change under any of these rebuilds that image ---------------------
+# conformance-runner takes its chainsaw/kubectl/fedora pins from infra/devcontainer/versions.env.
+# conformance/versions.env is NOT an input (it only RECORDS the published digest), and must not
+# become one: Renovate writes it after every publish, and an input there would loop.
+inputs() {
+  case "$1" in
+    toolbox)            echo "infra/devcontainer/Containerfile infra/devcontainer/versions.env infra/devcontainer/build-image.sh" ;;
+    agent-runner)       echo "infra/agent/" ;;
+    conformance-runner) echo "conformance/incluster/Containerfile conformance/incluster/conformance-clone conformance/incluster/build-image.sh infra/devcontainer/versions.env" ;;
+  esac
+}
+PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh"
+
+declare -A WANT=() BUILT=() DIGEST=() SECS=() NOTE=()
+
+changed_any() { # <paths...> -> 0 if BASE_SHA..HEAD_SHA touches any
+  # shellcheck disable=SC2086
+  [ -n "$(git diff --name-only "${BASE_SHA}" "${HEAD_SHA}" -- "$@")" ]
+}
+
+select_images() {
+  if [ "${IMAGES}" != "auto" ]; then
+    for i in ${IMAGES}; do
+      case " ${ALL_IMAGES[*]} " in *" ${i} "*) WANT[$i]=1; NOTE[$i]="requested" ;; *) die "unknown image '${i}'" ;; esac
+    done
+    return
+  fi
+  if [ -z "${BASE_SHA}" ] || [[ "${BASE_SHA}" =~ ^0+$ ]] || ! git cat-file -e "${BASE_SHA}^{commit}" 2>/dev/null; then
+    info "no usable diff base ('${BASE_SHA}'): building everything"
+    for i in "${ALL_IMAGES[@]}"; do WANT[$i]=1; NOTE[$i]="no diff base"; done
+    return
+  fi
+  info "diff base ${BASE_SHA} .. ${HEAD_SHA}"
+  local i
+  for i in "${ALL_IMAGES[@]}"; do
+    # shellcheck disable=SC2046
+    if changed_any $(inputs "$i"); then WANT[$i]=1; NOTE[$i]="inputs changed"; fi
+  done
+  # shellcheck disable=SC2086
+  if [ "${MODE}" = verify ] && changed_any ${PIPELINE_FILES}; then
+    for i in "${ALL_IMAGES[@]}"; do [ -n "${WANT[$i]:-}" ] || { WANT[$i]=1; NOTE[$i]="pipeline changed"; }; done
+  fi
+}
+
+# --- registry helpers (publish mode only) ----------------------------------------------------
+REG_TOKEN=""
+registry_token() {
+  [ -n "${REG_TOKEN}" ] && return 0
+  REG_TOKEN="$(curl -fsS --config <(printf 'user = "%s:%s"\n' "${REGISTRY_USER}" "${REGISTRY_TOKEN}") \
+    "https://${REGISTRY}/v2/token?service=container_registry&scope=*" | jq -r '.token // empty')" \
+    || die "could not obtain a registry bearer token from https://${REGISTRY}/v2/token"
+  [ -n "${REG_TOKEN}" ] || die "registry returned no bearer token"
+}
+# Prints the manifest digest the REGISTRY serves for <image>:<tag>; nothing if it is absent (404).
+# Any other answer (401, 5xx, no digest header) is fatal: "cannot tell" must never be read as
+# "not published", or the immutable-tag guard would be bypassed by an auth hiccup.
+registry_digest() {
+  registry_token
+  local hdrs code digest
+  hdrs="$(curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "${REG_TOKEN}") \
+    -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
+    -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
+    -H 'Accept: application/vnd.oci.image.index.v1+json' \
+    -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
+    "https://${REGISTRY}/v2/${NAMESPACE}/$1/manifests/$2" | tr -d '\r')" \
+    || die "registry unreachable while reading $1:$2"
+  code="$(awk 'NR==1{print $2}' <<<"${hdrs}")"
+  digest="$(awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' <<<"${hdrs}" | tail -1)"
+  case "${code}" in
+    200) [[ "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]] || die "registry answered 200 for $1:$2 with no usable digest"
+         echo "${digest}" ;;
+    404) ;;
+    *)   die "registry answered HTTP ${code} for $1:$2 — cannot tell whether it exists" ;;
+  esac
+}
+
+local_ref() { echo "${REGISTRY}/${NAMESPACE}/$1:ci-${HEAD_SHA:0:12}"; }
+
+build_one() { # <image> [agent base]
+  local img="$1" base="${2:-}" t0 tag
+  tag="ci-${HEAD_SHA:0:12}"
+  t0=$(date +%s)
+  log "BUILD + VERIFY ${img}"
+  case "${img}" in
+    toolbox)
+      ENGINE="${ENGINE}" IMAGE="$(local_ref toolbox)" BUILD_REVISION="${HEAD_SHA}" \
+        ./infra/devcontainer/build-image.sh ;;
+    agent-runner)
+      ENGINE="${ENGINE}" IMAGE_TAG="${tag}" AGENT_BASE_IMAGE="${base}" BUILD_REVISION="${HEAD_SHA}" \
+        ./infra/agent/build-image.sh ;;
+    conformance-runner)
+      ENGINE="${ENGINE}" IMAGE_TAG="${tag}" BUILD_REVISION="${HEAD_SHA}" \
+        ./conformance/incluster/build-image.sh ;;
+  esac
+  BUILT[$img]=1
+  SECS[$img]=$(( $(date +%s) - t0 ))
+  info "${img}: built and verified in ${SECS[$img]}s"
+}
+
+push_one() { # <image> -> sets DIGEST[image] to the registry-returned digest
+  local img="$1" repo src existing pushed got t0
+  repo="${REGISTRY}/${NAMESPACE}/${img}"
+  src="$(local_ref "${img}")"
+  t0=$(date +%s)
+  log "PUSH ${img} -> ${repo}:{${SHA_TAG},${MOVING_TAG}}"
+  existing="$(registry_digest "${img}" "${SHA_TAG}")"
+  if [ -n "${existing}" ]; then
+    # Immutable tag: never overwrite. A rebuild is not byte-reproducible, so re-pushing would
+    # silently change what `sha-<commit>` means after something may already have pinned it.
+    die "${repo}:${SHA_TAG} already exists (${existing}). Tags here are immutable; this run must not
+       replace it. (publish skips images that already exist at this commit BEFORE building them,
+       so reaching this line means the tag appeared mid-run.)"
+  fi
+  "${ENGINE}" tag "${src}" "${repo}:${SHA_TAG}"
+  "${ENGINE}" tag "${src}" "${repo}:${MOVING_TAG}"
+  pushed="$("${ENGINE}" push "${repo}:${SHA_TAG}" | tee /dev/stderr | grep -oE 'digest: sha256:[0-9a-f]{64}' | tail -1 | cut -d' ' -f2)"
+  "${ENGINE}" push "${repo}:${MOVING_TAG}" >/dev/null
+  # The AUTHORITATIVE digest is what the registry serves, read back — not what the client
+  # printed. They should agree; if they do not, nothing downstream may trust either.
+  got="$(registry_digest "${img}" "${SHA_TAG}")"
+  [ -n "${got}" ] || die "pushed ${repo}:${SHA_TAG} but the registry does not serve it"
+  [ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] \
+    || die "${img}: client reported ${pushed} but the registry serves ${got}"
+  [ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \
+    || die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})"
+  DIGEST[$img]="${got}"
+  SECS[$img]=$(( ${SECS[$img]:-0} + $(date +%s) - t0 ))
+  info "${img}: registry digest ${got}"
+}
+
+# --- main ---------------------------------------------------------------------------------
+log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}"
+command -v "${ENGINE}" >/dev/null || die "${ENGINE} is not on PATH in the job container"
+"${ENGINE}" info >/dev/null 2>&1 \
+  || die "the ${ENGINE} daemon is not reachable from the job container, so nothing can be built.
+       MEASURED ON THIS RUNNER (PR #81, 2026-10-29): act_runner 0.2.11 never gives a job a
+       daemon. Its DOCKER_HOST is tcp:// (the DinD sidecar), and internal/app/cmd/daemon.go then
+       forces container.docker_host to '-' (\"can't be mounted\"); separately, with
+       valid_volumes: [] gitea/act's sanitizeConfig drops EVERY bind, the socket included.
+       Fixing it means handing workflow code the privileged DinD daemon — an operator security
+       decision, set out in decisions/2026-10-29-image-pipeline.md. Not a fault in this script."
+"${ENGINE}" version 2>/dev/null | sed -n '1,12p' || true
+"${ENGINE}" system df || true
+
+if [ "${MODE}" = publish ]; then
+  : "${REGISTRY_TOKEN:?REGISTRY_TOKEN is required in publish mode}"
+  REGISTRY_USER="${REGISTRY_USER:-ci-image-publisher}"
+  printf '%s' "${REGISTRY_TOKEN}" | "${ENGINE}" login "${REGISTRY}" -u "${REGISTRY_USER}" --password-stdin >/dev/null \
+    || die "docker login ${REGISTRY} failed — check REGISTRY_PUSH_TOKEN (scope write:package, owner has package write on the org)"
+fi
+
+select_images
+
+# Publish idempotence: an image already published at THIS commit is not rebuilt. This is what
+# makes a re-run after a half-finished publish safe — and it is the only way a rerun can
+# finish, because the immutable sha tag must never be pushed twice.
+if [ "${MODE}" = publish ]; then
+  for i in "${ALL_IMAGES[@]}"; do
+    [ -n "${WANT[$i]:-}" ] || continue
+    d="$(registry_digest "${i}" "${SHA_TAG}")"
+    if [ -n "${d}" ]; then
+      info "${i}: already published at ${SHA_TAG} (${d}); not rebuilding"
+      unset "WANT[$i]"; DIGEST[$i]="${d}"; NOTE[$i]="already published at this commit"
+    fi
+  done
+fi
+
+# Decide the agent-runner base (see header).
+AGENT_BASE=""
+if [ -n "${WANT[agent-runner]:-}" ]; then
+  # shellcheck disable=SC1091
+  PINNED_BASE="$(. ./infra/agent/versions.env; echo "${AGENT_BASE_IMAGE}")"
+  if [ "${MODE}" = verify ] && [ -z "${WANT[toolbox]:-}" ]; then
+    WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)"
+  elif [ "${MODE}" = publish ] && [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then
+    case "${PINNED_BASE}" in
+      *@"${ZERO_DIGEST}") WANT[toolbox]=1; NOTE[toolbox]="never published; required as agent-runner's base" ;;
+      *@sha256:*)         AGENT_BASE="${PINNED_BASE}" ;;
+      *) die "infra/agent/versions.env AGENT_BASE_IMAGE='${PINNED_BASE}' is not digest-pinned; CI will not build on a floating base" ;;
+    esac
+  fi
+fi
+
+yn() { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; }
+for i in "${ALL_IMAGES[@]}"; do info "plan: ${i}: $(yn "${WANT[$i]:-}" BUILD skip)  (${NOTE[$i]:-inputs unchanged})"; done
+
+if [ -n "${WANT[toolbox]:-}" ]; then
+  build_one toolbox
+  if [ "${MODE}" = publish ]; then push_one toolbox; fi
+fi
+if [ -n "${WANT[agent-runner]:-}" ]; then
+  if [ -z "${AGENT_BASE}" ]; then
+    if [ "${MODE}" = publish ]; then
+      AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}"
+    else
+      AGENT_BASE="$(local_ref toolbox)"
+    fi
+  fi
+  info "agent-runner base: ${AGENT_BASE}"
+  build_one agent-runner "${AGENT_BASE}"
+  if [ "${MODE}" = publish ]; then push_one agent-runner; fi
+fi
+if [ -n "${WANT[conformance-runner]:-}" ]; then
+  build_one conformance-runner
+  if [ "${MODE}" = publish ]; then push_one conformance-runner; fi
+fi
+
+# --- summary ------------------------------------------------------------------------------
+{
+  echo "## Images (${MODE})"
+  echo
+  echo "commit \`${HEAD_SHA}\`"
+  echo
+  echo "| image | built + verified | seconds | published digest (registry-returned) | note |"
+  echo "|---|---|---|---|---|"
+  for i in "${ALL_IMAGES[@]}"; do
+    d="${DIGEST[$i]:-}"
+    echo "| \`${i}\` | $(yn "${BUILT[$i]:-}" yes no) | ${SECS[$i]:--} | $(yn "${d}" "\`${REGISTRY}/${NAMESPACE}/${i}@${d}\`" -) | ${NOTE[$i]:-inputs unchanged} |"
+  done
+  echo
+  if [ "${MODE}" = publish ]; then
+    echo "Tags pushed: \`${SHA_TAG}\` (immutable) and \`${MOVING_TAG}\` (moving)."
+    echo "Nothing here writes to git. Renovate opens the digest-bump PR on its next run."
+  else
+    echo "Verify mode: nothing was pushed and no registry credential was present."
+  fi
+} >>"${SUMMARY}"
+log "done"
@@ -73,7 +73,7 @@ spec:
        - name: FTLCONF_dns_rateLimit_count
          value: '0'
        - name: FTLCONF_misc_dnsmasq_lines
-          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
+          value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
        - name: FTLCONF_webserver_api_password
          valueFrom:
            secretKeyRef:
@@ -143,8 +143,39 @@ env:
  # resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
  # failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
  # every client that takes the ADVERTISED connection cannot reach the server at all.
+  #
+  # ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
+  # MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
+  #
+  #   dig home.keithjackson.dev @192.168.2.23 (pihole)  -> 192.168.2.20                TTL=0
+  #   dig home.keithjackson.dev @1.1.1.1      (public)  -> wan.home.keithjackson.dev   TTL=300
+  #   dig home.keithjackson.dev @192.168.2.1  (router)  -> wan....  TTL=153 -> 148 -> 143
+  #
+  # dnsmasq answers `address=` entries with **TTL 0** unless told otherwise, and that is the whole
+  # bug. The SOHO router forwards LAN queries to PiHole and CACHES the result, so:
+  #   * the CORRECT answer is UNCACHEABLE (TTL 0) -- the router must re-query PiHole on every
+  #     single lookup, which is maximum exposure to any momentary gap; while
+  #   * the WRONG answer is CACHEABLE FOR 5 MINUTES -- the instant the router falls back to its
+  #     upstream (standard forwarder behaviour when the forward target does not answer) it latches
+  #     the public WAN address and serves it to EVERY LAN CLIENT for the public record's 300s TTL.
+  # The decrementing TTL on the router's reply above is the proof it caches rather than forwarding
+  # live. So ONE sub-second PiHole gap -- a pod restart, an L2 announcement lease moving, an ARP
+  # flap, a kured node reboot -- becomes FIVE MINUTES of the whole house resolving to the WAN IP.
+  # That asymmetry, not the gap, is why this is visible as "flapping" rather than a blip.
+  #
+  # 60s is a deliberate floor, not a tuning knob: long enough to ride out a pod restart and a lease
+  # re-election (both seconds), short enough that a genuine VIP change still propagates inside a
+  # minute. These VIPs are pinned by name (`lbipam.cilium.io/ips`) and effectively static, so the
+  # propagation cost is theoretical while the flap it prevents is measured. If flaps persist across
+  # LONGER outages, raising this toward the public record's 300 would cover them -- at the cost of
+  # the same delay on a real change. It does NOT help when PiHole is down longer than the TTL: the
+  # cache expires, the router falls back, and the public answer latches again. That is a real
+  # outage, and the fix for it is PiHole availability, not DNS caching.
+  #
+  # Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
+  # configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
  - name: FTLCONF_misc_dnsmasq_lines
-    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
+    value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
  # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
  # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
  # carries the full rationale and the measurement that found it.
@@ -61,10 +61,13 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
  > digest-pinned to the conformance-runner image, `command` exactly
  > `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
  > `env`/`envFrom` were the same shape of hole and are closed in the same policy.
- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
-  image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
-  prints the exact `podman push --digestfile …` command for the operator instead. See
-  `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
+- `build-image.sh` — builds the above and **verifies the toolchain by running it** inside the
+  image as uid 65532 with `--read-only`. It **never pushes**, and neither does anyone from a
+  workstation: CI (`.gitea/workflows/images.yaml`) runs this same script, and on a push to
+  `main` publishes the verified image under `sha-<commit>` and `main`. Renovate then moves
+  `CONFORMANCE_RUNNER_IMAGE_DIGEST` in `conformance/versions.env` onto the registry-returned
+  digest in a pull request (that file is a sensitive path, so the bump needs an
+  administrator's `/approve-override`).
  `--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
  and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
  default verify stays offline so an air-gapped build is fully checked, not half-checked).
@@ -1,37 +1,41 @@
 #!/usr/bin/env bash
 ###############################################################################
 # conformance/incluster/build-image.sh — build the pinned `conformance-runner`
-# image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
+# image and prove it works. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
 #
-#   ./build-image.sh                 # build + verify + print the digest and push command
+#   ./build-image.sh                 # build + verify + print the image id
 #   ./build-image.sh --no-verify     # build only (skip the in-image tool proof)
 #   ./build-image.sh --verify-clone  # ALSO really clone this repo with conformance-clone.
 #                                    #   Opt-in because it needs the Gitea host reachable; the
 #                                    #   default verify is offline so an air-gapped build is
 #                                    #   still fully checked rather than half-checked.
 #   ENGINE=docker ./build-image.sh   # build with docker instead of podman
+#   IMAGE_TAG=foo ./build-image.sh   # override the local tag (CI uses this)
 #
-# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is
-# not an agent's to hold, and the pushed digest is what a fail-closed admission policy
-# then trusts. So this builds locally, proves the toolchain works INSIDE the image as the
-# non-root uid the policy demands, and prints the exact `podman push` line for a human.
+# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
+# Publishing is done by CI: .gitea/workflows/images.yaml calls this script, and on a push to
+# `main` it pushes what this script built AND verified to the Gitea registry, records the
+# registry-returned digest in the job summary, and Renovate then opens the pull request that
+# moves conformance/versions.env onto that digest. So this script's job is to build and to
+# prove the toolchain works INSIDE the image as the non-root uid the policy demands — the same
+# proof locally and in CI. It runs under ENGINE=docker in CI (Docker-in-Docker), so nothing
+# below may be podman-only.
 #
 # VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
 # single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
 # pins the toolbox gate runs with. Nothing in this file invents a version.
 #
-# THE DIGEST CAVEAT — read it before pasting a digest into anything.
-# The digest printed here is the LOCAL manifest digest. It is normally identical to the
-# registry digest (podman pushes the manifest it already has rather than recompressing),
-# but that is a property of the push, not a guarantee. So the push command below uses
-# `--digestfile`: the digest the registry actually returns is the authoritative one, and
-# that is the one that goes into conformance/versions.env.
+# NO DIGEST IS PRINTED. A manifest digest does not exist until a registry holds the manifest,
+# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at
+# push time (.gitea/workflows/scripts/images.sh) and Renovate carries it into
+# conformance/versions.env.
 ###############################################################################
 set -euo pipefail

 HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
 ENGINE="${ENGINE:-podman}"
+TAG_OVERRIDE="${IMAGE_TAG:-}"

 VERIFY=1
 VERIFY_CLONE=0
@@ -64,9 +68,9 @@ source "${TOOLCHAIN_ENV}"
 # gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
 # is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here.
 IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
-# The tag is build convenience only (a human-readable handle for the push). Admission
-# ignores it: only the digest is ever referenced from a Job.
-IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}"
+# The tag is a local handle only. CI re-tags what it publishes (git SHA + `main`), and
+# admission ignores tags entirely: only the digest is ever referenced from a Job.
+IMAGE_TAG="${TAG_OVERRIDE:-chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}}"
 IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}"

 log "Building ${IMAGE_REF}"
@@ -76,7 +80,11 @@ info "  CHAINSAW        ${CHAINSAW_VERSION}"
 info "  KUBECTL         ${KUBECTL_VERSION}"
 info "  FEDORA          ${FEDORA_VERSION}"

-"${ENGINE}" build \
+# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
+LABEL_ARGS=()
+[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
+
+"${ENGINE}" build "${LABEL_ARGS[@]}" \
  --file "${HERE}/Containerfile" \
  --tag "${IMAGE_REF}" \
  --build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
@@ -184,46 +192,12 @@ if [ "${VERIFY_CLONE}" -eq 1 ]; then
  log "conformance-clone verified against the real repository"
 fi

-# --- digest ------------------------------------------------------------------
-DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
+# --- result ------------------------------------------------------------------
 IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
 SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"

 log "Built ${IMAGE_REF}"
-info "local manifest digest  ${DIGEST}"
-info "image id               ${IMAGE_ID}"
-info "size                   $(( SIZE / 1024 / 1024 )) MiB"
-
-cat <<EOF
-
-================================================================================
-NOT PUSHED. Publishing is an operator action — this script holds no registry
-credential and will never acquire one.
-
-OPERATOR: run these two commands, then paste the digest from the digestfile into
-conformance/versions.env (CONFORMANCE_RUNNER_IMAGE) via a PR:
-
-    ${ENGINE} login gitea.home.keithjackson.dev
-
-    ${ENGINE} push --digestfile /tmp/conformance-runner.digest \\
-      ${IMAGE_REF}
-
-Then:
-
-    cat /tmp/conformance-runner.digest
-    # -> sha256:....  THIS is the authoritative digest, not the local one above.
-
-    # conformance/versions.env
-    CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)"
-
-WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml
-is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not
-${IMAGE_REPO}@sha256:<digest>. A tag reference is denied,
-and so is any other repository path.
-
-TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard
-\`${IMAGE_REPO}@sha256:*\` to be replaced with the
-LITERAL digest once the image exists, closing the "anyone who can push to this path" gap.
-That is a separate, human-reviewed PR against the security boundary.
-================================================================================
-EOF
+info "image id  ${IMAGE_ID}"
+info "size      $(( SIZE / 1024 / 1024 )) MiB"
+info "Not pushed. CI publishes this image from main (.gitea/workflows/images.yaml) and"
+info "Renovate moves conformance/versions.env onto the registry-returned digest."
@@ -29,7 +29,8 @@
 # THEN  (1) promtool is a PINNED member of the toolchain: PROMTOOL_VERSION is declared in BOTH
 #           infra/devcontainer/versions.env and ci/versions.env and the two AGREE (the repo's
 #           lockstep rule), the Containerfile takes it as a build ARG, installs it from the
-#           pinned prometheus release and records it for provenance, dev.sh passes it, and the
+#           pinned prometheus release and records it for provenance, build-image.sh (the one
+#           recipe `dev.sh build` and CI both use) passes it, and the
 #           CI runner installs it for the l0 stage;
 #       (2) the promtool ACTUALLY IN USE reports exactly that pinned version — a unit test is
 #           evidence only if the engine matches the one prod evaluates rules with;
@@ -246,10 +247,12 @@ check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text,
      "infra/devcontainer/Containerfile does not record promtool in /etc/dev-toolchain-versions — "
      "that provenance line is how an operator tells a stale image from a current one")

-DEVSH = ROOT / "infra/devcontainer/dev.sh"
-check(DEVSH.is_file() and "--build-arg PROMTOOL_VERSION=" in DEVSH.read_text(),
-      "infra/devcontainer/dev.sh does not pass --build-arg PROMTOOL_VERSION — the ARG would be "
-      "empty and the build would fetch a bogus URL")
+# The build recipe moved from dev.sh into build-image.sh on 2026-10-29, so that `dev.sh build`
+# and the CI image pipeline (.gitea/workflows/images.yaml) share ONE recipe. Check the recipe.
+BUILDSH = ROOT / "infra/devcontainer/build-image.sh"
+check(BUILDSH.is_file() and "--build-arg PROMTOOL_VERSION=" in BUILDSH.read_text(),
+      "infra/devcontainer/build-image.sh does not pass --build-arg PROMTOOL_VERSION — the ARG "
+      "would be empty and the build would fetch a bogus URL")

 INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh"
 if INSTALL.is_file():
@@ -40,32 +40,29 @@ PYTHON_MIN="3.9"
 #   gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*
 # Any other path, and any TAG reference, is DENIED at admission. Do not "tidy" these values.
 CONFORMANCE_RUNNER_IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
-# Build handle only. Admission never sees the tag; a Job must reference the digest.
+# Local build handle only. Admission never sees the tag; a Job must reference the digest.
+# CI publishes under `sha-<commit>` and `main` instead (.gitea/workflows/images.yaml).
 CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
 #
 # ############################################################################
-# # DIGEST DELIBERATELY EMPTY — THE IMAGE HAS NOT BEEN PUBLISHED YET.        #
+# # THE DIGEST IS WRITTEN BY RENOVATE, NOT BY HAND.                          #
 # ############################################################################
-# The image has been BUILT and its toolchain verified locally (non-root uid 65532,
-# read-only rootfs: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4, chainsaw 0.2.12,
-# and `./run.sh --layer l2 --no-cluster` completing with rc 0). It has NOT been pushed:
-# pushing needs a registry credential, and publishing the artifact a fail-closed security
-# boundary then trusts is an OPERATOR action, not an agent's.
+# CI builds and verifies the image (non-root uid 65532, read-only rootfs, the
+# conformance-clone refusals) and, on a push to main, publishes it and prints the digest the
+# REGISTRY returned (.gitea/workflows/images.yaml). Renovate follows the image's `main` tag
+# through the marker below and opens the pull request that moves this value. Nobody pushes
+# this image from a workstation, and nobody pastes a digest here.
 #
-# OPERATOR, to complete the pin:
-#   1. ./conformance/incluster/build-image.sh          # rebuild, verify, print the commands
-#   2. podman login gitea.home.keithjackson.dev
-#   3. podman push --digestfile /tmp/conformance-runner.digest \
-#        "${CONFORMANCE_RUNNER_IMAGE_REPO}:${CONFORMANCE_RUNNER_IMAGE_TAG}"
-#   4. Set CONFORMANCE_RUNNER_IMAGE_DIGEST below to `cat /tmp/conformance-runner.digest`
-#      — the digest the REGISTRY returned, not the local manifest digest the build printed.
-#   5. Land steps 4 in a PR (this file is GitOps input, not a live edit).
-#
-# UNTIL THEN: this is empty on purpose, NOT a placeholder to be guessed at. An empty value
-# makes any consumer that renders the Job fail visibly; a fake or stale digest would instead
-# produce an ImagePullBackOff that looks like a cluster fault. Do not invent one.
-CONFORMANCE_RUNNER_IMAGE_DIGEST=""
-CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_DIGEST:+${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}}"
+# ALL-ZERO UNTIL THE FIRST PUBLISH. Before 2026-10-29 this was EMPTY, so that a consumer
+# rendering the Job would fail visibly. It is now the same impossible all-zero digest
+# platform/pm-agent/ uses, because Renovate can only replace a digest that is there to match.
+# A consumer MUST treat the all-zero value as "not published" (no consumer exists yet: the
+# L2/L3 trigger is unwired, docs/plans/l2-l3-conformance-ci-wiring.md). Note also that
+# chainsaw-job.yaml's @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ takes the 64 hex digits WITHOUT the
+# `sha256:` prefix this value carries — the trigger must strip it.
+# renovate-digest: image=gitea.home.keithjackson.dev/home-network/conformance-runner tag=main
+CONFORMANCE_RUNNER_IMAGE_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000"
+CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}"
 #
 # AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for
 # its `@sha256:*` wildcard to be replaced with the LITERAL digest once the image exists —
@@ -185,6 +185,55 @@ Measured against the live cluster and nodes, read-only, on 2026-10-17.
 - **`renovate.json` parses** as valid JSON after my edits (`json.load`).
 - **kured is not installed** and the string `kured` appears nowhere in the repo — greenfield.

+### CORRECTION 2026-10-18 — the first real run FAILED, and the sentinel design was wrong
+
+`ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod` failed on all three
+nodes at the package task:
+
+```
+fatal: [k3s-prod-1]: FAILED! => "No package matching 'update-notifier-common' is available"
+```
+
+**`update-notifier-common` is an UBUNTU package and does not exist in Debian.** Measured on trixie:
+`apt-cache policy update-notifier-common` → `Candidate: (none)`. The original entry asserted this
+package "owns the hook that creates /var/run/reboot-required" — true on Ubuntu, false here, and the
+role could never have worked. This was the exact risk the first version of this entry flagged ("the
+`find` patterns are a guess at which package ships what") landing in a worse place than predicted:
+not a wrong glob, a nonexistent package.
+
+Three further measurements settled what to do instead:
+
+* `/usr/share/update-notifier/notify-reboot-required` is **absent**.
+* **No `linux-image-*` postinst references `notify-reboot-required`** — invoking it is an Ubuntu
+  patch, so merely placing a script at that path would never be called.
+* No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
+
+So Debian trixie has **no stock mechanism at all**, and the role now ships the writer itself:
+`/usr/local/sbin/k3s-notify-reboot-required` (compares the running kernel against the newest
+installed via `linux-version`, from `linux-base`, which every `linux-image` depends on — measured
+`linux-base 4.12.1` present) plus a `DPkg::Post-Invoke` hook at
+`/etc/apt/apt.conf.d/99-k3s-reboot-required`. `/run` is a tmpfs (measured), so the sentinel
+self-clears on boot and needs no cleanup task.
+
+The verify assert changed shape too: it was a `find` glob over `/etc/kernel/postinst.d` and
+`/etc/apt/apt.conf.d` for `*update-notifier*` / `*unattended-upgrades*`. That glob was **weaker
+than it looked** — it would have matched Debian's own `unattended-upgrades` apt conffile and
+reported "sentinel writer present" while no writer existed. It now stats the two exact paths this
+role owns and requires the script to be executable.
+
+**Newly verified (this correction):** the sentinel script was executed on prod-1 as the
+unprivileged `k3s` user under `sh -x`. It resolved `running=7.1.8+deb13-amd64`,
+`newest=7.1.8+deb13-amd64`, took the equal branch, exited 0 and wrote nothing — correct, since the
+running kernel *is* the newest. `sh -n` and `bash -n` clean.
+
+**Still NOT verified after this correction:** the *writing* half. Because running == newest on all
+three nodes, the `touch` path has never executed, so I have not seen the sentinel actually created,
+nor seen kured react to it. The honest test is the operator's next kernel upgrade. Also unverified:
+whether a `DPkg::Post-Invoke` hook fires in the `unattended-upgrades` code path specifically (it
+runs dpkg, so it should, but that is inference) — and whether `linux-version sort --reverse` orders
+a mixed backports/stable kernel set the way I assume when a 6.12 and a 7.1 are both installed. It
+produced the right answer with the current set; a future set could differ.
+
 ## What I did NOT verify

 The honest list. Most of this needs an actual upgrade cycle, which I did not and could not run.
@@ -0,0 +1,82 @@
+---
+date: 2026-10-18
+branch: fix/pihole-local-ttl
+title: "The split-horizon answer was uncacheable while the wrong answer cached for 5 minutes"
+---
+
+## What changed
+
+The operator reported DNS "flapping between the cluster ingress and the router (public IP)".
+PiHole was not at fault — both replicas answer correctly and consistently. The bug is an
+**asymmetry in cacheability** that we created.
+
+Measured on prod 2026-10-18:
+
+```
+dig home.keithjackson.dev @192.168.2.23 (pihole)  -> 192.168.2.20                TTL=0
+dig home.keithjackson.dev @1.1.1.1      (public)  -> wan.home.keithjackson.dev   TTL=300
+dig home.keithjackson.dev @192.168.2.1  (router)  -> wan....  TTL=153 -> 148 -> 143
+```
+
+dnsmasq answers `address=` entries with **TTL 0** unless told otherwise. The SOHO router forwards
+LAN queries to PiHole and **caches** the result — the decrementing TTL above is the proof it caches
+rather than forwarding live. So:
+
+* the **correct** answer is **uncacheable** (TTL 0): the router must re-query PiHole on *every*
+  lookup, which is maximum exposure to any momentary gap;
+* the **wrong** answer is **cacheable for 300s**: the instant the router falls back to its upstream
+  (standard forwarder behaviour when the forward target does not answer) it latches the public WAN
+  address and serves it to **every LAN client** for five minutes.
+
+So **one sub-second PiHole gap becomes five minutes of the whole house resolving to the WAN IP** —
+then it expires, the router re-queries PiHole, gets TTL 0 again, and is immediately re-exposed.
+That asymmetry, not the gap itself, is why this is visible as flapping rather than a blip.
+
+Fix: `local-ttl=60` prepended to `FTLCONF_misc_dnsmasq_lines` in `catalog/pihole.yaml`, so the
+private answer is cacheable and a short gap is invisible to clients. `apps/wave-0/pihole.yaml`
+regenerated with the canonical render command.
+
+## What I verified
+
+* **PiHole is NOT the fault.** Both replicas, queried directly by pod IP, answer `192.168.2.20`;
+  the VIP returned it 6/6; this workstation 8/8. DHCP hands out only `192.168.2.23`.
+* **The router caches rather than forwards live** — three successive queries returned TTL 153, 148,
+  143 (decrementing), which a pure forwarder could not do.
+* **The TTL asymmetry**, as quoted above: 0 from PiHole, 300 from public DNS.
+* **The router does answer DNS and returns the public chain** (`wan.home.keithjackson.dev` → the WAN
+  address), so a fallback always has a wrong answer available.
+* **`local-ttl` is the right directive**: dnsmasq applies it to replies made from configuration,
+  `/etc/hosts` and DHCP leases, which covers all five `address=` names with one directive. It is a
+  standalone option, not a sub-field of `address=`, so prepending it to the `;`-separated
+  `FTLCONF_misc_dnsmasq_lines` list is syntactically correct.
+* **The render is clean and scoped**: `catalog/render.py --all catalog -o apps --wave-layout
+  --no-image-automation` (the canonical command `ci/render-golden.sh` asserts against, and
+  deliberately **not** `--emit-waves`) changed exactly two files — `catalog/pihole.yaml` and
+  `apps/wave-0/pihole.yaml` — and the rendered diff is the single `local-ttl=60;` prefix.
+
+## What I did NOT verify
+
+* **UNAPPLIED.** Nothing has been reconciled, so I have not observed PiHole serving a non-zero TTL,
+  nor the router caching the private answer, nor the flap stopping. **Verify after merge with
+  `dig home.keithjackson.dev @192.168.2.23` (expect `TTL=60`, not 0) and then the same query at the
+  router (expect `192.168.2.20` with a decrementing TTL, not `wan.…`).**
+* **I did not confirm that PiHole/FTL accepts `local-ttl` via `FTLCONF_misc_dnsmasq_lines`.** The
+  directive is valid dnsmasq and the field is a passthrough, but `catalog/pihole.yaml`'s own header
+  records that FTL has previously written a bogus line into `dnsmasq.conf` and reported "0 invalid"
+  — so a malformed passthrough fails *silently*. If the TTL is still 0 after merge, that is the
+  cause, not this reasoning. Check the pod's `/etc/pihole/dnsmasq.conf`.
+* **60s is chosen, not derived.** Long enough for a pod restart and a lease re-election (both
+  seconds); it does **nothing** for an outage longer than the TTL, where the cache expires and the
+  public answer latches again. That is a real availability problem, not a caching one.
+* **This does not remove the trigger.** `192.168.2.23` is served by **two** LoadBalancer Services
+  (`pihole-dns-tcp`, `pihole-dns-udp`) with **two independently-elected L2 announcement leases**, so
+  two nodes can ARP for the same address at once — they sat on different nodes earlier the same day
+  (tcp on prod-2, udp on prod-1). The same pattern applies to crafty (`.22`) and unifi (`.21`).
+  Merging each pair into one mixed-protocol Service would remove the class; the operator has parked
+  that decision, and it is deliberately **not** in this change.
+* **Client-side bypasses are unexamined.** A device with its own resolver, or a browser with
+  DNS-over-HTTPS, bypasses PiHole entirely and always gets the public answer; this fix cannot help
+  there. Only this one workstation's resolver chain was inspected.
+* **No alert fires if the split-horizon answer regresses to the public address.** Nothing probes
+  `home.keithjackson.dev` from a LAN vantage point and compares it against `${GATEWAY_VIP}`, so the
+  next recurrence would again be found by a human noticing. That gap is not closed here.
@@ -0,0 +1,78 @@
+---
+date: 2026-10-18
+branch: fix/renovate-cron-window
+title: "The Renovate cron fired outside Renovate's own schedule, so it would never have opened a PR"
+---
+
+## What changed
+
+`.gitea/workflows/renovate.yaml` shipped (PR #73) with `cron: "15 3 * * *"`. **That schedule could
+never have produced a single PR.**
+
+Gitea Actions cron is **UTC**. `renovate.json` carries `"schedule": ["after 3am and before 6am"]`
+with `"timezone": "America/New_York"`, and that inner schedule is **authoritative** — Renovate
+re-checks it and declines to open PRs outside it. So:
+
+```
+cron 03:15 UTC  ->  23:15 EDT, the PREVIOUS EVENING in New York
+window          ->  03:00-06:00 local
+inside?         ->  NO (3h45m before the window opens)
+```
+
+The original comment claimed the cron was "slightly ahead of the repo window ... so a run is always
+in flight when the window opens". It is 3h45m *behind*, on the previous day. Every nightly run would
+have started, logged "not within schedule", opened **zero** PRs and exited **successfully** — a
+green tick forever, indistinguishable from "nothing needed updating". That is the same
+"failure-mode-is-a-green-tick" shape this repo keeps hitting, and it would have hidden the ~30
+pending updates PR #73 was written to surface.
+
+Changed to `cron: "15 8 * * *"`, chosen to survive DST rather than to be tidy. The window's UTC
+position moves:
+
+| | local window | in UTC |
+|---|---|---|
+| EDT (UTC-4, Mar–Nov) | 03:00–06:00 | 07:00–10:00 |
+| EST (UTC-5, Nov–Mar) | 03:00–06:00 | 08:00–11:00 |
+
+`08:15 UTC` is inside **both** (04:15 EDT / 03:15 EST), so no edit is needed at either switchover.
+A cron cannot follow DST, so this is the only way to get it right once.
+
+## What I verified
+
+* **The defect, by arithmetic rather than assertion:** `datetime(2026,10,18,03,15, UTC)` converted to
+  `America/New_York` via `zoneinfo` is `2026-10-17 23:15 EDT`; `3 <= hour < 6` is **False**.
+* **The fix, at both DST extremes:** `08:15 UTC` → `04:15 EDT` in July and `03:15 EST` in December,
+  both inside `03:00–06:00`. Computed, not reasoned.
+* **The inner schedule really is authoritative and really is set:** `renovate.json` on `main` carries
+  `timezone = America/New_York` and `schedule = ['after 3am and before 6am']`.
+* **`RENOVATE_TOKEN` now exists** as a repository Actions secret (listed by name via the Gitea API
+  alongside `SETTINGS_API_TOKEN`), so the workflow's fail-hard preflight will pass. It did not exist
+  when PR #73 was authored.
+* The edited workflow parses as YAML.
+
+## What I did NOT verify
+
+* **Renovate has still never executed.** This fixes *when* it runs, not *whether it works*. No run
+  has happened, so nothing here proves it extracts dependencies, authenticates to Gitea, or opens a
+  usable PR.
+* **`enabledManagers` may silently disable the custom managers — UNRESOLVED and the likelier
+  remaining bug.** `renovate.json` lists `enabledManagers: [..., 'regex', ...]` and defines
+  `customManagers` (correct modern key; `regexManagers` is absent, which is right). But modern
+  Renovate names the custom manager **`custom.regex`** in `enabledManagers`, not bare `regex`. If
+  the bare name is invalid or ignored, every inline `# renovate:` marker in the tree does nothing —
+  including the `external-snapshotter` `newTag` pin PR #73 flagged as probably inert. **I did not
+  change it**, because guessing at a config key that could fail validation and take the whole run
+  down is worse than leaving a question that one dry run answers definitively. The `dry_run: true`
+  dispatch log settles it: look for a per-manager dependency-extraction count, and for any config
+  validation warning naming `regex`.
+* **Whether a dry run inside vs outside the window differs usefully.** Renovate honours its schedule
+  even in dry-run, so a manual dispatch outside 03:00–06:00 local will also log "not within
+  schedule". That makes the first dispatch's output less informative than it looks; to see what
+  Renovate *would* do, either dispatch inside the local window or override the schedule for that
+  run. Not tested either way.
+* **No alert fires if the schedule stops firing.** PR #73 recorded this gap and it is still open: a
+  Renovate workflow that silently stops running (disabled Actions, expired token, deleted secret)
+  produces no signal at all. Closing it needs a check on last-successful-run age, which nothing
+  does.
+* No conformance test asserts the cron falls inside `renovate.json`'s window, so this exact bug can
+  return. A static check comparing the two is feasible and is not written.
@@ -0,0 +1,78 @@
+---
+date: 2026-10-18
+branch: fix/renovate-invalid-config
+title: "renovate.json was invalid, so the first manual run errored out"
+---
+
+## What changed
+
+The operator ran the Renovate workflow manually and it **failed**. The cause is not the workflow:
+**`renovate.json` was never a valid Renovate config.**
+
+It carried five pseudo-comment keys — `_comment`, `_kubernetes_comment`, `_unmanageable_comment`,
+`_ignorePaths_comment`, `_docker_comment` — because JSON has no comments. Renovate **rejects unknown
+top-level options**, so the entire config failed validation. Proven with the official validator
+rather than inferred:
+
+```
+$ podman run --rm -v "$PWD/renovate.json:/tmp/renovate.json:ro,Z" \
+    ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json
+ERROR: Found errors in configuration
+  Invalid configuration option: _comment
+  Invalid configuration option: _docker_comment
+  Invalid configuration option: _ignorePaths_comment
+  Invalid configuration option: _kubernetes_comment
+  Invalid configuration option: _unmanageable_comment
+```
+
+This is the second independent reason Renovate had never worked, on top of PR #77's cron firing
+outside `renovate.json`'s own schedule window. Either alone was sufficient to produce "nothing ever
+happens"; #73 shipped both.
+
+**Converted `renovate.json` → `renovate.json5`.** JSON5 permits real `//` comments, and Renovate
+discovers `renovate.json5` natively — nothing in `.gitea/` or `ci/` pins the old filename (only
+prose inside comments), and no `RENOVATE_CONFIG_FILE` is set. Every word of the original rationale
+is preserved as comments positioned next to what it explains, which a flattened `description`
+string would have lost.
+
+A header on the new file records the failure, the validator command, and the instruction to run it
+before committing — this class of bug is invisible to the repo's own gate.
+
+## What I verified
+
+* **The exact failure**, from the official validator, quoted above. Not read off a log, not inferred
+  from the workflow.
+* **The fix validates**: the same validator against `renovate.json5` returns
+  `INFO: Config validated successfully against 1 file(s)`.
+* **Nothing was dropped in the conversion**, checked programmatically against `git show
+  HEAD:renovate.json`: 18 non-comment keys before, 18 after, **no missing keys, no extra keys, and
+  no key whose value changed**. `docker: {enabled: true}` confirmed still present — the validator
+  prints a normalisation diff that mentions it, which is its own migration suggestion, not a
+  removal.
+* **74 comment lines survive** in the new file, so the rationale is intact.
+* **`RENOVATE_TOKEN` exists** as a repository Actions secret (listed by name via the Gitea API), so
+  the workflow's fail-hard preflight is satisfied.
+* **A suspicion was DISPROVED:** `enabledManagers` containing bare `regex` was thought to need to be
+  `custom.regex`, which would have silently disabled every inline `# renovate:` marker. The
+  validator accepts `regex`, so that concern was unfounded and nothing here changes it.
+* Nothing functionally references `renovate.json` by name — `grep -rn "renovate\.json" .gitea/ ci/
+  conformance/` returns only comment prose.
+
+## What I did NOT verify
+
+* **Renovate has still never completed a run.** This removes one of two blockers; PR #77 removes the
+  other. Neither is merged, so **no successful Renovate run has been observed** and I cannot say the
+  bot works — only that these two specific defects are fixed.
+* **I did not re-run the workflow after the fix.** The honest test is a `workflow_dispatch` with
+  `dry_run: true` once both this and #77 are merged. Expect ~30 dependencies behind, rate-limited to
+  `prHourlyLimit: 2` / `prConcurrentLimit: 5`.
+* **Whether the managers actually extract anything.** A valid config that matches no files is still
+  a no-op. The dry-run log's per-manager dependency counts are the only thing that proves the
+  `managerFilePatterns` are right, and I have not seen them.
+* **Whether `renovate.json5` is honoured by the pinned Renovate version in the workflow.** JSON5
+  discovery is long-standing and the validator accepted the file, but the workflow's image tag was
+  not cross-checked against the version that added it.
+* **Nothing in the gate catches an invalid Renovate config.** This bug would have recurred silently.
+  A one-line L0 check shelling out to `renovate-config-validator` would close it; it is not written
+  here, and `conformance/**` is a sensitive path.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
@@ -0,0 +1,283 @@
+---
+date: 2026-10-29
+branch: feat/image-pipeline
+title: "Images are built, verified and published by CI; Renovate carries the digests into git"
+---
+
+## What changed
+
+Operator rule (2026-10-29): container image creation, hosting and lifecycle belong to CI/CD and
+the codebase, never to a one-off workstation build/push. Before this branch the Home-Network org's
+package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and
+both `build-image.sh` scripts said "publishing is an operator action".
+
+- **`.gitea/workflows/images.yaml`** + **`.gitea/workflows/scripts/images.sh`** (the logic).
+  - Pull request: build and verify only. No push, and the step references no secret at all.
+  - Push to `main`, or `workflow_dispatch` on `main`: build, verify, then push to
+    `gitea.home.keithjackson.dev/home-network/{toolbox,agent-runner,conformance-runner}`.
+    Tags are `sha-<12 hex>` (immutable: the driver refuses to overwrite one) and `main` (moving).
+    The digest is read back from the registry with a HEAD on the manifest, compared with what the
+    client printed, and written to the job summary.
+  - Only images whose inputs changed are built. A re-run on the same commit skips images that are
+    already published.
+  - agent-runner builds FROM the toolbox built in the same run (on main, FROM its just-pushed
+    digest). Otherwise:
+    - in verify mode the toolbox is built anyway as its base, because a PR holds no credential;
+    - in publish mode it uses `AGENT_BASE_IMAGE` from `infra/agent/versions.env`;
+    - if that is still the zero placeholder, the toolbox is built and published first. This is
+      how the first run on main bootstraps.
+  - An always-run step prunes the DinD image store.
+- **One build recipe per image, reused by CI.**
+  - New `infra/devcontainer/build-image.sh`. `dev.sh build` now delegates to it. It has an
+    in-image verification step that the toolbox never had: every tool is on PATH, and
+    promtool, kubectl, chainsaw, kyverno and sops report their pins.
+  - `infra/agent/build-image.sh` and `conformance/incluster/build-image.sh` were made
+    engine-neutral, because CI runs `ENGINE=docker` against DinD:
+    - `podman image exists` became `image inspect`, with a pull fallback;
+    - `{{.Digest}}` and `--digestfile` are gone. No local digest is printed, since only the
+      registry's digest is meaningful;
+    - `probe-validation.sh` is fed over **stdin** instead of a bind mount. Under DinD a bind
+      source is resolved on the daemon's filesystem, which would silently mount an empty
+      directory;
+    - `probe-validation.sh` itself gained `${BASH_SOURCE[0]:-$0}` so it survives `bash -s`
+      under `set -u`.
+  - `AGENT_BASE_IMAGE` and `IMAGE_TAG` can be overridden from the environment, and
+    `BUILD_REVISION` adds an `org.opencontainers.image.revision` label.
+  - The "publishing is an operator action" text was replaced throughout.
+- **Digest references.** Each now carries a `# renovate-digest: image=<repo> tag=main` marker
+  on the line above it:
+  - `platform/pm-agent/deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`), which share
+    one dependency, so they bump in one PR;
+  - `infra/agent/versions.env` `AGENT_BASE_IMAGE`, which was `home-k3s-dev:local` and is now
+    the published toolbox by digest (zero placeholder until the first publish);
+  - `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to the
+    zero placeholder because Renovate can only replace a digest it can match.
+
+  The admission policies are **untouched**; the `<repo>@sha256:<digest>` form is preserved.
+- **`renovate.json5`**:
+  - a new `custom.regex` manager for the markers;
+  - `enabledManagers` `regex` → `custom.regex`;
+  - a packageRule that disables the `kubernetes` manager for these three repos. Otherwise it
+    would also see the digest-only lines and look them up as `latest`;
+  - an `own-image` label;
+  - `ignorePaths` narrowed from `conformance/**` to its subdirectories. `ignorePaths` applies
+    to custom managers too, so the blanket glob hid `conformance/versions.env`.
+- **`renovate.yaml`**: `renovate-version: 44.167.0` pinned, plus a `RENOVATE_HOST_RULES` docker
+  rule for the Gitea registry, using the same `RENOVATE_TOKEN`, which now also needs
+  `read:package`. See below for why the version pin is a real fix and not tidiness.
+- **`infra/gitea/registry-retention.sh`**: dry run by default, `--apply` to delete. This is an
+  operator action. Per image it keeps:
+  - `main` and `latest`;
+  - the 10 newest `sha-` tags;
+  - every tag whose digest is pinned anywhere in `origin/main`;
+  - every digest-named version.
+- **`infra/gitea/conformance-trigger-kubeconfig.sh`**: the "runner image is digest-pinned"
+  precondition grepped the literal line `CONFORMANCE_RUNNER_IMAGE="${…_REPO}@${…_DIGEST}"`
+  for 64 hex digits, so it **could never pass**. It now sources the file, evaluates the value,
+  and rejects the zero placeholder.
+- **`conformance/tests/l0-static/alert-rules-unit-tested/check.sh`**: its "the toolbox build
+  passes `--build-arg PROMTOOL_VERSION`" assertion now reads `infra/devcontainer/build-image.sh`
+  instead of `dev.sh`. The gate caught this on the first run, because the recipe had moved.
+- **Docs**: `platform/pm-agent/README.md`, `runbooks/pm-agent.md` (two rows),
+  `docs/plans/pm-agent-issue-to-pr.md` (follow-up 1), `conformance/incluster/README.md`,
+  `infra/devcontainer/README.md`, and a new "How images are built" section in
+  `.gitea/workflows/README.md`.
+
+### BLOCKER — the runner cannot build images, and fixing that is a security decision
+
+**Measured on this PR's own CI run (images run 325): the job failed in 6 s** with
+`failed to connect to the docker API at unix:///var/run/docker.sock ... no such file`. The job
+container is given **no Docker daemon at all**, for two independent reasons, both read from the
+exact versions deployed:
+
+- act_runner **v0.2.11** `internal/app/cmd/daemon.go`: when `DOCKER_HOST` has a scheme other than
+  `unix`/`npipe` — here `tcp://localhost:2376`, the DinD sidecar — it sets
+  `cfg.Container.DockerHost = "-"`, which means "never mount into job containers".
+- gitea/act **v0.261.2** (act_runner's `go.mod` replace) `pkg/container/docker_run.go`
+  `sanitizeConfig`: with `valid_volumes` empty (the runner has no config file, so the default
+  `[]`), `hostConfig.Binds = []string{}`. **Every** bind is dropped, the daemon socket included,
+  and so is any `volumes:` a workflow asks for.
+
+I assumed the opposite while designing (from act's `getDockerDaemonSocketMountPath`, which never
+gets the chance to run). That was wrong, and the PR run is what showed it.
+
+**Why I did not fix it in this PR.** Any fix gives workflow code the DinD daemon. That daemon is
+`privileged: true`, so `docker run --privileged -v /:/host` from a job is **root on the node**.
+Today a malicious workflow — for example one pushed by a prompt-injected pm-agent worker, which
+holds a repo write token — runs in an unprivileged container and holds no cluster credential.
+After the fix it does not. That is a change to the security boundary, and it is the operator's
+call. It also cannot be tested from a PR anyway: the runner only changes after merge + Flux.
+
+**Options, for the operator:**
+
+1. **Socket to every job (smallest change):** in `apps/wave-1/gitea-actions.yaml`,
+   - mount a shared `emptyDir` at `/var/run` in both the `dind` sidecar and the `runner`, so
+     dockerd's own `unix:///var/run/docker.sock` is visible at the same path in both;
+   - set the runner's `DOCKER_HOST=unix:///var/run/docker.sock`, and drop `DOCKER_TLS_VERIFY`
+     and `DOCKER_CERT_PATH`, because TLS over a unix socket breaks the client;
+   - add an act_runner config file (`CONFIG_FILE`) with `container.valid_volumes:
+     ["/var/run/docker.sock"]` and `runner.file: /data/.runner`, so the existing registration
+     is kept.
+
+   Consequence: any workflow on any branch can reach node root.
+2. **Option 1 on a SECOND runner** with its own label (e.g. `image-builder`) and its own DinD,
+   with `images.yaml` `runs-on` that label. The gate runner stays socket-less, and image builds
+   stop competing with the gate for the single slot and its 7 GB node disk. **It is not a
+   security boundary**: Gitea cannot restrict which workflow may target a label, so any branch
+   could still `runs-on: image-builder`.
+3. **A builder that is not a privileged daemon**, e.g. a rootless BuildKit sidecar exposed to
+   jobs. That bounds a job to BuildKit's rootless sandbox instead of node root. It needs the
+   verification steps rewritten, because they *run* the image (as uid 65532, `--read-only`),
+   which BuildKit alone does not do. This is a larger piece of work.
+
+Nothing in this PR has to change for options 1 or 2 except `runs-on` for option 2. The driver
+already produced verified images with a stock docker 27 CLI (see *What I verified*).
+
+### Decisions and pushback
+
+- **The push credential is a PAT secret, `REGISTRY_PUSH_TOKEN`; the Actions token cannot push.**
+  Checked against Gitea **v1.24.3** source:
+  - `services/auth/basic.go` `Basic.Verify` turns a task token into the pseudo-user
+    `gitea-actions` (ID −2) and sets no token scope;
+  - `routers/api/packages/container/container.go` `Authenticate` (the `/v2/token` exchange every
+    docker client performs) requires `read:package`/`write:package`/`all` scope, so it answers
+    401. The Actions token cannot even *pull*;
+  - `services/context/package.go` `determineAccessMode` has `// TODO: ActionUser permission check`
+    and no branch for that user. Even past login it would get at most read on a public org;
+  - `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3.
+
+  The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight
+  pattern.
+- **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate:
+  - `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` →
+    `custom.regex`;
+  - that migration runs on every repository run (`lib/workers/repository/init/merge.ts` →
+    `migrateAndValidate`), not only in the validator;
+  - `lib/modules/manager/index.ts` `getEnabledManagersList` accepts either spelling;
+  - `custom.regex` exists since 37.51.1 (renovatebot/renovate#24079).
+- **The open question hid a bigger bug: the Renovate config on main is invalid for the Renovate
+  the workflow actually runs.**
+  - `renovatebot/github-action@v41.0.9` defaults `renovate-version` to `39`
+    (`action.yml`, `src/docker.ts`), which is 39.264.0.
+  - The validator used in #79 was `ghcr.io/renovatebot/renovate` with no tag, i.e. `latest`,
+    44.167.0.
+  - **Measured:** `renovate:39.264.0 renovate-config-validator` on main's `renovate.json5` gives
+    `ERROR: Found errors in configuration` with five errors, all `managerFilePatterns`.
+    `fileMatch` was renamed in 40.2.0.
+  - So once #80 fixes the filename, every run would still die at `CONFIG_VALIDATION`.
+  - Fixed by pinning `renovate-version: 44.167.0`. This branch's config validates clean on it.
+    The only `--strict` complaint is a pending migration of the pre-existing top-level
+    `"docker": {"enabled": true}` block, which I left alone.
+- **Unworkable as briefed: tracking the worker Job template's digest.**
+  - `infra/agent/worker-job-template.yaml` is `COPY`ed into agent-runner. A Renovate bump there
+    changes agent-runner's inputs, CI publishes a new digest, Renovate bumps it again, and so on
+    forever.
+  - `pm-dispatch` overwrites that field from `PM_WORKER_IMAGE` at runtime
+    (`pm-dispatch.py:649`), so nothing is lost by leaving it all-zero. It is commented as
+    deliberately untracked.
+  - For the same loop reason, `conformance/versions.env` is **not** an input of the
+    conformance-runner build, and a pipeline-only change does not republish on main.
+- **Changed from the brief: retention is a script, not a Gitea cleanup rule.**
+  - Gitea 1.24.3 has **no API** for cleanup rules (none in `routers/api/v1/api.go`; web
+    session+CSRF forms only, in `routers/web/web.go`). So a rule cannot be created, or
+    drift-asserted, as code. **`ci/gitea-settings.sh` was therefore not extended.** A cheaper
+    proxy (count versions per image) would need `read:package` on `SETTINGS_API_TOKEN`, which
+    it lacks.
+  - A built-in rule is also **unsafe here**. Gitea stores a manifest as a file of the TAG
+    version it was pushed under (`routers/api/packages/container/manifest.go`
+    `createPackageAndVersion`), and resolves a by-digest pull from those files
+    (`models/packages/container/search.go`). Deleting the tag a deployed digest was pushed under
+    breaks the `@sha256:` pull, and the rule's "newest N" knows nothing about what git pins. The
+    script keeps every digest git pins.
+- **Auto-merge.** Digest bumps of these images auto-merge on a green gate (the existing
+  `digest` rule). The exception is the conformance-runner bump: `conformance/versions.env` is a
+  sensitive path, so it waits for an administrator's `/approve-override` every time.
+
+## What I verified
+
+- **Local gate, pinned toolbox, from this worktree, committed HEAD.** Command:
+  `ENGINE=<shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`.
+  The shim adds the `.git` mount. Result: see the PR body for the rc and stage list. They are
+  recorded there rather than here because this entry is part of the tree being gated.
+- **Local podman builds through the CI driver**, run as
+  `MODE=verify ENGINE=podman BASE_SHA=$(git merge-base origin/main HEAD) .gitea/workflows/scripts/images.sh`.
+  It returned rc 0 and built and verified all three images:
+  - toolbox: every tool on PATH, and the five pins matched;
+  - agent-runner: `PM_SELFTEST=1 pm-dispatch` OK, and the stdin-fed `probe-validation.sh`
+    gave "20 refusal(s), 0 failure(s)";
+  - conformance-runner: all seven `conformance-clone` refusals.
+- **The docker-CLI code path**, i.e. what CI runs. I ran the same driver inside
+  `docker:27-cli` (`ENGINE=docker`, `DOCKER_BUILDKIT=0`) against podman's Docker-compatible API
+  socket. Exit code 0; all three images built and verified (selftest OK; probe OK, 20 refusals).
+  This covers the docker-specific calls (`image inspect` formats, `run -i` with stdin,
+  `--tmpfs`, `--read-only`, `--user`). It does **not** cover BuildKit or a real DinD daemon.
+- **This PR's CI, run 325 (`images`): FAILED in 6 s**, because there is no Docker daemon in the
+  job container (see BLOCKER). Gate run 324 failed only on `approval-policy` (expected,
+  sensitive paths). Every other stage passed there, including `gitea-settings`.
+- **Other driver paths, run locally:**
+  - agent-runner alone builds the toolbox as its base;
+  - an empty diff builds nothing;
+  - publish mode without a token dies before doing anything;
+  - `infra/agent/build-image.sh` with the zero placeholder dies with build-locally
+    instructions.
+- **Renovate, with the pinned 44.167.0 image, locally:**
+  - `renovate-config-validator` exits 0;
+  - a `--platform=local` `extract` dry run finds all four marker references (custom regex manager,
+    `currentValue: main`, `currentDigest: sha256:000…`);
+  - a `lookup` dry run shows the `kubernetes`-manager duplicates as `skipReason: disabled` and the
+    marker deps taking the "unsupported/unversioned value main → getDigest(main)" path. That
+    lookup currently 404s because nothing is published yet.
+- **Renovate 39.264.0 rejects main's current `renovate.json5`.** Validator output reproduced
+  above.
+- **`registry-retention.sh` dry run against the live Gitea (read-only):** zero versions per image,
+  nothing to delete. The token exchange and the package listing both answered.
+- **Live, read-only measurements** (kubectl and the Gitea API):
+  - the runner pod is on k3s-prod-1, which had **7 GB** nodefs available;
+  - DinD holds only the 1.54 GB act image; buildx is v0.20.1; the daemon is 27.5.1 on overlay2
+    with cgroup v2;
+  - Gitea `/data` is 9.7G, 1% used;
+  - the Gateway's Envoy (Cilium) has `streamIdleTimeout: 300s`, `maxStreamDuration: 0s`, and
+    **no route `timeout`** on the gitea route;
+  - Gitea's `LIMIT_SIZE_CONTAINER` is unset (default −1 = unlimited, `modules/setting/packages.go`).
+
+## What I did NOT verify
+
+- **Nothing has been pushed to the registry, by me or by CI.** The publish path has never run:
+  - the registry HEAD/digest read-back;
+  - the immutable-tag guard;
+  - "already published at this commit" idempotence;
+  - agent-runner `FROM <toolbox>@sha256:`;
+  - `docker login` with a PAT.
+
+  The first merge to main is the first test, and it needs `REGISTRY_PUSH_TOKEN` to exist first.
+- **Large-layer push through the Gateway is UNVERIFIED.** Envoy has no request-body cap and no
+  stream duration cap here. But with no route `timeout` set, Envoy's **default 15 s upstream
+  timeout** applies from the end of the request to the end of the response. That covers Gitea
+  finalising a blob: copying the upload into package storage on Longhorn. A ~250 MB compressed
+  layer should finish well inside 15 s, but I could not measure it without pushing. If the first
+  publish shows 504s on blob PUTs, set `timeouts.request` on the gitea HTTPRoute.
+- **DinD fit is estimated, not measured** (no build has run there, see the BLOCKER):
+  - memory: 2Gi is shared by the job container and every build container;
+  - disk: the toolbox, agent-runner, conformance-runner, Fedora base and build cache should
+    total roughly 3 GB on top of the 1.5 GB act image, against 7 GB free on the node, with
+    `dind-storage` having no sizeLimit.
+- **No image has been built in this runner's DinD**, because it cannot be done until the BLOCKER
+  is resolved. Two things are therefore unmeasured until then:
+  - DinD memory (2Gi) and disk fit;
+  - BuildKit (`buildx` v0.20.1 is on the daemon) as opposed to the legacy builder I tested
+    with.
+- **Renovate has still never completed a run here.** Three things block it:
+  - #80, the `configurationFile` name;
+  - `RENOVATE_TOKEN` gaining `read:package`;
+  - this branch's version pin.
+
+  `renovatebot/github-action@v41.0.9` running a 44.x image is untested. The action only passes
+  the tag to `docker run`.
+- **Pods cannot pull these images yet regardless.** The kubelet pull credential is PR #61
+  (`pm-agent-registry` dockerconfigjson, `read:package`), still open. Until it merges, a correct
+  digest still ends in ImagePullBackOff (401), as before.
+- **`registry-retention.sh --apply` has never run**, and its keep logic has only run against an
+  empty registry.
+- **`git diff` change detection on Gitea `push` events relies on `github.event.before`.** A
+  force-push or a first push falls back to building everything, which is intended but not
+  observed.
@@ -244,10 +244,12 @@ about runtime behaviour in this document is a design claim, not a measurement.
   (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with
   the credential decision now being local Gitea accounts with PATs, that means a
   `dockerconfigjson` for an account whose token carries `read:package`. Only then does the
-   digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both
-   placeholder digests. Expect the build itself to need iteration too: the nodejs module
-   stream and the npm install have never run against this base. The same two blockers sit in
-   front of `conformance-runner`, which shares the registry path and has never been pulled.
+   digest matter. **Superseded 2026-10-29:** the build-and-push is no longer an operator step.
+   `.gitea/workflows/images.yaml` builds and verifies the image with
+   `infra/agent/build-image.sh`, publishes it on a push to `main`, and Renovate opens the one
+   PR that replaces both placeholder digests (`decisions/2026-10-29-image-pipeline.md`). The
+   same two blockers sit in front of `conformance-runner`, which shares the registry path and
+   has never been pulled.
 2. **Create the credentials** per `pm-agent-credentials.sops.yaml.example`, and **verify the
   bot's permission is `write`, not `admin`** — that measurement is the control that keeps
   the agent out of its own approval gate.
@@ -1,26 +1,36 @@
 #!/usr/bin/env bash
 ###############################################################################
-# infra/agent/build-image.sh — build the pinned `agent-runner` image and print its digest.
+# infra/agent/build-image.sh — build the pinned `agent-runner` image and prove it works.
 #
-#   ./build-image.sh                 # build + verify + print the digest and push command
+#   ./build-image.sh                 # build + verify + print the image id
 #   ./build-image.sh --no-verify     # build only (skip the in-image proof)
 #   ENGINE=docker ./build-image.sh   # build with docker instead of podman
+#   AGENT_BASE_IMAGE=home-k3s-dev:local ./build-image.sh
+#                                    # build on a LOCAL toolbox instead of the pinned one
+#                                    #   (`infra/devcontainer/dev.sh build` makes that tag)
+#   IMAGE_TAG=foo ./build-image.sh   # override the local tag (CI uses this)
 #
-# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is not
-# an agent's to hold, and the pushed digest is what a fail-closed admission policy then
-# trusts. So this builds locally, proves the toolchain and both entrypoints work INSIDE the
-# image as the non-root uid with the read-only root filesystem the policy forces, and prints
-# the exact push line for a human.
+# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
+# Publishing is done by CI: .gitea/workflows/images.yaml calls this script, and on a push to
+# `main` it pushes what this script built AND verified to the Gitea registry, records the
+# registry-returned digest in the job summary, and Renovate then opens the pull request that
+# moves platform/pm-agent/ onto that digest. So this script's job is to build, and to prove
+# the toolchain and both entrypoints work INSIDE the image as the non-root uid with the
+# read-only root filesystem the admission policy forces — the same proof locally and in CI.
 #
-# VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the four pins this image
-# adds (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds
-# the platform toolchain the base already carries. Nothing in this file invents a version.
+# VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the pins this image adds
+# (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds the
+# platform toolchain the base already carries. Nothing in this file invents a version.
 #
-# THE DIGEST CAVEAT — read it before pasting a digest into anything. The digest printed here
-# is the LOCAL manifest digest. It is normally identical to the registry digest, but that is
-# a property of the push rather than a guarantee, so the push command below uses
-# `--digestfile`: the digest the REGISTRY returns is the authoritative one, and that is the
-# one that goes into platform/pm-agent/configmap.yaml.
+# THE BASE. AGENT_BASE_IMAGE in versions.env is the PUBLISHED toolbox, pinned by digest and
+# kept current by Renovate. An AGENT_BASE_IMAGE in the environment overrides it — CI passes
+# the toolbox it built in the same run, and a developer can pass `home-k3s-dev:local`.
+#
+# ENGINE PORTABILITY. CI runs this with ENGINE=docker against a Docker-in-Docker daemon, so
+# nothing below may be podman-only (`image exists`, `--digestfile`, `{{.Digest}}`), and
+# nothing may BIND-MOUNT a file from this checkout into a container: under DinD the daemon
+# resolves bind sources on ITS filesystem, where this checkout does not exist, and silently
+# mounts an empty directory instead. Files go in over stdin.
 ###############################################################################
 set -euo pipefail

@@ -46,8 +56,11 @@ command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGI
 # --- pins: sourced, never invented -------------------------------------------
 AGENT_ENV="${HERE}/versions.env"
 [ -f "${AGENT_ENV}" ] || die "missing ${AGENT_ENV} — the pins live there, not here"
+BASE_OVERRIDE="${AGENT_BASE_IMAGE:-}"
+TAG_OVERRIDE="${IMAGE_TAG:-}"
 # shellcheck disable=SC1090
 source "${AGENT_ENV}"
+[ -z "${BASE_OVERRIDE}" ] || AGENT_BASE_IMAGE="${BASE_OVERRIDE}"
 : "${AGENT_BASE_IMAGE:?AGENT_BASE_IMAGE not set in ${AGENT_ENV}}"
 : "${NODEJS_MAJOR:?NODEJS_MAJOR not set in ${AGENT_ENV}}"
 : "${CLAUDE_CODE_VERSION:?CLAUDE_CODE_VERSION not set in ${AGENT_ENV}}"
@@ -55,16 +68,28 @@ source "${AGENT_ENV}"
 : "${AGENT_MODEL:?AGENT_MODEL not set in ${AGENT_ENV}}"
 : "${AGENT_IMAGE_REPO:?AGENT_IMAGE_REPO not set in ${AGENT_ENV}}"

-# --- the base must already exist ---------------------------------------------
-# Deliberately NOT built on demand. The toolbox build is its own reviewed artefact with its
-# own pins, and silently building it here would mean this script could change the toolchain
-# a worker gates with as a side effect of rebuilding the agent image.
-if ! "${ENGINE}" image exists "${AGENT_BASE_IMAGE}" 2>/dev/null; then
-  die "base image '${AGENT_BASE_IMAGE}' is not present. Build the pinned toolbox first:
-       ${REPO_ROOT}/infra/devcontainer/dev.sh build"
+# --- the base must exist, locally or in the registry ---------------------------
+# Deliberately NOT built on demand. The toolbox is its own reviewed artefact with its own
+# pins, and silently building it here would mean this script could change the toolchain a
+# worker gates with as a side effect of rebuilding the agent image.
+case "${AGENT_BASE_IMAGE}" in
+  *@sha256:0000000000000000000000000000000000000000000000000000000000000000)
+    die "AGENT_BASE_IMAGE is still the all-zero placeholder in ${AGENT_ENV}: the toolbox has
+       not been published yet. Either let CI publish it (.gitea/workflows/images.yaml) and
+       merge Renovate's digest bump, or build on a local toolbox:
+         ${REPO_ROOT}/infra/devcontainer/dev.sh build
+         AGENT_BASE_IMAGE=home-k3s-dev:local $0" ;;
+esac
+if ! "${ENGINE}" image inspect "${AGENT_BASE_IMAGE}" >/dev/null 2>&1; then
+  info "base ${AGENT_BASE_IMAGE} is not local; pulling it"
+  "${ENGINE}" pull "${AGENT_BASE_IMAGE}" >/dev/null \
+    || die "base image '${AGENT_BASE_IMAGE}' is neither local nor pullable (registry login?).
+       To build on a local toolbox instead:
+         ${REPO_ROOT}/infra/devcontainer/dev.sh build
+         AGENT_BASE_IMAGE=home-k3s-dev:local $0"
 fi

-IMAGE_TAG="claude${CLAUDE_CODE_VERSION}-node${NODEJS_MAJOR}"
+IMAGE_TAG="${TAG_OVERRIDE:-claude${CLAUDE_CODE_VERSION}-node${NODEJS_MAJOR}}"
 IMAGE_REF="${AGENT_IMAGE_REPO}:${IMAGE_TAG}"

 log "Building ${IMAGE_REF}"
@@ -75,7 +100,11 @@ info "  node        ${NODEJS_MAJOR}"
 info "  claude      ${CLAUDE_CODE_VERSION} (floor ${CLAUDE_CODE_MIN_VERSION})"
 info "  model       ${AGENT_MODEL}"

-"${ENGINE}" build \
+# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
+LABEL_ARGS=()
+[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
+
+"${ENGINE}" build "${LABEL_ARGS[@]}" \
  --file "${HERE}/Containerfile" \
  --tag "${IMAGE_REF}" \
  --build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}" \
@@ -138,63 +167,32 @@ if [ "${VERIFY}" -eq 1 ]; then
  # that will actually run in the cluster. The probe is mounted read-only rather than baked
  # in: it is a build-time check, not something a worker Job has any business carrying.
  log "Probing pm-worker's input validation inside the image"
-  "${ENGINE}" run --rm \
+  #
+  # The probe goes in over STDIN, not as a bind mount: under Docker-in-Docker (CI) a bind
+  # source is resolved on the daemon's filesystem, where this file does not exist, and the
+  # container would get an empty directory. PM_WORKER_PATH points it at the baked worker,
+  # since the probe's own default (a sibling of the probe file) has no meaning on stdin.
+  "${ENGINE}" run --rm -i \
    --user 65532:65532 \
    --read-only \
    --tmpfs /workspace:rw,mode=1777 \
    --tmpfs /tmp:rw,mode=1777 \
    --network none \
-    --volume "${HERE}/probe-validation.sh:/probe-validation.sh:ro,Z" \
+    -e PM_WORKER_PATH=/usr/local/bin/pm-worker \
    --entrypoint "" \
    "${IMAGE_REF}" \
-    /bin/bash /probe-validation.sh
+    /bin/bash -s < "${HERE}/probe-validation.sh"

  log "Image verified"
 fi

-# --- digest ------------------------------------------------------------------
-DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
+# --- result ------------------------------------------------------------------
+# No digest is printed: a manifest digest does not exist until a registry has the manifest,
+# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at
+# push time and prints it in the job summary (.gitea/workflows/scripts/images.sh).
 SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
-
 log "Built ${IMAGE_REF}"
-info "local manifest digest  ${DIGEST}"
-info "size                   $(( SIZE / 1024 / 1024 )) MiB"
-
-cat <<EOF
-
-================================================================================
-NOT PUSHED. Publishing is an operator action — this script holds no registry
-credential and will never acquire one.
-
-OPERATOR: run these, then put the digest from the digestfile into
-platform/pm-agent/configmap.yaml (PM_WORKER_IMAGE) via a pull request:
-
-    ${ENGINE} login gitea.home.keithjackson.dev
-
-    ${ENGINE} push --digestfile /tmp/agent-runner.digest ${IMAGE_REF}
-
-    cat /tmp/agent-runner.digest
-    # -> sha256:....   THIS is the authoritative digest, not the local one above.
-
-    # platform/pm-agent/configmap.yaml
-    PM_WORKER_IMAGE: "${AGENT_IMAGE_REPO}@\$(cat /tmp/agent-runner.digest)"
-
-WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/pm-agent-job-shape.yaml is a
-fail-closed Enforce policy. It DENIES any worker Job whose image is not
-${AGENT_IMAGE_REPO}@sha256:<digest> — a tag reference is denied,
-and so is any other repository path. The dispatcher also refuses to start if
-PM_WORKER_IMAGE carries no digest, so the failure names its own cause.
-
-THE SAME IMAGE RUNS THE DISPATCHER. platform/pm-agent/deployment.yaml pins it too, so a
-new digest is a two-line change in one pull request: the Deployment's image and
-PM_WORKER_IMAGE. They must be the same digest — the dispatcher and its workers share the
-committed Job template and the env-name contract, and a mismatched pair would disagree
-about it.
-
-TIGHTEN THE BOUNDARY AFTERWARDS: the policy admits the wildcard
-\`${AGENT_IMAGE_REPO}@sha256:*\`, so anyone who can push to that registry path can publish
-a different digest. Replacing the wildcard with the LITERAL digest closes that gap; it is a
-one-line change, reviewed in a pull request, which is exactly where a change to the
-security boundary belongs.
-================================================================================
-EOF
+info "image id  $("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
+info "size      $(( SIZE / 1024 / 1024 )) MiB"
+info "Not pushed. CI publishes this image from main (.gitea/workflows/images.yaml); see"
+info "platform/pm-agent/README.md for how a new digest reaches the cluster."
@@ -22,7 +22,9 @@
 ###############################################################################
 set -uo pipefail

-HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# `:-$0` because build-image.sh feeds this script over STDIN (`bash -s`), where BASH_SOURCE is
+# empty and `set -u` would abort on it; PM_WORKER_PATH is always set in that case.
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)"
 WORKER="${PM_WORKER_PATH:-${HERE}/pm-worker}"
 [ -r "$WORKER" ] || { echo "cannot read ${WORKER}" >&2; exit 2; }

@@ -9,10 +9,16 @@
 # truth and, the first time the two drifted, the worker's "green" would stop meaning the
 # gate's green. So this file pins ONLY what the toolbox does not carry.

-# The toolbox image the runner is layered on. Built by `infra/devcontainer/dev.sh build`
-# (tag `home-k3s-dev:local`); infra/agent/build-image.sh asserts it exists rather than
-# silently building a different base.
-AGENT_BASE_IMAGE=home-k3s-dev:local
+# The toolbox image the runner is layered on: the PUBLISHED toolbox, by digest. CI builds and
+# publishes it (.gitea/workflows/images.yaml) and Renovate moves this line onto each new digest
+# of its `main` tag via the marker below, in a pull request — nobody edits it by hand.
+#
+# ALL-ZERO UNTIL THE FIRST PUBLISH. The first run of the image pipeline on main sees this
+# placeholder, builds and publishes the toolbox first, and builds agent-runner FROM that fresh
+# digest. For a LOCAL build on a local toolbox, override it:
+#   infra/devcontainer/dev.sh build && AGENT_BASE_IMAGE=home-k3s-dev:local infra/agent/build-image.sh
+# renovate-digest: image=gitea.home.keithjackson.dev/home-network/toolbox tag=main
+AGENT_BASE_IMAGE=gitea.home.keithjackson.dev/home-network/toolbox@sha256:0000000000000000000000000000000000000000000000000000000000000000

 # Node.js — the runtime for the Claude Code CLI, installed from the Fedora base's module
 # stream. A major version, not a patch: the CLI supports the active LTS line and the
@@ -80,6 +80,12 @@ spec:
          # Replaced by the dispatcher with the digest-pinned image. A tag here would be
          # DENIED by admission; the dispatcher refuses a non-digest PM_WORKER_IMAGE at
          # startup so the error names the real cause.
+          #
+          # DELIBERATELY ALL-ZERO, AND DELIBERATELY NOT TRACKED BY RENOVATE. This file is
+          # baked INTO agent-runner, so it cannot carry agent-runner's own digest: bumping it
+          # would change the image's inputs, CI would publish a new digest, Renovate would bump
+          # it again — forever. pm-dispatch overwrites this field from PM_WORKER_IMAGE
+          # (platform/pm-agent/configmap.yaml, which IS tracked) on every Job it creates.
          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000
          imagePullPolicy: IfNotPresent
          # The pinned entrypoint, with NO args. Args on a pinned command are
@@ -20,10 +20,19 @@ restarted, and reboots are delegated to **kured** (`platform/kured`).

 Deploying either alone produces a mechanism that looks healthy and does nothing:

- **kured alone** watches a file that nothing creates. Measured 2026-10-17:
-  `update-notifier-common` — the package that owns the hook which writes
-  `/var/run/reboot-required` — was **not installed on any node**. kured would sit at 1/1 Running
-  reporting `kured_reboot_required=0` forever.
+- **kured alone** watches a file that **nothing on Debian ever creates**. Measured 2026-10-18,
+  after a first attempt to install Ubuntu's `update-notifier-common` failed on all three nodes
+  with *"No package matching 'update-notifier-common' is available"*:
+  - `apt-cache policy update-notifier-common` → `Candidate: (none)` — the package does not exist
+    in Debian; it is Ubuntu-only.
+  - `/usr/share/update-notifier/notify-reboot-required` → absent.
+  - **No `linux-image-*` postinst references `notify-reboot-required`** — calling it is an Ubuntu
+    patch, so even dropping a script at that path would never be invoked.
+  - No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
+
+  So **this role ships the writer itself** (`/usr/local/sbin/k3s-notify-reboot-required` plus a
+  `DPkg::Post-Invoke` hook) rather than installing a package. Without it kured would sit at 1/1
+  Running reporting `kured_reboot_required=0` forever.
 - **this role alone** installs patches that a long-running process never picks up, and when a
  reboot *is* needed there is nothing safe to take it. `Automatic-Reboot "true"` is not the
  answer: three control-plane nodes with etcd quorum 2/3, each rebooting at its own time, is how
@@ -36,7 +45,7 @@ Measured on all three prod nodes, 2026-10-17:
 | | |
 |---|---|
 | `unattended-upgrades` | **not installed** |
-| `update-notifier-common` | **not installed** |
+| `update-notifier-common` | **not installable** — Ubuntu-only, `Candidate: (none)` on trixie |
 | `needrestart` | **not installed** |
 | `apt-mark showhold` | **empty** — no package held on any node |
 | `/etc/apt/preferences.d/` | **empty** — no apt pin declares the backports-kernel intent |
@@ -157,7 +166,9 @@ first two):
 ```bash
 ssh [email protected] 'apt-config dump | grep -iE "unattended-upgrade::(automatic-reboot|package-blacklist|origins)"'
 ssh [email protected] 'systemctl list-timers apt-daily-upgrade.timer --no-pager'
-ssh [email protected] 'ls -la /var/run/reboot-required 2>&1; dpkg -l update-notifier-common needrestart unattended-upgrades | tail -4'
+ssh [email protected] 'ls -la /var/run/reboot-required* 2>&1 | head -2
+  ls -l /usr/local/sbin/k3s-notify-reboot-required /etc/apt/apt.conf.d/99-k3s-reboot-required 2>&1
+  dpkg -l needrestart unattended-upgrades | tail -3'
 ```

 Expect: `Automatic-Reboot "false"`, a blacklist containing `^linux-`, an `Origins-Pattern`
@@ -92,13 +92,11 @@ unattended_upgrades_enabled: false
 unattended_upgrades_packages:
  # The upgrader itself.
  - unattended-upgrades
-  # THE SENTINEL WRITER, AND THE WHOLE REASON kured CAN WORK. On Debian, /var/run/reboot-required is
-  # not created by the kernel, by apt, or by unattended-upgrades — it is created by a hook shipped in
-  # update-notifier-common. Measured ABSENT on all three nodes. Without this package the file is
-  # never written, kured correctly never reboots, every dashboard stays green, and the reboot half of
-  # this PR is an elaborate no-op. The role asserts the hook exists after install rather than
-  # assuming the package shipped one (see tasks/main.yml).
-  - update-notifier-common
+  # NOTE: `update-notifier-common` was listed here and REMOVED (2026-10-18). It is an UBUNTU
+  # package and does not exist in Debian at all -- `apt-cache policy update-notifier-common` on
+  # trixie returns `Candidate: (none)`, so the play failed outright with "No package matching
+  # 'update-notifier-common' is available" on all three nodes. The sentinel writer is now provided
+  # by this role itself; see unattended_upgrades_sentinel_* below and tasks/main.yml.
  # Restarts services still mapped to a deleted library after a security upgrade. Without it a
  # security-only policy patches bytes on disk that no running process reads — see DECISION 3.
  - needrestart
@@ -170,3 +168,32 @@ unattended_upgrades_needrestart_blacklist:
 unattended_upgrades_download_time: "03:30"
 unattended_upgrades_upgrade_time: "04:00"
 unattended_upgrades_randomised_delay: 60m
+
+
+# ── THE REBOOT SENTINEL, PROVIDED BY THIS ROLE ──────────────────────────────────────────────────
+# /var/run/reboot-required is the ONLY thing kured watches, and on Debian trixie **nothing creates
+# it**. Measured on all three nodes 2026-10-18:
+#   * `update-notifier-common` (which owns that hook on Ubuntu) has `Candidate: (none)` — the
+#     package does not exist in Debian. Installing it is what broke the first run of this play.
+#   * `/usr/share/update-notifier/notify-reboot-required` is absent.
+#   * NO `linux-image-*` postinst references `notify-reboot-required` — calling it is an Ubuntu
+#     patch, not Debian behaviour, so merely dropping a script at that path would never be invoked.
+#   * No hook under /etc/apt/apt.conf.d or /etc/kernel/ mentions reboot-required.
+# So the writer has to be ours. Without it kured is a permanent, INVISIBLE no-op: it stays 1/1
+# Running and reports kured_reboot_required=0 forever, which is the "failure mode is a green tick"
+# shape this repo keeps getting burned by.
+#
+# MECHANISM: a dpkg Post-Invoke hook runs a script that compares the RUNNING kernel against the
+# newest INSTALLED one and touches the sentinel when they differ. Deliberately kernel-only, which
+# matches the design: unattended-upgrades is security-only with `^linux-` blacklisted, so it will
+# essentially never set this — the sentinel appears when the OPERATOR deliberately runs the
+# node_baseline kernel play, and kured then automates only the mechanical reboot.
+#
+# It uses `linux-version` (from `linux-base`, a dependency of every linux-image package, so
+# guaranteed present — measured `linux-base 4.12.1` installed) rather than parsing `dpkg -l`
+# output, because kernel version ordering is not string ordering.
+#
+# /run is a tmpfs (measured), so the sentinel self-clears on boot: after a reboot the running
+# kernel IS the newest and the script writes nothing. No cleanup task is needed, and none exists.
+unattended_upgrades_sentinel_script: /usr/local/sbin/k3s-notify-reboot-required
+unattended_upgrades_sentinel_hook: /etc/apt/apt.conf.d/99-k3s-reboot-required
@@ -35,9 +35,9 @@
      tags: [packages, config, verify]

    # ── PACKAGES ────────────────────────────────────────────────────────────────────────────────
-    - name: Install unattended-upgrades, the reboot-sentinel writer and needrestart
-      # update-notifier-common is NOT optional padding: it owns the hook that creates
-      # /var/run/reboot-required, which is the only thing kured watches. See defaults/main.yml.
+    - name: Install unattended-upgrades and needrestart
+      # The reboot-sentinel writer is NOT a package here. On Debian nothing ships one -- see
+      # defaults/main.yml -- so this role installs its own script and dpkg hook, two tasks below.
      ansible.builtin.apt:
        name: "{{ unattended_upgrades_packages }}"
        state: present
@@ -45,6 +45,60 @@
        cache_valid_time: 3600
      tags: [packages]

+    # ── THE REBOOT SENTINEL WRITER (ours, because Debian ships none) ────────────────────────────
+    # Full rationale and the four measurements behind it: defaults/main.yml. Short version: kured
+    # watches /var/run/reboot-required and nothing on Debian creates it, so without these two files
+    # the entire reboot half is an invisible no-op.
+    - name: Install the reboot-sentinel writer
+      ansible.builtin.copy:
+        dest: "{{ unattended_upgrades_sentinel_script }}"
+        owner: root
+        group: root
+        mode: "0755"
+        content: |
+          #!/bin/sh
+          # MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits are reverted.
+          #
+          # Touch /var/run/reboot-required when the RUNNING kernel is not the newest INSTALLED one.
+          # kured watches that path; see the role's defaults/main.yml for why this is ours to ship.
+          #
+          # `linux-version` comes from linux-base (a dependency of every linux-image package), and
+          # is used rather than sorting `dpkg -l` output because kernel version ordering is not
+          # string ordering. If it is somehow missing, exit 0 and write nothing: a dpkg hook must
+          # never fail an apt transaction, and a missing sentinel is caught by the verify assert.
+          set -eu
+          command -v linux-version >/dev/null 2>&1 || exit 0
+
+          running="$(uname -r)"
+          newest="$(linux-version list 2>/dev/null | linux-version sort --reverse 2>/dev/null | head -n1)"
+
+          # No newest (nothing installed we can read) -> nothing to say.
+          [ -n "$newest" ] || exit 0
+
+          if [ "$running" != "$newest" ]; then
+            touch /var/run/reboot-required 2>/dev/null || exit 0
+            # kured only needs the file, but the .pkgs companion is what an operator reads to find
+            # out WHY a node is waiting, so write a reason rather than an empty marker.
+            printf 'linux-image-%s\n' "$newest" > /var/run/reboot-required.pkgs 2>/dev/null || true
+          fi
+          exit 0
+      tags: [packages, config]
+
+    - name: Install the dpkg hook that runs the sentinel writer
+      # Post-Invoke, so it runs after dpkg has finished and a newly installed kernel is already
+      # visible to `linux-version list`. `|| true` so a failure here can never abort an apt
+      # transaction -- an unwritten sentinel is a missed reboot, but a broken apt is a broken node.
+      ansible.builtin.copy:
+        dest: "{{ unattended_upgrades_sentinel_hook }}"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          // MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits reverted.
+          // Writes /var/run/reboot-required when the running kernel is not the newest installed.
+          DPkg::Post-Invoke { "{{ unattended_upgrades_sentinel_script }} || true"; };
+      tags: [packages, config]
+
    # ── CONFIG ──────────────────────────────────────────────────────────────────────────────────
    # Written as a whole file rather than patched in place, for the reason node_baseline gives for
    # its own apt source: an idempotent full-file write with a managed-by header is reviewable and
@@ -246,21 +300,24 @@
          Effective apt config confirmed: kernel blacklisted, automatic reboot disabled.
      tags: [verify]

-    - name: Find the hooks that can create /var/run/reboot-required
-      # THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN, and
-      # it is the one part that cannot be asserted from the repo — it depends on what
-      # update-notifier-common actually ships on this Debian release. Rather than hard-code a path
-      # that may not exist, discover the candidates and assert that at least one is present.
-      ansible.builtin.find:
-        paths:
-          - /etc/kernel/postinst.d
-          - /etc/apt/apt.conf.d
-        patterns:
-          - "*update-notifier*"
-          - "*unattended-upgrades*"
-          - "99update-notifier"
-        file_type: any
-      register: unattended_upgrades_sentinel_hooks
+    # THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN.
+    # This was originally a `find` over /etc/kernel/postinst.d and /etc/apt/apt.conf.d, on the
+    # theory that the writer came from a package and the role could only discover whatever that
+    # package happened to ship. That theory was wrong: no such package exists on Debian (see
+    # defaults/main.yml), so the writer is now OURS and lives at two known paths. Checking two
+    # exact paths is strictly stronger than a glob -- a glob would have matched Debian's own
+    # unattended-upgrades apt conffile and passed while no sentinel writer existed at all.
+    - name: Stat the sentinel writer script
+      ansible.builtin.stat:
+        path: "{{ unattended_upgrades_sentinel_script }}"
+      register: unattended_upgrades_sentinel_script_stat
+      changed_when: false
+      tags: [verify]
+
+    - name: Stat the sentinel dpkg hook
+      ansible.builtin.stat:
+        path: "{{ unattended_upgrades_sentinel_hook }}"
+      register: unattended_upgrades_sentinel_hook_stat
      changed_when: false
      tags: [verify]

@@ -272,18 +329,21 @@
      # before, so it is an assert and not a debug message.
      ansible.builtin.assert:
        that:
-          - unattended_upgrades_sentinel_hooks.matched | int > 0
+          - unattended_upgrades_sentinel_script_stat.stat.exists
+          - unattended_upgrades_sentinel_script_stat.stat.executable | default(false)
+          - unattended_upgrades_sentinel_hook_stat.stat.exists
        fail_msg: >-
-          NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required. No
-          update-notifier / unattended-upgrades hook was found under /etc/kernel/postinst.d or
-          /etc/apt/apt.conf.d. kured watches that path and reboots only when it exists, so with no
-          writer the reboot automation is permanently and INVISIBLY idle -- kured stays 1/1 Running
-          and reports kured_reboot_required=0 on every node forever. Confirm
-          update-notifier-common is installed (`dpkg -l update-notifier-common`) and inspect
-          `dpkg -L update-notifier-common | grep -E 'postinst.d|apt.conf.d'`.
+          NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required.
+          {{ unattended_upgrades_sentinel_script }} (executable) and
+          {{ unattended_upgrades_sentinel_hook }} must both exist -- this role ships both, because
+          Debian provides no sentinel writer of its own (update-notifier-common is an UBUNTU
+          package and is not installable here). kured watches that path and reboots only when it
+          exists, so with no writer the reboot automation is permanently and INVISIBLY idle --
+          kured stays 1/1 Running and reports kured_reboot_required=0 on every node forever.
+          Re-run this role without `--tags verify` to install them.
        success_msg: >-
-          Reboot sentinel writer present ({{ unattended_upgrades_sentinel_hooks.matched }} hook(s)
-          found) -- a kernel or flagged package upgrade will signal kured.
+          Reboot sentinel writer present ({{ unattended_upgrades_sentinel_script }} + dpkg hook)
+          -- a kernel change will signal kured on the next apt transaction.
      tags: [verify]

    - name: Report the pending-reboot state of this node
@@ -34,6 +34,22 @@ cd infra/devcontainer
 ```
 `ENGINE=docker ./dev.sh` to use docker instead of rootless podman.

+### The published toolbox
+
+`./dev.sh build` delegates to [`build-image.sh`](build-image.sh), which is the **one** build
+recipe: CI runs the same script (with its in-image verification — every tool executes, and the
+version-sensitive ones report their `versions.env` pins) and, on a push to `main` that changes
+`Containerfile`, `versions.env` or `build-image.sh`, **publishes** the result as
+`gitea.home.keithjackson.dev/home-network/toolbox` (tags `sha-<commit>` and `main`) —
+see [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). `agent-runner` is
+built `FROM` that published digest (`infra/agent/versions.env`, kept current by Renovate).
+Nobody pushes this image from a workstation. To use the published one locally instead of
+building, after `podman login gitea.home.keithjackson.dev` (a token with `read:package`):
+
+```bash
+IMAGE=gitea.home.keithjackson.dev/home-network/toolbox:main ./dev.sh run make -C ci gate
+```
+
 ## Choosing a cluster — `CLUSTER=dev|prod`

 ```bash
@@ -0,0 +1,108 @@
+#!/usr/bin/env bash
+###############################################################################
+# infra/devcontainer/build-image.sh — build the pinned dev `toolbox` image and prove it works.
+#
+#   ./build-image.sh                 # build + verify (tag: home-k3s-dev:local)
+#   ./build-image.sh --no-verify     # build only
+#   IMAGE=foo:bar ./build-image.sh   # build under another local name
+#   ENGINE=docker ./build-image.sh   # build with docker instead of podman
+#
+# ONE BUILD DEFINITION, TWO CALLERS. `dev.sh build` calls this for local development, and the
+# CI image pipeline (.gitea/workflows/images.yaml -> .gitea/workflows/scripts/images.sh) calls
+# it to build the image it then PUBLISHES as gitea.home.keithjackson.dev/home-network/toolbox.
+# Keeping the build args in exactly one place is what makes "the toolbox CI publishes" and "the
+# toolbox you gate with locally" the same recipe.
+#
+# THIS SCRIPT NEVER PUSHES. Publishing is CI's job (images.yaml, on push to main), not a
+# workstation's: an image that exists only because someone ran `podman push` from a laptop has
+# no build log, no recorded inputs and no verification anyone else can read.
+#
+# VERSIONS ARE NOT DEFINED HERE. infra/devcontainer/versions.env is the single source of truth.
+###############################################################################
+set -euo pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ENGINE="${ENGINE:-podman}"
+IMAGE="${IMAGE:-home-k3s-dev:local}"
+
+VERIFY=1
+for arg in "$@"; do
+  case "$arg" in
+    --no-verify) VERIFY=0 ;;
+    -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
+    *) echo "unknown argument: $arg" >&2; exit 2 ;;
+  esac
+done
+
+log()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
+die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
+
+command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
+
+# shellcheck disable=SC1091
+source "${HERE}/versions.env"
+
+# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
+LABEL_ARGS=()
+[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
+
+log "Building ${IMAGE}"
+"$ENGINE" build "${LABEL_ARGS[@]}" \
+  --build-arg FEDORA_VERSION="$FEDORA_VERSION" \
+  --build-arg OPENTOFU_VERSION="$OPENTOFU_VERSION" \
+  --build-arg AGE_VERSION="$AGE_VERSION" \
+  --build-arg SOPS_VERSION="$SOPS_VERSION" \
+  --build-arg FLUX_VERSION="$FLUX_VERSION" \
+  --build-arg KUBECTL_VERSION="$KUBECTL_VERSION" \
+  --build-arg HELM_VERSION="$HELM_VERSION" \
+  --build-arg KYVERNO_VERSION="$KYVERNO_VERSION" \
+  --build-arg KYVERNO_JSON_VERSION="$KYVERNO_JSON_VERSION" \
+  --build-arg CHAINSAW_VERSION="$CHAINSAW_VERSION" \
+  --build-arg YAMLLINT_VERSION="$YAMLLINT_VERSION" \
+  --build-arg YQ_VERSION="$YQ_VERSION" \
+  --build-arg CHECK_JSONSCHEMA_VERSION="$CHECK_JSONSCHEMA_VERSION" \
+  --build-arg PROMTOOL_VERSION="$PROMTOOL_VERSION" \
+  -t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}"
+
+# --- verify ------------------------------------------------------------------
+# A build that succeeded is not evidence the toolbox can run the gate. Every tool the gate and
+# the runbooks call must EXECUTE, and the version-sensitive ones must report the version
+# versions.env pins — promtool above all, because ci/versions.env asserts a promtool/Prometheus
+# lockstep and a drifted engine evaluates alert-rule tests differently (see versions.env).
+if [ "${VERIFY}" -eq 1 ]; then
+  log "Verifying the toolchain inside ${IMAGE}"
+  "$ENGINE" run --rm --network none --entrypoint "" \
+    -e PROMTOOL_VERSION="${PROMTOOL_VERSION}" \
+    -e KUBECTL_VERSION="${KUBECTL_VERSION}" \
+    -e CHAINSAW_VERSION="${CHAINSAW_VERSION}" \
+    -e KYVERNO_VERSION="${KYVERNO_VERSION}" \
+    -e SOPS_VERSION="${SOPS_VERSION}" \
+    "$IMAGE" /bin/bash -euo pipefail -c '
+      echo "--- baked versions ---"; cat /etc/dev-toolchain-versions
+      echo "--- every tool is on PATH ---"
+      for t in tofu age age-keygen sops flux kubectl helm kyverno kyverno-json chainsaw yq \
+               yamllint check-jsonschema promtool ansible-playbook git make jq diff python3 \
+               wg qrencode; do
+        if command -v "$t" >/dev/null 2>&1; then printf "    ok      %s\n" "$t";
+        else printf "    MISSING %s\n" "$t"; missing=1; fi
+      done
+      [ -z "${missing:-}" ] || { echo "toolbox is missing tools"; exit 1; }
+      echo "--- version-sensitive tools report their pins ---"
+      check() { # name pin actual-output
+        if printf "%s" "$3" | grep -qF "$2"; then printf "    ok      %-10s %s\n" "$1" "$2";
+        else printf "    DRIFT   %-10s want %s, got: %s\n" "$1" "$2" "$(printf "%s" "$3" | head -1)"; drift=1; fi
+      }
+      check promtool "${PROMTOOL_VERSION}" "$(promtool --version 2>&1)"
+      check kubectl  "${KUBECTL_VERSION}"  "$(kubectl version --client 2>&1)"
+      check chainsaw "${CHAINSAW_VERSION}" "$(chainsaw version 2>&1)"
+      check kyverno  "${KYVERNO_VERSION}"  "$(kyverno version 2>&1)"
+      check sops     "${SOPS_VERSION}"     "$(sops --version --disable-version-check 2>&1 || sops --version 2>&1)"
+      [ -z "${drift:-}" ] || { echo "a tool does not report its pinned version"; exit 1; }
+      echo "--- python can import what the renderer and the gate need ---"
+      python3 -c "import yaml; print(\"    PyYAML\", yaml.__version__)"
+    '
+  log "Toolbox verified"
+fi
+
+log "Built ${IMAGE}"
+printf '    image id  %s\n' "$("$ENGINE" image inspect --format '{{.Id}}' "$IMAGE")"
@@ -72,22 +72,10 @@ esac
 CLUSTER_KUBECONFIG="${REPO_ROOT}/infra/ansible/.cache/${CLUSTER}.kubeconfig"

 build() {
-  "$ENGINE" build \
-    --build-arg FEDORA_VERSION="$FEDORA_VERSION" \
-    --build-arg OPENTOFU_VERSION="$OPENTOFU_VERSION" \
-    --build-arg AGE_VERSION="$AGE_VERSION" \
-    --build-arg SOPS_VERSION="$SOPS_VERSION" \
-    --build-arg FLUX_VERSION="$FLUX_VERSION" \
-    --build-arg KUBECTL_VERSION="$KUBECTL_VERSION" \
-    --build-arg HELM_VERSION="$HELM_VERSION" \
-    --build-arg KYVERNO_VERSION="$KYVERNO_VERSION" \
-    --build-arg KYVERNO_JSON_VERSION="$KYVERNO_JSON_VERSION" \
-    --build-arg CHAINSAW_VERSION="$CHAINSAW_VERSION" \
-    --build-arg YAMLLINT_VERSION="$YAMLLINT_VERSION" \
-    --build-arg YQ_VERSION="$YQ_VERSION" \
-    --build-arg CHECK_JSONSCHEMA_VERSION="$CHECK_JSONSCHEMA_VERSION" \
-    --build-arg PROMTOOL_VERSION="$PROMTOOL_VERSION" \
-    -t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}"
+  # The build recipe lives in build-image.sh so that CI (.gitea/workflows/images.yaml), which
+  # builds and PUBLISHES this image, and a local `dev.sh build` cannot drift apart. --no-verify
+  # keeps the local loop as fast as it always was; CI runs the verification.
+  IMAGE="$IMAGE" ENGINE="$ENGINE" "${HERE}/build-image.sh" --no-verify
 }

 run() {
@@ -252,11 +252,16 @@ step "preconditions — repo side (these block ONLY arming ${VARIABLE}; the secr

 # R1: the runner image is built and digest-pinned. Without it the Job cannot run run.sh at
 # all (the stock chainsaw image has no bash/python3/kubectl/git) — plan §3 item 1.
+# EVALUATED, not grepped: the line is `CONFORMANCE_RUNNER_IMAGE="${..._REPO}@${..._DIGEST}"`, so
+# its literal text never contains 64 hex digits and the grep this replaced could never pass. The
+# all-zero digest is the "not published yet" placeholder (Renovate replaces it after CI's first
+# publish, .gitea/workflows/images.yaml) and must not count as pinned.
 VERSIONS="${REPO_ROOT}/conformance/versions.env"
-if grep -Eq '^[[:space:]]*CONFORMANCE_RUNNER_IMAGE=.*@sha256:[0-9a-f]{64}' "$VERSIONS" 2>/dev/null; then
-  pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env"
+runner_img="$( (set +u; . "$VERSIONS" >/dev/null 2>&1; printf '%s' "${CONFORMANCE_RUNNER_IMAGE:-}") )"
+if [[ "$runner_img" =~ @sha256:[0-9a-f]{64}$ ]] && [[ ! "$runner_img" =~ @sha256:0{64}$ ]]; then
+  pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env (${runner_img##*@})"
 else
-  fail_arm "conformance/versions.env has no digest-pinned CONFORMANCE_RUNNER_IMAGE (plan §3 item 1) — the Job cannot run ./run.sh"
+  fail_arm "conformance/versions.env has no PUBLISHED digest for CONFORMANCE_RUNNER_IMAGE (got '${runner_img}') — the image pipeline has not published it yet, or Renovate's bump is unmerged (plan §3 item 1)"
 fi

 # R2: the Job template is the real thing, not the skeleton: the admitted image AND an
@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+# Registry retention for this repo's OWN images (toolbox, agent-runner, conformance-runner):
+# keep what is deployed and what is recent, delete the rest. DRY RUN unless --apply.
+#
+#   ./infra/gitea/registry-retention.sh              # show what would be deleted
+#   ./infra/gitea/registry-retention.sh --apply      # delete it
+#   KEEP=15 ./infra/gitea/registry-retention.sh      # keep the 15 newest sha- tags (default 10)
+#
+# RUNNING IT IS AN OPERATOR ACTION. It deletes registry content, so it is not wired into CI and
+# no agent runs it with --apply.
+#
+# WHY A SCRIPT, AND NOT GITEA'S BUILT-IN "CLEANUP RULE" — measured against Gitea 1.24.3's source:
+#
+#   1. There is NO API for cleanup rules. routers/api/v1/api.go has no such route; the rules live
+#      only behind the web UI's session+CSRF forms (routers/web/web.go, org settings
+#      `/packages/rules/add`). A rule therefore cannot be created, read or drift-checked as code —
+#      which is also why ci/gitea-settings.sh cannot assert one exists.
+#   2. A built-in rule would DELETE WHAT IS DEPLOYED. Every reference to these images is
+#      digest-only (`<repo>@sha256:...`, required by admission). Gitea stores a manifest as a
+#      file OF THE TAG VERSION it was pushed under (routers/api/packages/container/manifest.go
+#      createPackageAndVersion), and resolves a by-digest pull by searching those files
+#      (models/packages/container/search.go). So deleting the tag a digest was pushed under makes
+#      that digest unpullable — MANIFEST_UNKNOWN, i.e. ImagePullBackOff on the next reschedule.
+#      The built-in rule keeps "the newest N" and a name pattern, and knows nothing about which
+#      digest git currently pins. Renovate's bump PRs can lag, so "newest N" is not a guarantee.
+#
+# WHAT IS KEPT, per image:
+#   * the `main` and `latest` tags;
+#   * the KEEP newest `sha-<commit>` tags (by creation time);
+#   * EVERY tag whose manifest digest is referenced anywhere in the tracked tree at REF
+#     (default origin/main) — i.e. whatever Flux is deploying, and the toolbox digest
+#     agent-runner builds FROM;
+#   * every version whose NAME is a digest (`sha256:...`): those are index children, and Gitea's
+#     own cleanup handles them (services/packages/container/cleanup.go ShouldBeSkipped).
+# Everything else is deleted. Gitea's `cleanup_packages` cron then reclaims the unreferenced
+# blobs (default: daily, blobs older than 24h) — space comes back the day after, not instantly.
+#
+# THE STORAGE IT PROTECTS. Packages live on the SAME 10Gi Longhorn PVC as every repository and
+# Gitea's database (catalog/gitea.yaml). The toolbox is ~1.4 GB uncompressed and every main push
+# that touches its inputs adds a complete, non-reproducible copy. Unpruned, the registry is what
+# fills the volume the git host lives on.
+#
+# CREDENTIAL: the one `git` has stored for the host (as the other scripts here). It needs
+# `read:package` to list and `write:package` to --apply, and the account needs package write on
+# the org. The secret is never printed.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; NS=home-network
+IMAGES=(toolbox agent-runner conformance-runner)
+KEEP="${KEEP:-10}"
+REF="${REF:-origin/main}"
+APPLY=0
+for a in "$@"; do
+  case "$a" in
+    --apply) APPLY=1 ;;
+    -h|--help) sed -n '2,/^set -euo/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
+    *) echo "unknown argument: $a" >&2; exit 2 ;;
+  esac
+done
+[[ "$KEEP" =~ ^[0-9]+$ ]] && [ "$KEEP" -ge 1 ] || { echo "KEEP must be a positive integer" >&2; exit 2; }
+command -v jq >/dev/null || { echo "jq is required" >&2; exit 1; }
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+
+git -C "$REPO_ROOT" fetch -q origin main 2>/dev/null || echo "   (could not fetch origin/main; using the local ${REF})" >&2
+git -C "$REPO_ROOT" rev-parse -q --verify "${REF}^{commit}" >/dev/null || { echo "no such ref: ${REF}" >&2; exit 1; }
+
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
+echo "authenticating as '${U}' (secret not shown)"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+
+REG_TOKEN="$(api "https://${HOST}/v2/token?service=container_registry" | jq -r '.token // empty')"
+[ -n "$REG_TOKEN" ] || { echo "registry token exchange failed (does the credential have read:package?)" >&2; exit 1; }
+manifest_digest() { # <image> <tag> -> digest, or nothing
+  curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "$REG_TOKEN") \
+    -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
+    -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
+    -H 'Accept: application/vnd.oci.image.index.v1+json' \
+    -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
+    "https://${HOST}/v2/${NS}/$1/manifests/$2" \
+    | tr -d '\r' | awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' | tail -1
+}
+
+TOTAL_DEL=0
+for img in "${IMAGES[@]}"; do
+  echo
+  echo "==> ${NS}/${img}"
+  # Digests git pins at REF. The marker convention puts the digest on the line after
+  # `# renovate-digest: image=<repo>`; the plain `<repo>@sha256:` grep catches any other use.
+  repo="${HOST}/${NS}/${img}"
+  pinned="$( { git -C "$REPO_ROOT" grep -h -A1 -F "renovate-digest: image=${repo} " "$REF" -- . 2>/dev/null || true
+               git -C "$REPO_ROOT" grep -h -F "${repo}@sha256:" "$REF" -- . 2>/dev/null || true; } \
+             | grep -oE 'sha256:[0-9a-f]{64}' | grep -v 'sha256:0\{64\}' | sort -u || true)"
+  echo "    pinned at ${REF}: $(wc -w <<<"$pinned") digest(s)"
+
+  versions="[]"; page=1
+  while :; do
+    chunk="$(api "https://${HOST}/api/v1/packages/${OWNER}?type=container&q=${img}&limit=50&page=${page}")"
+    jq -e 'type=="array"' >/dev/null <<<"$chunk" || { echo "    package list failed: $(head -c 200 <<<"$chunk")" >&2; exit 1; }
+    [ "$(jq length <<<"$chunk")" -gt 0 ] || break
+    versions="$(jq -c --argjson a "$versions" --arg n "$img" '$a + [ .[] | select(.name==$n) | {version, created_at} ]' <<<"$chunk")"
+    page=$((page + 1))
+  done
+  echo "    versions: $(jq length <<<"$versions")"
+
+  keep_sha="$(jq -r '[ .[] | select(.version|startswith("sha-")) ] | sort_by(.created_at) | reverse | .[:'"$KEEP"'][] .version' <<<"$versions")"
+  while IFS= read -r v; do
+    [ -n "$v" ] || continue
+    case "$v" in
+      main|latest)  echo "    keep    ${v}  (moving tag)"; continue ;;
+      sha256:*)     echo "    keep    ${v}  (digest-named version; Gitea's cleanup owns those)"; continue ;;
+    esac
+    if grep -qxF "$v" <<<"$keep_sha"; then echo "    keep    ${v}  (among the ${KEEP} newest)"; continue; fi
+    d="$(manifest_digest "$img" "$v")"
+    if [ -z "$d" ]; then echo "    keep    ${v}  (could not read its digest — never delete what cannot be checked)"; continue; fi
+    if grep -qxF "$d" <<<"$pinned"; then echo "    keep    ${v}  (${d} is pinned at ${REF})"; continue; fi
+    if [ "$APPLY" -eq 1 ]; then
+      code="$(api -o /dev/null -w '%{http_code}' -X DELETE \
+        "https://${HOST}/api/v1/packages/${OWNER}/container/${img}/$(jq -rn --arg v "$v" '$v|@uri')")"
+      if [ "$code" = 204 ]; then echo "    DELETED ${v}  (${d})"; TOTAL_DEL=$((TOTAL_DEL + 1));
+      else echo "    FAILED  ${v}  HTTP ${code}" >&2; fi
+    else
+      echo "    would delete ${v}  (${d})"; TOTAL_DEL=$((TOTAL_DEL + 1))
+    fi
+  done < <(jq -r '.[].version' <<<"$versions")
+done
+
+echo
+if [ "$APPLY" -eq 1 ]; then echo "deleted ${TOTAL_DEL} version(s). Blobs are reclaimed by Gitea's cleanup_packages cron."
+else echo "DRY RUN: ${TOTAL_DEL} version(s) would be deleted. Re-run with --apply to delete them."; fi
@@ -45,11 +45,18 @@ waive the sensitive-path human review on its own pull request.

 Both are deliberate, and both fail **visibly** rather than silently:

-1. **The image does not exist.** `platform/pm-agent/deployment.yaml` and
-   `PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder. Build it with
-   `infra/agent/build-image.sh` (which verifies it and refuses to push), push it as the
-   operator, then update **both** places with the digest `podman push --digestfile` returned,
-   in one pull request. Until then: `ImagePullBackOff`.
+1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
+   `PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder until the first
+   publish. `.gitea/workflows/images.yaml` builds `agent-runner` with
+   `infra/agent/build-image.sh` (the same script, and the same in-image verification, you can
+   run locally), and on a push to `main` publishes it to
+   `gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
+   Renovate then follows the `main` tag through the `# renovate-digest:` marker above **both**
+   lines and moves them together in one pull request. Nobody builds-and-pushes this image from
+   a workstation, and nobody pastes a digest. Until the first publish, and until that PR
+   merges: `ImagePullBackOff`. The one operator prerequisite on the publishing side is the
+   `REGISTRY_PUSH_TOKEN` Actions secret (see the header of `.gitea/workflows/images.yaml`);
+   without it the publish job fails loudly rather than skipping.

   **Two things have to be true before any digest will pull, and neither is declared in this
   repository.** Both were measured after PR #57 merged — this service is the first workload
@@ -78,8 +85,8 @@ Both are deliberate, and both fail **visibly** rather than silently:
     account whose token carries `read:package`, referenced from the Deployment and the
     committed worker Job template — not built yet, because minting that token is yours.

-   The same two apply to `conformance-runner`: same registry path, same push-then-pin model,
-   never pulled either.
+   The same two apply to `conformance-runner`: same registry path, same publish-then-Renovate
+   model, never pulled either.
 2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the
   repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off —
   which Gitea account to create, which scopes, and why it must *not* be an administrator —
@@ -52,12 +52,13 @@ data:
  # contract. A tag here is refused at startup (and would be DENIED by admission anyway).
  #
  # PLACEHOLDER UNTIL THE IMAGE IS PUBLISHED. The all-zero digest below is deliberately
-  # impossible: `infra/agent/build-image.sh` builds and verifies the image, the OPERATOR
-  # pushes it, and the authoritative digest from `--digestfile` replaces this value in a
-  # reviewed pull request (the same two-step the conformance-runner image follows, and for
-  # the same reason: a registry credential is not an agent's to hold). Until then the
-  # dispatcher starts, fails to pull, and says so — which is the visible failure, not a
-  # silent one.
+  # impossible. CI builds, verifies and — on main — publishes agent-runner
+  # (.gitea/workflows/images.yaml), and Renovate replaces this value with the digest the
+  # REGISTRY returned for the image's `main` tag, in a pull request that also moves the
+  # Deployment's image (same marker, same branch). Nobody pushes this image from a workstation.
+  # Until the first publish the dispatcher starts, fails to pull, and says so — which is the
+  # visible failure, not a silent one.
+  # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
  PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000"

  # --- the model -------------------------------------------------------------
@@ -66,9 +66,12 @@ spec:
        - name: dispatch
          # THE SAME IMAGE THE WORKERS RUN, at the same digest. They share the committed Job
          # template and the env-name contract baked into it, so a mismatched pair would
-          # disagree about the shape the admission policy enforces. Replaced together, in one
-          # pull request, with the digest `infra/agent/build-image.sh` printed — the all-zero
+          # disagree about the shape the admission policy enforces. Replaced together, in ONE
+          # Renovate pull request: both lines carry the same marker, so Renovate treats them as
+          # one dependency and bumps them in the same branch. The digest comes from CI's publish
+          # of agent-runner's `main` tag (.gitea/workflows/images.yaml) — the all-zero
          # placeholder below is impossible on purpose (see configmap.yaml PM_WORKER_IMAGE).
+          # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000
          imagePullPolicy: IfNotPresent
          command:
@@ -1,99 +0,0 @@
-{
-  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
-  "_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17 RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags. It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009 split that gave image tags to Flux image-automation is formally closed — see .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write anyway; and enabling it would need an in-cluster git WRITE credential for the repo that defines the cluster). image-automation has been commented out at platform/kustomization.yaml:52 since it was authored and the cluster holds zero ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down. CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag` line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER: .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.",
-  "extends": [
-    "config:recommended",
-    ":dependencyDashboard",
-    ":semanticCommits",
-    "helpers:pinGitHubActionDigests"
-  ],
-  "timezone": "America/New_York",
-  "schedule": ["after 3am and before 6am"],
-  "prConcurrentLimit": 5,
-  "prHourlyLimit": 2,
-  "commitMessagePrefix": "chore(deps):",
-  "labels": ["dependencies", "automated-update"],
-  "ignoreUnstable": true,
-  "respectLatest": true,
-  "enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
-  "flux": {
-    "managerFilePatterns": ["/(^|/)platform/.+\\.ya?ml$/", "/(^|/)apps/.+\\.ya?ml$/"]
-  },
-  "kustomize": {
-    "managerFilePatterns": ["/(^|/)kustomization\\.ya?ml$/"]
-  },
-  "_kubernetes_comment": "ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes manifests, and it is NOT auto-detected — it only looks at files matched here, which is why this block is as broad as it is. Scope covers every place this repo writes a container image: platform components, the rendered app waves, the ServiceDefinition catalog that generates them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the rendered output is reverted by the next render and would also fail ci/render-golden.sh. Renovate will raise BOTH, and both must land together.",
-  "kubernetes": {
-    "managerFilePatterns": [
-      "/(^|/)platform/.+\\.ya?ml$/",
-      "/(^|/)platform-config/.+\\.ya?ml$/",
-      "/(^|/)apps/.+\\.ya?ml$/",
-      "/(^|/)catalog/.+\\.ya?ml$/",
-      "/(^|/)runbooks/serving/.+\\.ya?ml$/",
-      "/(^|/)infra/agent/.+\\.ya?ml$/"
-    ]
-  },
-  "customManagers": [
-    {
-      "customType": "regex",
-      "description": "Inline `# renovate: datasource=... depName=...` markers on pinned URLs / version literals (e.g. platform/kubevirt/kustomization.yaml release URLs, ci/versions.env, infra/devcontainer/versions.env).",
-      "managerFilePatterns": [
-        "/(^|/)kustomization\\.ya?ml$/",
-        "/(^|/)versions\\.env$/"
-      ],
-      "matchStrings": [
-        "#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n.*?(?<currentValue>v?[0-9]+\\.[0-9]+\\.[0-9]+)"
-      ],
-      "versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
-    }
-  ],
-  "packageRules": [
-    {
-      "description": "Auto-merge the allowed low-risk classes on green CI — no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.",
-      "matchUpdateTypes": ["minor", "patch", "digest"],
-      "automerge": true,
-      "automergeType": "pr",
-      "platformAutomerge": true
-    },
-    {
-      "description": "MAJOR / potentially-breaking bumps require human review (docs/11 safety rail). Never auto-merged.",
-      "matchUpdateTypes": ["major"],
-      "automerge": false,
-      "addLabels": ["needs-review", "breaking"]
-    },
-    {
-      "description": "Keep the KubeVirt + CDI operator pair on a compatible track — grouped so they bump together (platform/kubevirt/kustomization.yaml).",
-      "matchDepNames": ["kubevirt/kubevirt", "kubevirt/containerized-data-importer"],
-      "groupName": "kubevirt-cdi"
-    },
-    {
-      "description": "Pinned tool versions (ci/versions.env, infra/devcontainer/versions.env) are a deliberate change surface — label but do not auto-merge; a version bump there changes CI reproducibility.",
-      "matchFileNames": ["ci/versions.env", "infra/devcontainer/versions.env"],
-      "automerge": false,
-      "addLabels": ["ci-tooling"]
-    },
-    {
-      "description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check — it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
-      "matchPackageNames": [
-        "kured",
-        "longhorn",
-        "cilium",
-        "cloudnative-pg",
-        "kube-prometheus-stack"
-      ],
-      "automerge": false,
-      "addLabels": ["needs-review", "blast-radius"]
-    }
-  ],
-  "_unmanageable_comment": "WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest, lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete version before any automation can maintain them, and until then they drift on every pull with nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it has been told not to leave; raising those ceilings is deliberate per-component work and is NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.",
-  "ignorePaths": [
-    "**/*.example",
-    "**/*.sops.yaml",
-    "conformance/**",
-    "platform/image-automation/per-service/**",
-    "platform/kyverno/tests/**"
-  ],
-  "_ignorePaths_comment": "platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest` and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed shape, not software to keep current. `conformance/**` was already ignored for the same reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise have opened.",
-  "_docker_comment": "The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation over a running tag. That controller was never enabled and never will be here (see _comment), so the only thing those two flags were achieving was leaving every plain `image:` tag in the repo unmanaged. `docker` is left out of enabledManagers because this repo contains no Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.",
-  "docker": { "enabled": true }
-}
@@ -0,0 +1,272 @@
+// =================================================================================================
+// Renovate configuration — JSON5, so the rationale can live NEXT TO what it explains.
+// =================================================================================================
+// THIS FILE WAS `renovate.json` AND IT WAS INVALID. It carried five `_comment` / `_*_comment`
+// pseudo-comment keys, because JSON has no comments. Renovate REJECTS unknown top-level options, so
+// the whole config failed validation and the first manual run of the workflow ERRORED OUT. Proven
+// with the official validator (`ghcr.io/renovatebot/renovate renovate-config-validator`):
+//
+//   ERROR: Found errors in configuration
+//     Invalid configuration option: _comment
+//     Invalid configuration option: _docker_comment
+//     Invalid configuration option: _ignorePaths_comment
+//     Invalid configuration option: _kubernetes_comment
+//     Invalid configuration option: _unmanageable_comment
+//
+// Renovate discovers `renovate.json5` natively (no RENOVATE_CONFIG_FILE needed, and nothing in
+// .gitea/ or ci/ pins the old filename — only prose in comments), and JSON5 permits real `//`
+// comments. So the fix keeps every word of the original rationale, positioned, instead of
+// flattening it into a `description` string.
+//
+// VALIDATE BEFORE COMMITTING — it is one command and it is the only thing that catches this class:
+//   podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \
+//     ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json5
+//
+// CORRECTION (2026-10-29): THE VALIDATOR ABOVE WAS NOT THE RENOVATE THE WORKFLOW RUNS. The bare
+// `ghcr.io/renovatebot/renovate` tag is `latest` (44.167.0 when this was measured), while
+// renovatebot/[email protected] defaults `renovate-version` to `39` (its action.yml), i.e.
+// 39.264.0. Validated against THAT, this file failed with five errors — every
+// `managerFilePatterns` key ("Custom Manager contains disallowed fields", "Invalid configuration
+// option: flux.managerFilePatterns", ...), because 39.x only knows `fileMatch`; the rename landed
+// in 40.2.0. So even after the workflow found this file, every run would have stopped at
+// CONFIG_VALIDATION. renovate.yaml now pins `renovate-version` to the exact release this file is
+// validated against. VALIDATE WITH THAT SAME TAG:
+//   podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \
+//     ghcr.io/renovatebot/renovate:<renovate-version from renovate.yaml> \
+//     renovate-config-validator /tmp/renovate.json5
+// (Not `--strict`: that also fails on a pending migration of the pre-existing top-level
+// `"docker": {"enabled": true}` block at the bottom, which is a warning, not an error.)
+//
+// `regex` vs `custom.regex` in enabledManagers, RESOLVED FROM SOURCE (Renovate 39.264.0 and
+// later): both work. lib/config/migrations/custom/enabled-managers-migration.ts rewrites
+// `regex` -> `custom.regex`, and that migration runs on every real repository run
+// (lib/workers/repository/init/merge.ts -> migrateAndValidate), not only in the validator;
+// lib/modules/manager/index.ts getEnabledManagersList accepts either spelling. `custom.regex` is
+// the canonical name (introduced in 37.51.1, renovatebot/renovate#24079), so it is used below and
+// nothing depends on the migration any more.
+{  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
+  // Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17
+  // RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags.
+  // It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009
+  // split that gave image tags to Flux image-automation is formally closed — see
+  // .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than
+  // opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with
+  // user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write
+  // anyway; and enabling it would need an in-cluster git WRITE credential for the repo that
+  // defines the cluster). image-automation has been commented out at
+  // platform/kustomization.yaml:52 since it was authored and the cluster holds zero
+  // ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down.
+  // CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid
+  // fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag`
+  // line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth
+  // sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER:
+  // .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.
+  "extends": ["config:recommended", ":dependencyDashboard", ":semanticCommits", "helpers:pinGitHubActionDigests"],
+  "timezone": "America/New_York",
+  "schedule": ["after 3am and before 6am"],
+  "prConcurrentLimit": 5,
+  "prHourlyLimit": 2,
+  "commitMessagePrefix": "chore(deps):",
+  "labels": ["dependencies", "automated-update"],
+  "ignoreUnstable": true,
+  "respectLatest": true,
+  "enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "custom.regex", "github-actions"],
+  "flux": {
+    "managerFilePatterns": [
+      "/(^|/)platform/.+\\.ya?ml$/",
+      "/(^|/)apps/.+\\.ya?ml$/"
+    ]
+  },
+  "kustomize": {
+    "managerFilePatterns": [
+      "/(^|/)kustomization\\.ya?ml$/"
+    ]
+  },
+  // ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes
+  // manifests, and it is NOT auto-detected — it only looks at files matched here, which is why
+  // this block is as broad as it is. Scope covers every place this repo writes a container image:
+  // platform components, the rendered app waves, the ServiceDefinition catalog that generates
+  // them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as
+  // `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the
+  // rendered output is reverted by the next render and would also fail ci/render-golden.sh.
+  // Renovate will raise BOTH, and both must land together.
+  "kubernetes": {
+    "managerFilePatterns": [
+      "/(^|/)platform/.+\\.ya?ml$/",
+      "/(^|/)platform-config/.+\\.ya?ml$/",
+      "/(^|/)apps/.+\\.ya?ml$/",
+      "/(^|/)catalog/.+\\.ya?ml$/",
+      "/(^|/)runbooks/serving/.+\\.ya?ml$/",
+      "/(^|/)infra/agent/.+\\.ya?ml$/"
+    ]
+  },
+  "customManagers": [
+    {
+      "customType": "regex",
+      "description": "Inline `# renovate: datasource=... depName=...` markers on pinned URLs / version literals (e.g. platform/kubevirt/kustomization.yaml release URLs, ci/versions.env, infra/devcontainer/versions.env).",
+      "managerFilePatterns": [
+        "/(^|/)kustomization\\.ya?ml$/",
+        "/(^|/)versions\\.env$/"
+      ],
+      "matchStrings": [
+        "#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n.*?(?<currentValue>v?[0-9]+\\.[0-9]+\\.[0-9]+)"
+      ],
+      "versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
+    },
+    // ADDED 2026-10-29 — THIS REPO'S OWN IMAGES (toolbox, agent-runner, conformance-runner).
+    // CI publishes them (.gitea/workflows/images.yaml) under an immutable `sha-<commit>` tag and a
+    // moving `main` tag, and never writes to git. Every reference to them is DIGEST-ONLY
+    // (`<repo>@sha256:<hex>`, no tag), because the admission policies
+    // (platform/kyverno/admission/pm-agent-job-shape.yaml, conformance-trigger-job-shape.yaml)
+    // admit exactly that form. A digest alone gives Renovate nothing to follow, so the tag to
+    // follow rides in a marker comment on the line ABOVE the reference:
+    //
+    //   # renovate-digest: image=<registry>/<namespace>/<image> tag=main
+    //   <anything>sha256:<64 hex>
+    //
+    // `main` is not a version, so Renovate skips the release lookup and proposes a `digest`
+    // update whenever the tag's digest moves (lib/workers/repository/process/lookup/index.ts:
+    // an unversioned currentValue WITH a currentDigest yields updateType `digest`). The
+    // `sha256:` prefix is INSIDE currentDigest on purpose: Renovate replaces that capture
+    // verbatim with the new digest, which carries the prefix.
+    //
+    // DELIBERATELY NOT TRACKED: infra/agent/worker-job-template.yaml. It is baked INTO
+    // agent-runner, so bumping it would rebuild agent-runner and loop forever; pm-dispatch
+    // overwrites that field from PM_WORKER_IMAGE at runtime anyway.
+    // REGISTRY AUTH: the Gitea registry answers 401 to anonymous reads, and the platform token is
+    // scoped to hostType `gitea`, which a docker lookup never matches
+    // (lib/util/host-rules.ts find). renovate.yaml supplies a hostType `docker` rule.
+    {
+      "customType": "regex",
+      "description": "This repo's own CI-published images, pinned by digest, following the tag named in a `# renovate-digest:` marker on the previous line.",
+      "managerFilePatterns": [
+        "/(^|/)versions\\.env$/",
+        "/(^|/)platform/pm-agent/.+\\.ya?ml$/"
+      ],
+      "matchStrings": [
+        "#\\s*renovate-digest:\\s*image=(?<depName>\\S+)\\s+tag=(?<currentValue>\\S+)\\s*\\n[^\\n]*?(?<currentDigest>sha256:[a-f0-9]{64})"
+      ],
+      "datasourceTemplate": "docker",
+      "versioningTemplate": "docker"
+    }
+  ],
+  "packageRules": [
+    {
+      "description": "Auto-merge the allowed low-risk classes on green CI \u2014 no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.",
+      "matchUpdateTypes": [
+        "minor",
+        "patch",
+        "digest"
+      ],
+      "automerge": true,
+      "automergeType": "pr",
+      "platformAutomerge": true
+    },
+    {
+      "description": "MAJOR / potentially-breaking bumps require human review (docs/11 safety rail). Never auto-merged.",
+      "matchUpdateTypes": [
+        "major"
+      ],
+      "automerge": false,
+      "addLabels": [
+        "needs-review",
+        "breaking"
+      ]
+    },
+    {
+      "description": "Keep the KubeVirt + CDI operator pair on a compatible track \u2014 grouped so they bump together (platform/kubevirt/kustomization.yaml).",
+      "matchDepNames": [
+        "kubevirt/kubevirt",
+        "kubevirt/containerized-data-importer"
+      ],
+      "groupName": "kubevirt-cdi"
+    },
+    {
+      "description": "Pinned tool versions (ci/versions.env, infra/devcontainer/versions.env) are a deliberate change surface \u2014 label but do not auto-merge; a version bump there changes CI reproducibility.",
+      "matchFileNames": [
+        "ci/versions.env",
+        "infra/devcontainer/versions.env"
+      ],
+      "automerge": false,
+      "addLabels": [
+        "ci-tooling"
+      ]
+    },
+    {
+      "description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check \u2014 it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
+      "matchPackageNames": [
+        "kured",
+        "longhorn",
+        "cilium",
+        "cloudnative-pg",
+        "kube-prometheus-stack"
+      ],
+      "automerge": false,
+      "addLabels": [
+        "needs-review",
+        "blast-radius"
+      ]
+    },
+    {
+      "description": "ADDED 2026-10-29. This repo's own images are followed ONLY through the `# renovate-digest:` markers above. The `kubernetes` manager also sees platform/pm-agent/deployment.yaml and infra/agent/worker-job-template.yaml, where the image has a digest and no tag; left on, it would look those up as `latest` (a tag CI never pushes) and, for the template, re-open the self-referential loop the custom manager deliberately avoids.",
+      "matchManagers": ["kubernetes"],
+      "matchPackageNames": [
+        "gitea.home.keithjackson.dev/home-network/toolbox",
+        "gitea.home.keithjackson.dev/home-network/agent-runner",
+        "gitea.home.keithjackson.dev/home-network/conformance-runner"
+      ],
+      "enabled": false
+    },
+    {
+      "description": "ADDED 2026-10-29. Digest bumps for this repo's own images: the image was built and verified by CI before it was published, so these auto-merge on a green gate like any digest (rule 1). Labelled so they are easy to find. NOTE: the conformance-runner bump edits conformance/versions.env, a sensitive path (infra/gitea/approval-policy.json), so it will wait for an administrator's /approve-override regardless of this rule.",
+      "matchManagers": ["custom.regex"],
+      "matchPackageNames": [
+        "gitea.home.keithjackson.dev/home-network/toolbox",
+        "gitea.home.keithjackson.dev/home-network/agent-runner",
+        "gitea.home.keithjackson.dev/home-network/conformance-runner"
+      ],
+      "addLabels": ["own-image"]
+    }
+  ],
+  // WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A
+  // FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it
+  // and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest,
+  // lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak
+  // (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete
+  // version before any automation can maintain them, and until then they drift on every pull with
+  // nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12,
+  // e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it
+  // has been told not to leave; raising those ceilings is deliberate per-component work and is
+  // NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.
+  // platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest`
+  // and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed
+  // shape, not software to keep current. `conformance/**` was already ignored for the same
+  // reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise
+  // have opened.
+  // conformance/** NARROWED 2026-10-29 to its subdirectories: ignorePaths applies to custom
+  // managers too (lib/workers/repository/extract/file-match.ts getMatchingFiles ->
+  // filterIgnoredFiles), so the blanket glob also hid conformance/versions.env, which now carries
+  // the conformance-runner digest Renovate must maintain. Everything else under conformance/
+  // stays ignored, for the original reason (test fixtures with deliberately fixed shapes).
+  "ignorePaths": [
+    "**/*.example",
+    "**/*.sops.yaml",
+    "conformance/config/**",
+    "conformance/incluster/**",
+    "conformance/lib/**",
+    "conformance/requirements/**",
+    "conformance/runs/**",
+    "conformance/tests/**",
+    "platform/image-automation/per-service/**",
+    "platform/kyverno/tests/**"
+  ],
+  // The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were
+  // BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation
+  // over a running tag. That controller was never enabled and never will be here (see _comment),
+  // so the only thing those two flags were achieving was leaving every plain `image:` tag in the
+  // repo unmanaged. `docker` is left out of enabledManagers because this repo contains no
+  // Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.
+  "docker": {
+    "enabled": true
+  }
+}
@@ -67,10 +67,10 @@ Work through these in order; the first three are by far the most likely.
 | Symptom in the log | Cause | Fix |
 |---|---|---|
 | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
-| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first |
+| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been published; the committed digest is the deliberate placeholder | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
 | `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
 | `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
-| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
+| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | restore the digest form; the digest comes from CI's publish (`.gitea/workflows/images.yaml` job summary) and Renovate's bump PR, never from a hand push |
 | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
 | `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |
 | `at the worker cap` / `worker limit reached` | the per-cycle cap, draining a backlog one issue at a time | expected; it will continue next cycle |
Author	SHA1	Message	Date
jakman85andClaude Opus 5.5	65a7769f55	fix(images): name the measured runner blocker; the job container gets no Docker daemon gate / gate (pull_request) Failing after 1m57s Details images / images (pull_request) Failing after 6s Details gate / conformance (pull_request) Has been skipped Details PR #81's first images run failed in 6s: act_runner 0.2.11 forces docker_host '-' for a tcp:// DOCKER_HOST, and gitea/act drops every bind when valid_volumes is empty. Giving jobs the privileged DinD daemon is node-root for any workflow author, so it is left to the operator; options are in the handoff entry. The driver's failure now names the cause. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-10-29 21:44:38 -05:00
jakman85andClaude Opus 5.5	0da34ddbc6	feat(images): build, verify and publish the three images in CI; Renovate carries the digests gate / gate (pull_request) Failing after 1m56s Details images / images (pull_request) Failing after 6s Details gate / conformance (pull_request) Has been skipped Details Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a workstation build/push. Nothing had ever been pushed to the Gitea registry. - .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as sha-<commit> + main, reading the digest back from the registry. Only changed images build. - Each image's own build-image.sh does the build and in-image verification, now engine-neutral (DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh. - Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex manager for them. The worker Job template is deliberately untracked (it would loop). - renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION. - Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-10-29 21:34:06 -05:00
jakman85	1301af9a4c	Merge pull request 'fix(renovate): the config was INVALID, so the first manual run errored out' (#79 ) from fix/renovate-invalid-config into main gate / gate (push) Successful in 1m55s Details gate / conformance (push) Has been skipped Details renovate / renovate (push) Failing after 9s Details Reviewed-on: #79 Reviewed-by: Keith Jackson <[email protected]>	2026-10-18 14:29:34 +00:00
jakman85	42ce001d71	Merge pull request 'fix(renovate): the cron fired outside Renovate's own window, so it would never open a PR' (#77 ) from fix/renovate-cron-window into main gate / conformance (push) Has been cancelled Details gate / gate (push) Has been cancelled Details Reviewed-on: #77 Reviewed-by: Keith Jackson <[email protected]>	2026-10-18 14:28:59 +00:00
jakman85	84bb0f0acd	Merge pull request 'fix(pihole): make the split-horizon answer cacheable (local-ttl=60)' (#78 ) from fix/pihole-local-ttl into main gate / gate (push) Successful in 1m53s Details gate / conformance (push) Has been skipped Details Reviewed-on: #78 Reviewed-by: Keith Jackson <[email protected]>	2026-10-18 08:19:28 +00:00
jakman85andClaude Opus 5	0d4bd7297d	fix(renovate): the cron fired outside Renovate's own window, so it would never open a PR gate / conformance (pull_request) Has been skipped Details gate / gate (pull_request) Successful in 1m55s Details .gitea/workflows/renovate.yaml shipped with `cron: "15 3 * * "`. Gitea Actions cron is UTC, and renovate.json carries schedule ["after 3am and before 6am"] with timezone America/New_York -- and THAT inner schedule is authoritative: Renovate re-checks it and declines to open PRs outside it. 03:15 UTC -> 23:15 EDT, the PREVIOUS EVENING in New York window -> 03:00-06:00 local => NOT INSIDE (3h45m early) The original comment claimed the cron sat "slightly ahead" of the window. It sat 3h45m behind it, on the previous day. Every nightly run would have started, logged "not within schedule", opened ZERO PRs and exited SUCCESSFULLY -- a green tick forever, indistinguishable from "nothing needed updating", hiding the ~30 pending updates PR #73 existed to surface. Changed to `15 8 * *`, chosen to survive DST rather than to look tidy: EDT (UTC-4): local 03:00-06:00 == 07:00-10:00 UTC EST (UTC-5): local 03:00-06:00 == 08:00-11:00 UTC 08:15 UTC is inside BOTH (04:15 EDT / 03:15 EST), verified by computation at both extremes, so neither switchover needs an edit. A cron cannot follow DST. Also confirmed while here: RENOVATE_TOKEN now exists as a repo Actions secret, so the workflow's fail-hard preflight will pass. STILL UNRESOLVED and the likelier remaining bug: renovate.json's `enabledManagers` lists bare `regex`, where modern Renovate names the custom manager `custom.regex`. If the bare name is ignored, every inline `# renovate:` marker in the tree does nothing. Deliberately NOT changed -- guessing at a key that could fail validation and take the whole run down is worse than a question one dry run answers. See the handoff. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-18 03:12:28 -05:00
jakman85andClaude Opus 5	713b0c3dba	fix(renovate): the config was INVALID, so the first manual run errored out gate / gate (pull_request) Successful in 1m55s Details gate / conformance (pull_request) Has been skipped Details The operator ran the Renovate workflow manually and it failed. The cause is not the workflow: renovate.json was never a valid Renovate config. It carried five pseudo-comment keys (_comment, _kubernetes_comment, _unmanageable_comment, _ignorePaths_comment, _docker_comment) because JSON has no comments, and Renovate REJECTS unknown top-level options. Proven with the official validator: ERROR: Found errors in configuration Invalid configuration option: _comment Invalid configuration option: _docker_comment Invalid configuration option: _ignorePaths_comment Invalid configuration option: _kubernetes_comment Invalid configuration option: _unmanageable_comment This is the SECOND independent reason Renovate never worked, on top of PR #77's cron firing outside renovate.json's own schedule window. Either alone was enough to produce "nothing ever happens"; #73 shipped both. Converted renovate.json -> renovate.json5. JSON5 permits real // comments and Renovate discovers renovate.json5 natively; nothing in .gitea/ or ci/ pins the old filename and no RENOVATE_CONFIG_FILE is set. Every word of the original rationale is preserved as positioned comments (74 comment lines) rather than flattened into a description string. Verified the conversion dropped nothing: 18 non-comment keys before and after, no missing/extra keys, no changed values, docker: {enabled: true} still present. The validator now returns "Config validated successfully". DISPROVED while here: `enabledManagers` containing bare `regex` was suspected of needing to be `custom.regex` (which would silently disable every inline `# renovate:` marker). The validator accepts `regex`; unfounded, unchanged. Renovate has still never completed a run -- this fixes one of two blockers and nothing in the gate catches an invalid config. See the handoff. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-18 03:10:25 -05:00
jakman85andClaude Opus 5	fda5c31a94	fix(pihole): make the split-horizon answer cacheable, or one blip latches the WAN IP for 5 minutes gate / gate (pull_request) Successful in 1m56s Details gate / conformance (pull_request) Has been skipped Details Operator reported DNS flapping between the cluster ingress and the router's public IP. PiHole is not at fault -- both replicas answer 192.168.2.20 consistently (6/6 via the VIP, 8/8 from a client). The bug is an asymmetry in CACHEABILITY that we created. Measured: pihole 192.168.2.23 -> 192.168.2.20 TTL=0 public 1.1.1.1 -> wan.home.keithjackson.dev TTL=300 router 192.168.2.1 -> wan.... TTL=153 -> 148 -> 143 (decrementing = caching) dnsmasq answers `address=` entries with TTL 0 unless told otherwise. The router forwards LAN queries to PiHole and caches the result, so the CORRECT answer is uncacheable -- re-queried on every lookup, maximum exposure to any gap -- while the WRONG answer caches for 300s. The instant the router falls back to upstream (standard forwarder behaviour when the target does not answer) it latches the WAN address for the whole LAN for five minutes, then expires, re-queries PiHole, gets TTL 0, and is immediately re-exposed. So ONE sub-second PiHole gap -- pod restart, L2 lease move, ARP flap, kured reboot -- becomes FIVE MINUTES of the house resolving to the WAN IP. That asymmetry, not the gap, is why it reads as "flapping". Fix: local-ttl=60 prepended to FTLCONF_misc_dnsmasq_lines. dnsmasq applies local-ttl to replies made from configuration, so one directive covers all five address= names. 60s rides out a pod restart and a lease re-election while a real VIP change still propagates inside a minute; these VIPs are pinned and static. apps/wave-0/pihole.yaml regenerated with the canonical render command (--all catalog -o apps --wave-layout --no-image-automation, NOT --emit-waves); exactly two files changed and the rendered diff is the single local-ttl prefix. UNAPPLIED. Does NOT remove the trigger: .23 still has two independently-elected L2 leases (as do crafty .22 and unifi .21), which the operator has parked. If the TTL is still 0 after merge, suspect FTL silently rejecting the passthrough line -- see the handoff. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-18 02:56:49 -05:00
jakman85	c340bbfc4b	Merge pull request 'fix(conformance): one regex made the merge gate take 28 minutes instead of 2' (#76 ) from fix/ha-check-catastrophic-regex into main gate / gate (push) Successful in 1m56s Details gate / conformance (push) Has been skipped Details Reviewed-on: #76 Reviewed-by: Keith Jackson <[email protected]>	2026-10-18 07:51:51 +00:00
jakman85	a9a529750d	Merge pull request 'fix(unattended-upgrades): Debian has no reboot-sentinel package, so ship the writer' (#75 ) from fix/unattended-upgrades-debian-sentinel into main gate / conformance (push) Has been cancelled Details gate / gate (push) Has been cancelled Details Reviewed-on: #75	2026-10-18 07:40:20 +00:00
jakman85andClaude Opus 5	ef8ef5ac29	fix(unattended-upgrades): Debian has no reboot-sentinel package, so ship the writer gate / gate (pull_request) Successful in 27m51s Details gate / conformance (pull_request) Has been skipped Details The first real run failed on all three nodes: fatal: [k3s-prod-1]: "No package matching 'update-notifier-common' is available" update-notifier-common is an UBUNTU package and does not exist in Debian -- `apt-cache policy` returns Candidate: (none) on trixie. The role asserted it "owns the hook that creates /var/run/reboot-required", which is true on Ubuntu and false here, so the role could never have worked. Three further measurements settled the replacement: notify-reboot-required is absent; NO linux-image-* postinst references it (invoking it is an Ubuntu patch, so dropping a script at that path would never be called); and no hook under apt.conf.d or /etc/kernel mentions reboot-required. Debian trixie ships no mechanism at all. So the role now provides it: * /usr/local/sbin/k3s-notify-reboot-required -- touches the sentinel when the running kernel is not the newest installed, using `linux-version` (from linux-base, a dependency of every linux-image, measured present) because kernel version ordering is not string ordering. * /etc/apt/apt.conf.d/99-k3s-reboot-required -- DPkg::Post-Invoke, `\|\| true` so it can never abort an apt transaction. /run is tmpfs (measured), so the sentinel self-clears on boot; no cleanup task. The verify assert was ALSO weaker than it looked: a `find` glob for update-notifier / unattended-upgrades under apt.conf.d would have matched Debian's own unattended-upgrades conffile and reported "writer present" while none existed. It now stats the two exact paths this role owns and requires the script to be executable. Newly verified: the script ran on prod-1 as unprivileged k3s under sh -x -- running=newest=7.1.8+deb13-amd64, took the equal branch, exited 0, wrote nothing. Correct. sh -n and bash -n clean. Still unverified: the WRITING half never executes while running == newest, so the sentinel has not been seen created and kured has not been seen react. The honest test is the next kernel upgrade. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-18 01:43:43 -05:00