feat(images): build, verify and publish the three images in CI; Renovate carries the digests

Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a workstation build/push. Nothing had ever been pushed to the Gitea registry. - .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as sha-<commit> + main, reading the digest back from the registry. Only changed images build. - Each image's own build-image.sh does the build and in-image verification, now engine-neutral (DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh. - Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex manager for them. The worker Job template is deliberately untracked (it would loop). - renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION. - Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh. Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 21:34:06 -05:00
1 parent 1301af9a4c
commit 0da34ddbc6
@@ -115,14 +115,47 @@ CI. That is a real coverage gap, not a phase.
 | variable | `REQUIRE_CLUSTER_TESTS` | `'true'` (Phase 1+) makes unverified cluster-needing touched reqs a hard block |
 | secret | `CONFORMANCE_TRIGGER_KUBECONFIG` | narrow creds to create/await the in-cluster Chainsaw Job (not cluster-admin) |

+## How images are built
+
+This repo's three images — `toolbox` (`infra/devcontainer/`), `agent-runner` (`infra/agent/`,
+built `FROM` the toolbox) and `conformance-runner` (`conformance/incluster/`) — are built,
+verified and published by [`images.yaml`](images.yaml), never from a workstation
+(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh); each
+image is built by **its own** `build-image.sh`, the same script a developer runs locally, so the
+in-image verification (non-root uid, read-only rootfs, tool presence, `PM_SELFTEST=1
+pm-dispatch`, the `conformance-clone` refusals) exists once.
+
+| Trigger | What happens |
+|---|---|
+| pull request touching an image's inputs | build + verify only. Nothing is pushed and **no registry credential is referenced** |
+| push to `main` touching an image's inputs | build + verify + publish to `gitea.home.keithjackson.dev/home-network/<image>` as `sha-<commit>` (immutable) and `main` (moving); the **registry-returned** digest is in the job summary |
+| `workflow_dispatch` | the same as a push when run on `main` (all images by default); verify-only elsewhere |
+
+Only images whose inputs changed are built. The publish job **never writes to git**: Renovate
+follows each image's `main` tag through a `# renovate-digest:` marker next to every digest
+reference (`platform/pm-agent/`, `infra/agent/versions.env`, `conformance/versions.env`) and
+opens the bump PR, which goes through the gate like anything else. A PR opened with the Actions
+token would not trigger the required gate, which is why it is Renovate's job and not this one's.
+
+Credential: the repository secret **`REGISTRY_PUSH_TOKEN`** (scope `write:package`, owned by an
+account with package write on the org). The automatic Actions token cannot push in Gitea 1.24.3
+— the reasoning, with source citations, is in the header of `images.yaml`. Retention:
+[`infra/gitea/registry-retention.sh`](../../infra/gitea/registry-retention.sh), an operator
+action, keeps every digest git pins plus the newest tags.
+
+Cost on this runner: the job shares the single act_runner slot with the gate, and the DinD store
+is a node `emptyDir`; the job prunes it on exit.
+
 ## Files

 ```
 .gitea/workflows/
  merge-gate.yaml            the pipeline (lint → l0 → l1 → conformance → decision)
+  images.yaml                build/verify/publish this repo's container images (see above)
  scripts/
    install-tools.sh         installs the pinned tools per stage (from ci/versions.env)
    decide.sh                the R-TEST-03 merge decision over job results
+    images.sh                the image pipeline's driver (what to build, base, push, digest)
  README.md                  this file
 ```

@@ -0,0 +1,174 @@
+# Container images — built, verified and published by CI, never from a workstation.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# WHY THIS FILE EXISTS (operator rule, 2026-10-29)
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# "Container image creation, hosting and lifecycle must be part of CI/CD and the codebase, never a
+# one-off build/push from a workstation." Before this workflow, three images existed only as
+# workstation builds and NOTHING had ever been pushed to the Gitea registry: the Home-Network org's
+# package list was empty, platform/pm-agent/ pinned an all-zero digest and sat in ImagePullBackOff,
+# and conformance/versions.env carried an empty digest. Both build scripts said "publishing is an
+# operator action" — a step with no log, no recorded inputs and no verification anyone could read.
+#
+# WHAT IT DOES — the logic is in scripts/images.sh; this file only wires triggers and secrets.
+#   pull_request -> build + VERIFY only. No push, and no registry credential is referenced.
+#   push to main -> build + verify + PUBLISH to gitea.home.keithjackson.dev/home-network/<image>
+#   dispatch     -> same as a push when run on main; verify-only on any other ref.
+# Images: toolbox (infra/devcontainer), agent-runner (infra/agent, FROM toolbox) and
+# conformance-runner (conformance/incluster). Only images whose inputs changed are built.
+#
+# THE DIGEST LIFECYCLE — this workflow NEVER writes to git.
+#   1. publish pushes `sha-<commit>` (immutable) and `main` (moving), and prints the digest the
+#      REGISTRY returns in the job summary.
+#   2. Renovate (renovate.yaml) follows the `main` tag of each image through the
+#      `# renovate-digest:` markers next to every digest reference, and opens the bump PR.
+#   3. That PR goes through the gate like any other change, and Flux deploys it.
+# The bump PR is Renovate's and not this job's because a PR opened with the Actions token does not
+# trigger workflows, so it could never pass the REQUIRED gate check; Renovate's token is a real
+# user's.
+#
+# THE PUSH CREDENTIAL — `REGISTRY_PUSH_TOKEN`, a repository Actions secret, referenced BY NAME.
+# The automatic Actions token CANNOT be used. Gitea 1.24.3 source, verified:
+#   * services/auth/basic.go (Basic.Verify) turns a task token into the pseudo-user
+#     `gitea-actions` (ActionsUserID -2) and sets NO token scope;
+#   * routers/api/packages/container/container.go (Authenticate, the /v2/token exchange every
+#     docker client uses) then requires read:package / write:package / all scope and answers 401;
+#   * services/context/package.go (determineAccessMode) has `// TODO: ActionUser permission check`
+#     and no branch for that user, so even past login it would get at most READ on a public org.
+# So: a personal access token with scope `write:package`, minted by an account that has package
+# WRITE on the Home-Network org (org owner, or a team with the Packages unit at write). A dedicated
+# bot account in a packages-only team is the least-privilege choice. The username sent with it is
+# not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER`
+# is optional and only labels the login.
+#
+# THE RUNNER (apps/wave-1/gitea-actions.yaml): act_runner 0.2.11, one job at a time, Docker
+# executor on a privileged `docker:27-dind` sidecar whose image store is an emptyDir on the node
+# (no sizeLimit) and whose memory limit is 2Gi — shared by the job container AND every build
+# container. The toolbox is ~1.4 GB uncompressed and agent-runner adds ~0.3 GB on top of it, so the
+# cleanup step below is not optional: without it each run would leave ~3 GB on a node that had
+# 7 GB free when this was written, and the next eviction would take the runner with it.
+name: images
+
+on:
+  pull_request:
+    branches: [main]
+    paths:
+      - infra/devcontainer/Containerfile
+      - infra/devcontainer/versions.env
+      - infra/devcontainer/build-image.sh
+      - infra/agent/**
+      - conformance/incluster/Containerfile
+      - conformance/incluster/conformance-clone
+      - conformance/incluster/build-image.sh
+      - .gitea/workflows/images.yaml
+      - .gitea/workflows/scripts/images.sh
+  push:
+    branches: [main]
+    paths:
+      - infra/devcontainer/Containerfile
+      - infra/devcontainer/versions.env
+      - infra/devcontainer/build-image.sh
+      - infra/agent/**
+      - conformance/incluster/Containerfile
+      - conformance/incluster/conformance-clone
+      - conformance/incluster/build-image.sh
+      - .gitea/workflows/images.yaml
+      - .gitea/workflows/scripts/images.sh
+  workflow_dispatch:
+    inputs:
+      images:
+        description: "Which images: 'all', or a space-separated subset of: toolbox agent-runner conformance-runner"
+        required: false
+        default: "all"
+
+concurrency:
+  # Never cancel a publish half-way: a pushed `sha-` tag with no matching `main` move is
+  # recoverable (a rerun skips what is already published) but pointless to provoke.
+  group: images-${{ github.ref }}
+  cancel-in-progress: false
+
+jobs:
+  images:
+    runs-on: ubuntu-latest
+    # Measured on the PR that introduced this file — see its decisions entry. Generous because a
+    # cold DinD store downloads the Fedora base and every pinned tool on each run.
+    timeout-minutes: 75
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0   # the per-image change detection diffs against the base commit
+
+      # ── pull requests: build + verify, NO credential ───────────────────────────────────────
+      - name: Build and verify (pull request — nothing is pushed)
+        if: github.event_name == 'pull_request'
+        env:
+          MODE: verify
+          IMAGES: auto
+          PR_BASE_REF: ${{ github.base_ref }}
+        run: |
+          set -euo pipefail
+          BASE_SHA="$(git merge-base "origin/${PR_BASE_REF}" HEAD)"
+          export BASE_SHA
+          ./.gitea/workflows/scripts/images.sh
+
+      # ── non-main dispatch: verify only ──────────────────────────────────────────────────────
+      - name: Build and verify (dispatch off main — nothing is pushed)
+        if: github.event_name == 'workflow_dispatch' && github.ref != 'refs/heads/main'
+        env:
+          MODE: verify
+          IMAGES: ${{ inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images }}
+        run: ./.gitea/workflows/scripts/images.sh
+
+      # ── main: publish ───────────────────────────────────────────────────────────────────────
+      - name: Preflight - the push token must exist, and its absence must be LOUD
+        if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
+        env:
+          REGISTRY_PUSH_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
+        run: |
+          set -euo pipefail
+          if [ -z "${REGISTRY_PUSH_TOKEN:-}" ]; then
+            echo "::error::REGISTRY_PUSH_TOKEN is not set on this repository." >&2
+            cat >&2 <<'MSG'
+
+          Images cannot be published without a registry credential, and this step fails rather
+          than skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would
+          make "nothing was published" look exactly like "published". That is how this repo went
+          a month with images that existed only on one workstation.
+
+          WHY NOT THE BUILT-IN ACTIONS TOKEN: Gitea 1.24.3 gives it no package scope at the
+          registry's /v2/token exchange (routers/api/packages/container/container.go,
+          Authenticate) — see the header of .gitea/workflows/images.yaml.
+
+          TO FIX: as an account with package WRITE on the Home-Network org (ideally a dedicated
+          bot in a packages-only team), create an access token with scope `write:package` and
+          nothing else, then add it to this repository as the Actions secret
+          REGISTRY_PUSH_TOKEN. Optionally set the Actions variable REGISTRY_PUSH_USER to that
+          account's name. Do not commit the token anywhere. Then re-run this workflow
+          (workflow_dispatch on main, images: all).
+          MSG
+            exit 1
+          fi
+          echo "REGISTRY_PUSH_TOKEN is present (value not printed)."
+
+      - name: Build, verify and publish (main)
+        if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
+        env:
+          MODE: publish
+          IMAGES: ${{ github.event_name == 'workflow_dispatch' && (inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images) || 'auto' }}
+          BASE_SHA: ${{ github.event.before }}
+          REGISTRY_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
+          REGISTRY_USER: ${{ vars.REGISTRY_PUSH_USER }}
+        run: ./.gitea/workflows/scripts/images.sh
+
+      # ── always: give the DinD store back ────────────────────────────────────────────────────
+      # The runner's image store is a node emptyDir with no sizeLimit (see header). Everything this
+      # job built or pulled is removed; `prune -a` cannot remove the job's own image, because the
+      # job container is still running on it.
+      - name: Clean the DinD image store
+        if: always()
+        run: |
+          docker logout gitea.home.keithjackson.dev >/dev/null 2>&1 || true
+          docker image prune -af >/dev/null || true
+          docker builder prune -af >/dev/null || true
+          docker system df || true
@@ -60,8 +60,9 @@
 # entry's "What I did NOT verify" — that gap is stated, not papered over.
 #
 # OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE:
-#   `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository` and
-#   `read:user`. Set it as an Actions secret on this repository. Nothing in this repo mints,
+#   `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository`,
+#   `read:user` and (since 2026-10-29) `read:package`. Set it as an Actions secret on this
+#   repository. Nothing in this repo mints,
 #   stores or references its value; it is consumed by name only.
 #   Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a
 #   visible red is the correct state for "automation configured but not credentialed".
@@ -139,8 +140,9 @@ jobs:
          seven files claimed "Renovate maintains this pin".

          TO FIX: create a Gitea access token for the bot account with scopes
-          `write:repository` and `read:user`, then add it to this repository as the Actions
-          secret RENOVATE_TOKEN. Do not commit it anywhere.
+          `write:repository`, `read:user` and `read:package` (the last one lets Renovate read
+          this repo's own images in the Gitea registry), then add it to this repository as the
+          Actions secret RENOVATE_TOKEN. Do not commit it anywhere.
          MSG
            exit 1
          fi
@@ -154,6 +156,12 @@ jobs:
          # This is the one dependency in the repo whose updater is its own updater.
          configurationFile: renovate.json
          token: ${{ secrets.RENOVATE_TOKEN }}
+          # PINNED 2026-10-29. Unset, this action version runs `ghcr.io/renovatebot/renovate:39`
+          # (its action.yml default), i.e. 39.264.0 — and renovate.json5 is written for, and
+          # validated against, a release that knows `managerFilePatterns` (>= 40.2.0). Under 39.x
+          # the config fails validation with five errors and the run stops at CONFIG_VALIDATION.
+          # Validate renovate.json5 with THIS tag; see its header for the command.
+          renovate-version: 44.167.0
        env:
          # Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
          # header said were "set at runtime by the operator's renovate runner" — this is that
@@ -170,6 +178,14 @@ jobs:
          RENOVATE_REQUIRE_CONFIG: "required"
          RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }}
          LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }}
+          # ADDED 2026-10-29: registry read access for this repo's OWN images (toolbox,
+          # agent-runner, conformance-runner — published by .gitea/workflows/images.yaml and
+          # followed through `# renovate-digest:` markers). The Gitea registry answers 401 to
+          # anonymous reads, and Renovate scopes the platform token to hostType `gitea`, which a
+          # docker lookup never matches — so without this rule every digest lookup fails. Same
+          # token; it therefore needs scope `read:package` in addition to the ones listed in the
+          # preflight above. Gitea ignores the username for token auth (services/auth/basic.go).
+          RENOVATE_HOST_RULES: '[{"hostType":"docker","matchHost":"gitea.home.keithjackson.dev","username":"renovate","password":"${{ secrets.RENOVATE_TOKEN }}"}]'

      - name: Summary
        if: always()
@@ -0,0 +1,291 @@
+#!/usr/bin/env bash
+###############################################################################
+# .gitea/workflows/scripts/images.sh — the image pipeline's driver (images.yaml calls this).
+#
+# Builds, verifies and (on main) PUBLISHES the three images this repo ships:
+#
+#   toolbox             infra/devcontainer/   the pinned dev/gate toolchain
+#   agent-runner        infra/agent/          FROM toolbox; the PM dispatcher + worker
+#   conformance-runner  conformance/incluster/ the in-cluster L2/L3 Chainsaw runner
+#
+# Each image is built by ITS OWN build-image.sh — the same script a developer runs locally —
+# so the in-image verification (non-root uid, read-only rootfs, tool presence, pm-dispatch
+# selftest, conformance-clone refusals, ...) is written once and runs in both places. This
+# file only decides WHAT to build, in which order, FROM which base, and where it goes.
+#
+# MODES (env MODE):
+#   verify   build + verify, never push, holds NO credential. Pull requests.
+#   publish  build + verify + push to ${REGISTRY}/${NAMESPACE}/<image> with tags
+#            `sha-<12 hex>` (immutable: never re-pushed) and `main` (moving; Renovate follows
+#            it). The digest recorded is the one the REGISTRY returns, read back after the push.
+#
+# WHAT GETS BUILT (env IMAGES): `auto` (default) diffs BASE_SHA..HEAD_SHA against each image's
+# inputs; otherwise a space-separated list. A change to the pipeline itself (images.yaml, this
+# file) builds everything in verify mode, and nothing extra in publish mode: re-publishing
+# unchanged inputs would only mint new digests and churn Renovate PRs.
+#
+# THE agent-runner BASE — the one ordering rule, and the reason this is not three jobs:
+#   * toolbox built in this run  -> agent-runner builds FROM it (in publish mode, FROM the
+#     digest the registry just returned for it).
+#   * verify mode, toolbox not changed -> the toolbox is built anyway as a base. A pull request
+#     holds no registry credential, so it cannot pull the published one.
+#   * publish mode, toolbox not changed -> AGENT_BASE_IMAGE from infra/agent/versions.env,
+#     i.e. the published toolbox digest that Renovate keeps current. If that is still the
+#     all-zero placeholder the toolbox has never been published, so it is built and published
+#     first (this is how the very first run bootstraps).
+#
+# LOOP-FREEDOM. Renovate turns each publish into a digest-bump PR. Those bumps land in
+# platform/pm-agent/ (no image input), conformance/versions.env (no image input) and
+# infra/agent/versions.env (agent-runner input -> ONE more agent-runner build, whose own bump
+# lands in platform/pm-agent/ and stops). The worker Job template is deliberately NOT tracked:
+# it is baked INTO agent-runner, so bumping it would rebuild agent-runner, mint a new digest,
+# bump it again, forever. pm-dispatch overwrites that field at runtime anyway.
+###############################################################################
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
+cd "${REPO_ROOT}"
+
+MODE="${MODE:-verify}"
+IMAGES="${IMAGES:-auto}"
+ENGINE="${ENGINE:-docker}"
+REGISTRY="${REGISTRY:-gitea.home.keithjackson.dev}"
+NAMESPACE="${NAMESPACE:-home-network}"
+HEAD_SHA="${HEAD_SHA:-$(git rev-parse HEAD)}"
+BASE_SHA="${BASE_SHA:-}"
+SUMMARY="${GITHUB_STEP_SUMMARY:-/dev/null}"
+ZERO_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000"
+
+ALL_IMAGES=(toolbox agent-runner conformance-runner)
+SHA_TAG="sha-${HEAD_SHA:0:12}"
+MOVING_TAG="main"
+
+log()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
+info() { printf '    %s\n' "$*"; }
+die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; printf -- '- **ERROR**: %s\n' "$*" >>"${SUMMARY}"; exit 1; }
+
+case "${MODE}" in verify|publish) ;; *) die "MODE must be verify or publish, got '${MODE}'" ;; esac
+
+# --- inputs per image: a change under any of these rebuilds that image ---------------------
+# conformance-runner takes its chainsaw/kubectl/fedora pins from infra/devcontainer/versions.env.
+# conformance/versions.env is NOT an input (it only RECORDS the published digest), and must not
+# become one: Renovate writes it after every publish, and an input there would loop.
+inputs() {
+  case "$1" in
+    toolbox)            echo "infra/devcontainer/Containerfile infra/devcontainer/versions.env infra/devcontainer/build-image.sh" ;;
+    agent-runner)       echo "infra/agent/" ;;
+    conformance-runner) echo "conformance/incluster/Containerfile conformance/incluster/conformance-clone conformance/incluster/build-image.sh infra/devcontainer/versions.env" ;;
+  esac
+}
+PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh"
+
+declare -A WANT=() BUILT=() DIGEST=() SECS=() NOTE=()
+
+changed_any() { # <paths...> -> 0 if BASE_SHA..HEAD_SHA touches any
+  # shellcheck disable=SC2086
+  [ -n "$(git diff --name-only "${BASE_SHA}" "${HEAD_SHA}" -- "$@")" ]
+}
+
+select_images() {
+  if [ "${IMAGES}" != "auto" ]; then
+    for i in ${IMAGES}; do
+      case " ${ALL_IMAGES[*]} " in *" ${i} "*) WANT[$i]=1; NOTE[$i]="requested" ;; *) die "unknown image '${i}'" ;; esac
+    done
+    return
+  fi
+  if [ -z "${BASE_SHA}" ] || [[ "${BASE_SHA}" =~ ^0+$ ]] || ! git cat-file -e "${BASE_SHA}^{commit}" 2>/dev/null; then
+    info "no usable diff base ('${BASE_SHA}'): building everything"
+    for i in "${ALL_IMAGES[@]}"; do WANT[$i]=1; NOTE[$i]="no diff base"; done
+    return
+  fi
+  info "diff base ${BASE_SHA} .. ${HEAD_SHA}"
+  local i
+  for i in "${ALL_IMAGES[@]}"; do
+    # shellcheck disable=SC2046
+    if changed_any $(inputs "$i"); then WANT[$i]=1; NOTE[$i]="inputs changed"; fi
+  done
+  # shellcheck disable=SC2086
+  if [ "${MODE}" = verify ] && changed_any ${PIPELINE_FILES}; then
+    for i in "${ALL_IMAGES[@]}"; do [ -n "${WANT[$i]:-}" ] || { WANT[$i]=1; NOTE[$i]="pipeline changed"; }; done
+  fi
+}
+
+# --- registry helpers (publish mode only) ----------------------------------------------------
+REG_TOKEN=""
+registry_token() {
+  [ -n "${REG_TOKEN}" ] && return 0
+  REG_TOKEN="$(curl -fsS --config <(printf 'user = "%s:%s"\n' "${REGISTRY_USER}" "${REGISTRY_TOKEN}") \
+    "https://${REGISTRY}/v2/token?service=container_registry&scope=*" | jq -r '.token // empty')" \
+    || die "could not obtain a registry bearer token from https://${REGISTRY}/v2/token"
+  [ -n "${REG_TOKEN}" ] || die "registry returned no bearer token"
+}
+# Prints the manifest digest the REGISTRY serves for <image>:<tag>; nothing if it is absent (404).
+# Any other answer (401, 5xx, no digest header) is fatal: "cannot tell" must never be read as
+# "not published", or the immutable-tag guard would be bypassed by an auth hiccup.
+registry_digest() {
+  registry_token
+  local hdrs code digest
+  hdrs="$(curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "${REG_TOKEN}") \
+    -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
+    -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
+    -H 'Accept: application/vnd.oci.image.index.v1+json' \
+    -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
+    "https://${REGISTRY}/v2/${NAMESPACE}/$1/manifests/$2" | tr -d '\r')" \
+    || die "registry unreachable while reading $1:$2"
+  code="$(awk 'NR==1{print $2}' <<<"${hdrs}")"
+  digest="$(awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' <<<"${hdrs}" | tail -1)"
+  case "${code}" in
+    200) [[ "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]] || die "registry answered 200 for $1:$2 with no usable digest"
+         echo "${digest}" ;;
+    404) ;;
+    *)   die "registry answered HTTP ${code} for $1:$2 — cannot tell whether it exists" ;;
+  esac
+}
+
+local_ref() { echo "${REGISTRY}/${NAMESPACE}/$1:ci-${HEAD_SHA:0:12}"; }
+
+build_one() { # <image> [agent base]
+  local img="$1" base="${2:-}" t0 tag
+  tag="ci-${HEAD_SHA:0:12}"
+  t0=$(date +%s)
+  log "BUILD + VERIFY ${img}"
+  case "${img}" in
+    toolbox)
+      ENGINE="${ENGINE}" IMAGE="$(local_ref toolbox)" BUILD_REVISION="${HEAD_SHA}" \
+        ./infra/devcontainer/build-image.sh ;;
+    agent-runner)
+      ENGINE="${ENGINE}" IMAGE_TAG="${tag}" AGENT_BASE_IMAGE="${base}" BUILD_REVISION="${HEAD_SHA}" \
+        ./infra/agent/build-image.sh ;;
+    conformance-runner)
+      ENGINE="${ENGINE}" IMAGE_TAG="${tag}" BUILD_REVISION="${HEAD_SHA}" \
+        ./conformance/incluster/build-image.sh ;;
+  esac
+  BUILT[$img]=1
+  SECS[$img]=$(( $(date +%s) - t0 ))
+  info "${img}: built and verified in ${SECS[$img]}s"
+}
+
+push_one() { # <image> -> sets DIGEST[image] to the registry-returned digest
+  local img="$1" repo src existing pushed got t0
+  repo="${REGISTRY}/${NAMESPACE}/${img}"
+  src="$(local_ref "${img}")"
+  t0=$(date +%s)
+  log "PUSH ${img} -> ${repo}:{${SHA_TAG},${MOVING_TAG}}"
+  existing="$(registry_digest "${img}" "${SHA_TAG}")"
+  if [ -n "${existing}" ]; then
+    # Immutable tag: never overwrite. A rebuild is not byte-reproducible, so re-pushing would
+    # silently change what `sha-<commit>` means after something may already have pinned it.
+    die "${repo}:${SHA_TAG} already exists (${existing}). Tags here are immutable; this run must not
+       replace it. (publish skips images that already exist at this commit BEFORE building them,
+       so reaching this line means the tag appeared mid-run.)"
+  fi
+  "${ENGINE}" tag "${src}" "${repo}:${SHA_TAG}"
+  "${ENGINE}" tag "${src}" "${repo}:${MOVING_TAG}"
+  pushed="$("${ENGINE}" push "${repo}:${SHA_TAG}" | tee /dev/stderr | grep -oE 'digest: sha256:[0-9a-f]{64}' | tail -1 | cut -d' ' -f2)"
+  "${ENGINE}" push "${repo}:${MOVING_TAG}" >/dev/null
+  # The AUTHORITATIVE digest is what the registry serves, read back — not what the client
+  # printed. They should agree; if they do not, nothing downstream may trust either.
+  got="$(registry_digest "${img}" "${SHA_TAG}")"
+  [ -n "${got}" ] || die "pushed ${repo}:${SHA_TAG} but the registry does not serve it"
+  [ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] \
+    || die "${img}: client reported ${pushed} but the registry serves ${got}"
+  [ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \
+    || die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})"
+  DIGEST[$img]="${got}"
+  SECS[$img]=$(( ${SECS[$img]:-0} + $(date +%s) - t0 ))
+  info "${img}: registry digest ${got}"
+}
+
+# --- main ---------------------------------------------------------------------------------
+log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}"
+command -v "${ENGINE}" >/dev/null || die "${ENGINE} is not on PATH in the job container"
+"${ENGINE}" info >/dev/null 2>&1 \
+  || die "the ${ENGINE} daemon is not reachable from the job container — the runner did not
+       hand it a Docker socket (act_runner container.docker_host). Nothing can be built."
+"${ENGINE}" version 2>/dev/null | sed -n '1,12p' || true
+"${ENGINE}" system df || true
+
+if [ "${MODE}" = publish ]; then
+  : "${REGISTRY_TOKEN:?REGISTRY_TOKEN is required in publish mode}"
+  REGISTRY_USER="${REGISTRY_USER:-ci-image-publisher}"
+  printf '%s' "${REGISTRY_TOKEN}" | "${ENGINE}" login "${REGISTRY}" -u "${REGISTRY_USER}" --password-stdin >/dev/null \
+    || die "docker login ${REGISTRY} failed — check REGISTRY_PUSH_TOKEN (scope write:package, owner has package write on the org)"
+fi
+
+select_images
+
+# Publish idempotence: an image already published at THIS commit is not rebuilt. This is what
+# makes a re-run after a half-finished publish safe — and it is the only way a rerun can
+# finish, because the immutable sha tag must never be pushed twice.
+if [ "${MODE}" = publish ]; then
+  for i in "${ALL_IMAGES[@]}"; do
+    [ -n "${WANT[$i]:-}" ] || continue
+    d="$(registry_digest "${i}" "${SHA_TAG}")"
+    if [ -n "${d}" ]; then
+      info "${i}: already published at ${SHA_TAG} (${d}); not rebuilding"
+      unset "WANT[$i]"; DIGEST[$i]="${d}"; NOTE[$i]="already published at this commit"
+    fi
+  done
+fi
+
+# Decide the agent-runner base (see header).
+AGENT_BASE=""
+if [ -n "${WANT[agent-runner]:-}" ]; then
+  # shellcheck disable=SC1091
+  PINNED_BASE="$(. ./infra/agent/versions.env; echo "${AGENT_BASE_IMAGE}")"
+  if [ "${MODE}" = verify ] && [ -z "${WANT[toolbox]:-}" ]; then
+    WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)"
+  elif [ "${MODE}" = publish ] && [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then
+    case "${PINNED_BASE}" in
+      *@"${ZERO_DIGEST}") WANT[toolbox]=1; NOTE[toolbox]="never published; required as agent-runner's base" ;;
+      *@sha256:*)         AGENT_BASE="${PINNED_BASE}" ;;
+      *) die "infra/agent/versions.env AGENT_BASE_IMAGE='${PINNED_BASE}' is not digest-pinned; CI will not build on a floating base" ;;
+    esac
+  fi
+fi
+
+yn() { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; }
+for i in "${ALL_IMAGES[@]}"; do info "plan: ${i}: $(yn "${WANT[$i]:-}" BUILD skip)  (${NOTE[$i]:-inputs unchanged})"; done
+
+if [ -n "${WANT[toolbox]:-}" ]; then
+  build_one toolbox
+  if [ "${MODE}" = publish ]; then push_one toolbox; fi
+fi
+if [ -n "${WANT[agent-runner]:-}" ]; then
+  if [ -z "${AGENT_BASE}" ]; then
+    if [ "${MODE}" = publish ]; then
+      AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}"
+    else
+      AGENT_BASE="$(local_ref toolbox)"
+    fi
+  fi
+  info "agent-runner base: ${AGENT_BASE}"
+  build_one agent-runner "${AGENT_BASE}"
+  if [ "${MODE}" = publish ]; then push_one agent-runner; fi
+fi
+if [ -n "${WANT[conformance-runner]:-}" ]; then
+  build_one conformance-runner
+  if [ "${MODE}" = publish ]; then push_one conformance-runner; fi
+fi
+
+# --- summary ------------------------------------------------------------------------------
+{
+  echo "## Images (${MODE})"
+  echo
+  echo "commit \`${HEAD_SHA}\`"
+  echo
+  echo "| image | built + verified | seconds | published digest (registry-returned) | note |"
+  echo "|---|---|---|---|---|"
+  for i in "${ALL_IMAGES[@]}"; do
+    d="${DIGEST[$i]:-}"
+    echo "| \`${i}\` | $(yn "${BUILT[$i]:-}" yes no) | ${SECS[$i]:--} | $(yn "${d}" "\`${REGISTRY}/${NAMESPACE}/${i}@${d}\`" -) | ${NOTE[$i]:-inputs unchanged} |"
+  done
+  echo
+  if [ "${MODE}" = publish ]; then
+    echo "Tags pushed: \`${SHA_TAG}\` (immutable) and \`${MOVING_TAG}\` (moving)."
+    echo "Nothing here writes to git. Renovate opens the digest-bump PR on its next run."
+  else
+    echo "Verify mode: nothing was pushed and no registry credential was present."
+  fi
+} >>"${SUMMARY}"
+log "done"