home-network-k3s-impl/decisions/2026-10-29-image-pipeline.md

---
date: 2026-10-29
branch: feat/image-pipeline
title: "Images are built, verified and published by CI; Renovate carries the digests into git"
---

## What changed

Operator rule (2026-10-29): container image creation, hosting and lifecycle belong to CI/CD and
the codebase, never to a one-off workstation build/push. Before this branch the Home-Network org's
package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and
both `build-image.sh` scripts said "publishing is an operator action".

- **`.gitea/workflows/images.yaml`** + **`.gitea/workflows/scripts/images.sh`** (the logic).
  - Pull request: build and verify only. No push, and the step references no secret at all.
  - Push to `main`, or `workflow_dispatch` on `main`: build, verify, then push to
    `gitea.home.keithjackson.dev/home-network/{toolbox,agent-runner,conformance-runner}`.
    Tags are `sha-<12 hex>` (immutable: the driver refuses to overwrite one) and `main` (moving).
    The digest is read back from the registry with a HEAD on the manifest, compared with what the
    client printed, and written to the job summary.
  - Only images whose inputs changed are built. A re-run on the same commit skips images that are
    already published.
  - agent-runner builds FROM the toolbox built in the same run (on main, FROM its just-pushed
    digest). Otherwise:
    - in verify mode the toolbox is built anyway as its base, because a PR holds no credential;
    - in publish mode it uses `AGENT_BASE_IMAGE` from `infra/agent/versions.env`;
    - if that is still the zero placeholder, the toolbox is built and published first. This is
      how the first run on main bootstraps.
  - An always-run step prunes the DinD image store.
- **One build recipe per image, reused by CI.**
  - New `infra/devcontainer/build-image.sh`. `dev.sh build` now delegates to it. It has an
    in-image verification step that the toolbox never had: every tool is on PATH, and
    promtool, kubectl, chainsaw, kyverno and sops report their pins.
  - `infra/agent/build-image.sh` and `conformance/incluster/build-image.sh` were made
    engine-neutral, because CI runs `ENGINE=docker` against DinD:
    - `podman image exists` became `image inspect`, with a pull fallback;
    - `{{.Digest}}` and `--digestfile` are gone. No local digest is printed, since only the
      registry's digest is meaningful;
    - `probe-validation.sh` is fed over **stdin** instead of a bind mount. Under DinD a bind
      source is resolved on the daemon's filesystem, which would silently mount an empty
      directory;
    - `probe-validation.sh` itself gained `${BASH_SOURCE[0]:-$0}` so it survives `bash -s`
      under `set -u`.
  - `AGENT_BASE_IMAGE` and `IMAGE_TAG` can be overridden from the environment, and
    `BUILD_REVISION` adds an `org.opencontainers.image.revision` label.
  - The "publishing is an operator action" text was replaced throughout.
- **Digest references.** Each now carries a `# renovate-digest: image=<repo> tag=main` marker
  on the line above it:
  - `platform/pm-agent/deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`), which share
    one dependency, so they bump in one PR;
  - `infra/agent/versions.env` `AGENT_BASE_IMAGE`, which was `home-k3s-dev:local` and is now
    the published toolbox by digest (zero placeholder until the first publish);
  - `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to the
    zero placeholder because Renovate can only replace a digest it can match.

  The admission policies are **untouched**; the `<repo>@sha256:<digest>` form is preserved.
- **`renovate.json5`**:
  - a new `custom.regex` manager for the markers;
  - `enabledManagers` `regex` → `custom.regex`;
  - a packageRule that disables the `kubernetes` manager for these three repos. Otherwise it
    would also see the digest-only lines and look them up as `latest`;
  - an `own-image` label;
  - `ignorePaths` narrowed from `conformance/**` to its subdirectories. `ignorePaths` applies
    to custom managers too, so the blanket glob hid `conformance/versions.env`.
- **`renovate.yaml`**: `renovate-version: 44.167.0` pinned, plus a `RENOVATE_HOST_RULES` docker
  rule for the Gitea registry, using the same `RENOVATE_TOKEN`, which now also needs
  `read:package`. See below for why the version pin is a real fix and not tidiness.
- **`infra/gitea/registry-retention.sh`**: dry run by default, `--apply` to delete. This is an
  operator action. Per image it keeps:
  - `main` and `latest`;
  - the 10 newest `sha-` tags;
  - every tag whose digest is pinned anywhere in `origin/main`;
  - every digest-named version.
- **`infra/gitea/conformance-trigger-kubeconfig.sh`**: the "runner image is digest-pinned"
  precondition grepped the literal line `CONFORMANCE_RUNNER_IMAGE="${…_REPO}@${…_DIGEST}"`
  for 64 hex digits, so it **could never pass**. It now sources the file, evaluates the value,
  and rejects the zero placeholder.
- **`conformance/tests/l0-static/alert-rules-unit-tested/check.sh`**: its "the toolbox build
  passes `--build-arg PROMTOOL_VERSION`" assertion now reads `infra/devcontainer/build-image.sh`
  instead of `dev.sh`. The gate caught this on the first run, because the recipe had moved.
- **Docs**: `platform/pm-agent/README.md`, `runbooks/pm-agent.md` (two rows),
  `docs/plans/pm-agent-issue-to-pr.md` (follow-up 1), `conformance/incluster/README.md`,
  `infra/devcontainer/README.md`, and a new "How images are built" section in
  `.gitea/workflows/README.md`.

### Decisions and pushback

- **The push credential is a PAT secret, `REGISTRY_PUSH_TOKEN`; the Actions token cannot push.**
  Checked against Gitea **v1.24.3** source:
  - `services/auth/basic.go` `Basic.Verify` turns a task token into the pseudo-user
    `gitea-actions` (ID −2) and sets no token scope;
  - `routers/api/packages/container/container.go` `Authenticate` (the `/v2/token` exchange every
    docker client performs) requires `read:package`/`write:package`/`all` scope, so it answers
    401. The Actions token cannot even *pull*;
  - `services/context/package.go` `determineAccessMode` has `// TODO: ActionUser permission check`
    and no branch for that user. Even past login it would get at most read on a public org;
  - `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3.

  The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight
  pattern.
- **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate:
  - `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` →
    `custom.regex`;
  - that migration runs on every repository run (`lib/workers/repository/init/merge.ts` →
    `migrateAndValidate`), not only in the validator;
  - `lib/modules/manager/index.ts` `getEnabledManagersList` accepts either spelling;
  - `custom.regex` exists since 37.51.1 (renovatebot/renovate#24079).
- **The open question hid a bigger bug: the Renovate config on main is invalid for the Renovate
  the workflow actually runs.**
  - `renovatebot/[email protected]` defaults `renovate-version` to `39`
    (`action.yml`, `src/docker.ts`), which is 39.264.0.
  - The validator used in #79 was `ghcr.io/renovatebot/renovate` with no tag, i.e. `latest`,
    44.167.0.
  - **Measured:** `renovate:39.264.0 renovate-config-validator` on main's `renovate.json5` gives
    `ERROR: Found errors in configuration` with five errors, all `managerFilePatterns`.
    `fileMatch` was renamed in 40.2.0.
  - So once #80 fixes the filename, every run would still die at `CONFIG_VALIDATION`.
  - Fixed by pinning `renovate-version: 44.167.0`. This branch's config validates clean on it.
    The only `--strict` complaint is a pending migration of the pre-existing top-level
    `"docker": {"enabled": true}` block, which I left alone.
- **Unworkable as briefed: tracking the worker Job template's digest.**
  - `infra/agent/worker-job-template.yaml` is `COPY`ed into agent-runner. A Renovate bump there
    changes agent-runner's inputs, CI publishes a new digest, Renovate bumps it again, and so on
    forever.
  - `pm-dispatch` overwrites that field from `PM_WORKER_IMAGE` at runtime
    (`pm-dispatch.py:649`), so nothing is lost by leaving it all-zero. It is commented as
    deliberately untracked.
  - For the same loop reason, `conformance/versions.env` is **not** an input of the
    conformance-runner build, and a pipeline-only change does not republish on main.
- **Changed from the brief: retention is a script, not a Gitea cleanup rule.**
  - Gitea 1.24.3 has **no API** for cleanup rules (none in `routers/api/v1/api.go`; web
    session+CSRF forms only, in `routers/web/web.go`). So a rule cannot be created, or
    drift-asserted, as code. **`ci/gitea-settings.sh` was therefore not extended.** A cheaper
    proxy (count versions per image) would need `read:package` on `SETTINGS_API_TOKEN`, which
    it lacks.
  - A built-in rule is also **unsafe here**. Gitea stores a manifest as a file of the TAG
    version it was pushed under (`routers/api/packages/container/manifest.go`
    `createPackageAndVersion`), and resolves a by-digest pull from those files
    (`models/packages/container/search.go`). Deleting the tag a deployed digest was pushed under
    breaks the `@sha256:` pull, and the rule's "newest N" knows nothing about what git pins. The
    script keeps every digest git pins.
- **Auto-merge.** Digest bumps of these images auto-merge on a green gate (the existing
  `digest` rule). The exception is the conformance-runner bump: `conformance/versions.env` is a
  sensitive path, so it waits for an administrator's `/approve-override` every time.

## What I verified

- **Local gate, pinned toolbox, from this worktree, committed HEAD.** Command:
  `ENGINE=<shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`.
  The shim adds the `.git` mount. Result: see the PR body for the rc and stage list. They are
  recorded there rather than here because this entry is part of the tree being gated.
- **Local podman builds through the CI driver**, run as
  `MODE=verify ENGINE=podman BASE_SHA=$(git merge-base origin/main HEAD) .gitea/workflows/scripts/images.sh`.
  It returned rc 0 and built and verified all three images:
  - toolbox: every tool on PATH, and the five pins matched;
  - agent-runner: `PM_SELFTEST=1 pm-dispatch` OK, and the stdin-fed `probe-validation.sh`
    gave "20 refusal(s), 0 failure(s)";
  - conformance-runner: all seven `conformance-clone` refusals.
- **Other driver paths, run locally:**
  - agent-runner alone builds the toolbox as its base;
  - an empty diff builds nothing;
  - publish mode without a token dies before doing anything;
  - `infra/agent/build-image.sh` with the zero placeholder dies with build-locally
    instructions.
- **Renovate, with the pinned 44.167.0 image, locally:**
  - `renovate-config-validator` exits 0;
  - a `--platform=local` `extract` dry run finds all four marker references (custom regex manager,
    `currentValue: main`, `currentDigest: sha256:000…`);
  - a `lookup` dry run shows the `kubernetes`-manager duplicates as `skipReason: disabled` and the
    marker deps taking the "unsupported/unversioned value main → getDigest(main)" path. That
    lookup currently 404s because nothing is published yet.
- **Renovate 39.264.0 rejects main's current `renovate.json5`.** Validator output reproduced
  above.
- **`registry-retention.sh` dry run against the live Gitea (read-only):** zero versions per image,
  nothing to delete. The token exchange and the package listing both answered.
- **Live, read-only measurements** (kubectl and the Gitea API):
  - the runner pod is on k3s-prod-1, which had **7 GB** nodefs available;
  - DinD holds only the 1.54 GB act image; buildx is v0.20.1; the daemon is 27.5.1 on overlay2
    with cgroup v2;
  - Gitea `/data` is 9.7G, 1% used;
  - the Gateway's Envoy (Cilium) has `streamIdleTimeout: 300s`, `maxStreamDuration: 0s`, and
    **no route `timeout`** on the gitea route;
  - Gitea's `LIMIT_SIZE_CONTAINER` is unset (default −1 = unlimited, `modules/setting/packages.go`).

## What I did NOT verify

- **Nothing has been pushed to the registry, by me or by CI.** The publish path has never run:
  - the registry HEAD/digest read-back;
  - the immutable-tag guard;
  - "already published at this commit" idempotence;
  - agent-runner `FROM <toolbox>@sha256:`;
  - `docker login` with a PAT.

  The first merge to main is the first test, and it needs `REGISTRY_PUSH_TOKEN` to exist first.
- **Large-layer push through the Gateway is UNVERIFIED.** Envoy has no request-body cap and no
  stream duration cap here. But with no route `timeout` set, Envoy's **default 15 s upstream
  timeout** applies from the end of the request to the end of the response. That covers Gitea
  finalising a blob: copying the upload into package storage on Longhorn. A ~250 MB compressed
  layer should finish well inside 15 s, but I could not measure it without pushing. If the first
  publish shows 504s on blob PUTs, set `timeouts.request` on the gitea HTTPRoute.
- **DinD fit is estimated, not measured** (until the PR run reports):
  - memory: 2Gi is shared by the job container and every build container;
  - disk: the toolbox, agent-runner, conformance-runner, Fedora base and build cache should
    total roughly 3 GB on top of the 1.5 GB act image, against 7 GB free on the node, with
    `dind-storage` having no sizeLimit.
- **That act_runner hands the job container a Docker socket is inferred** from act_runner's
  DOCKER_HOST handling, not measured. The driver fails loudly if it does not.
- **Renovate has still never completed a run here.** Three things block it:
  - #80, the `configurationFile` name;
  - `RENOVATE_TOKEN` gaining `read:package`;
  - this branch's version pin.

  `renovatebot/[email protected]` running a 44.x image is untested. The action only passes
  the tag to `docker run`.
- **Pods cannot pull these images yet regardless.** The kubelet pull credential is PR #61
  (`pm-agent-registry` dockerconfigjson, `read:package`), still open. Until it merges, a correct
  digest still ends in ImagePullBackOff (401), as before.
- **`registry-retention.sh --apply` has never run**, and its keep logic has only run against an
  empty registry.
- **`git diff` change detection on Gitea `push` events relies on `github.event.before`.** A
  force-push or a first push falls back to building everything, which is intended but not
  observed.