--- date: 2026-10-29 branch: feat/image-pipeline title: "Images are built, verified and published by CI; Renovate carries the digests into git" --- ## What changed Operator rule (2026-10-29): container image creation, hosting and lifecycle belong to CI/CD and the codebase, never to a one-off workstation build/push. Before this branch the Home-Network org's package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and both `build-image.sh` scripts said "publishing is an operator action". - **`.gitea/workflows/images.yaml`** + **`.gitea/workflows/scripts/images.sh`** (the logic). - Pull request: build and verify only. No push, and the step references no secret at all. - Push to `main`, or `workflow_dispatch` on `main`: build, verify, then push to `gitea.home.keithjackson.dev/home-network/{toolbox,agent-runner,conformance-runner}`. Tags are `sha-<12 hex>` (immutable: the driver refuses to overwrite one) and `main` (moving). The digest is read back from the registry with a HEAD on the manifest, compared with what the client printed, and written to the job summary. - Only images whose inputs changed are built. A re-run on the same commit skips images that are already published. - agent-runner builds FROM the toolbox built in the same run (on main, FROM its just-pushed digest). Otherwise: - in verify mode the toolbox is built anyway as its base, because a PR holds no credential; - in publish mode it uses `AGENT_BASE_IMAGE` from `infra/agent/versions.env`; - if that is still the zero placeholder, the toolbox is built and published first. This is how the first run on main bootstraps. - An always-run step prunes the DinD image store. - **One build recipe per image, reused by CI.** - New `infra/devcontainer/build-image.sh`. `dev.sh build` now delegates to it. It has an in-image verification step that the toolbox never had: every tool is on PATH, and promtool, kubectl, chainsaw, kyverno and sops report their pins. - `infra/agent/build-image.sh` and `conformance/incluster/build-image.sh` were made engine-neutral, because CI runs `ENGINE=docker` against DinD: - `podman image exists` became `image inspect`, with a pull fallback; - `{{.Digest}}` and `--digestfile` are gone. No local digest is printed, since only the registry's digest is meaningful; - `probe-validation.sh` is fed over **stdin** instead of a bind mount. Under DinD a bind source is resolved on the daemon's filesystem, which would silently mount an empty directory; - `probe-validation.sh` itself gained `${BASH_SOURCE[0]:-$0}` so it survives `bash -s` under `set -u`. - `AGENT_BASE_IMAGE` and `IMAGE_TAG` can be overridden from the environment, and `BUILD_REVISION` adds an `org.opencontainers.image.revision` label. - The "publishing is an operator action" text was replaced throughout. - **Digest references.** Each now carries a `# renovate-digest: image= tag=main` marker on the line above it: - `platform/pm-agent/deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`), which share one dependency, so they bump in one PR; - `infra/agent/versions.env` `AGENT_BASE_IMAGE`, which was `home-k3s-dev:local` and is now the published toolbox by digest (zero placeholder until the first publish); - `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to the zero placeholder because Renovate can only replace a digest it can match. The admission policies are **untouched**; the `@sha256:` form is preserved. - **`renovate.json5`**: - a new `custom.regex` manager for the markers; - `enabledManagers` `regex` → `custom.regex`; - a packageRule that disables the `kubernetes` manager for these three repos. Otherwise it would also see the digest-only lines and look them up as `latest`; - an `own-image` label; - `ignorePaths` narrowed from `conformance/**` to its subdirectories. `ignorePaths` applies to custom managers too, so the blanket glob hid `conformance/versions.env`. - **`renovate.yaml`**: `renovate-version: 44.167.0` pinned, plus a `RENOVATE_HOST_RULES` docker rule for the Gitea registry, using the same `RENOVATE_TOKEN`, which now also needs `read:package`. See below for why the version pin is a real fix and not tidiness. - **`infra/gitea/registry-retention.sh`**: dry run by default, `--apply` to delete. This is an operator action. Per image it keeps: - `main` and `latest`; - the 10 newest `sha-` tags; - every tag whose digest is pinned anywhere in `origin/main`; - every digest-named version. - **`infra/gitea/conformance-trigger-kubeconfig.sh`**: the "runner image is digest-pinned" precondition grepped the literal line `CONFORMANCE_RUNNER_IMAGE="${…_REPO}@${…_DIGEST}"` for 64 hex digits, so it **could never pass**. It now sources the file, evaluates the value, and rejects the zero placeholder. - **`conformance/tests/l0-static/alert-rules-unit-tested/check.sh`**: its "the toolbox build passes `--build-arg PROMTOOL_VERSION`" assertion now reads `infra/devcontainer/build-image.sh` instead of `dev.sh`. The gate caught this on the first run, because the recipe had moved. - **Docs**: `platform/pm-agent/README.md`, `runbooks/pm-agent.md` (two rows), `docs/plans/pm-agent-issue-to-pr.md` (follow-up 1), `conformance/incluster/README.md`, `infra/devcontainer/README.md`, and a new "How images are built" section in `.gitea/workflows/README.md`. ### Decisions and pushback - **The push credential is a PAT secret, `REGISTRY_PUSH_TOKEN`; the Actions token cannot push.** Checked against Gitea **v1.24.3** source: - `services/auth/basic.go` `Basic.Verify` turns a task token into the pseudo-user `gitea-actions` (ID −2) and sets no token scope; - `routers/api/packages/container/container.go` `Authenticate` (the `/v2/token` exchange every docker client performs) requires `read:package`/`write:package`/`all` scope, so it answers 401. The Actions token cannot even *pull*; - `services/context/package.go` `determineAccessMode` has `// TODO: ActionUser permission check` and no branch for that user. Even past login it would get at most read on a public org; - `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3. The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight pattern. - **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate: - `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` → `custom.regex`; - that migration runs on every repository run (`lib/workers/repository/init/merge.ts` → `migrateAndValidate`), not only in the validator; - `lib/modules/manager/index.ts` `getEnabledManagersList` accepts either spelling; - `custom.regex` exists since 37.51.1 (renovatebot/renovate#24079). - **The open question hid a bigger bug: the Renovate config on main is invalid for the Renovate the workflow actually runs.** - `renovatebot/github-action@v41.0.9` defaults `renovate-version` to `39` (`action.yml`, `src/docker.ts`), which is 39.264.0. - The validator used in #79 was `ghcr.io/renovatebot/renovate` with no tag, i.e. `latest`, 44.167.0. - **Measured:** `renovate:39.264.0 renovate-config-validator` on main's `renovate.json5` gives `ERROR: Found errors in configuration` with five errors, all `managerFilePatterns`. `fileMatch` was renamed in 40.2.0. - So once #80 fixes the filename, every run would still die at `CONFIG_VALIDATION`. - Fixed by pinning `renovate-version: 44.167.0`. This branch's config validates clean on it. The only `--strict` complaint is a pending migration of the pre-existing top-level `"docker": {"enabled": true}` block, which I left alone. - **Unworkable as briefed: tracking the worker Job template's digest.** - `infra/agent/worker-job-template.yaml` is `COPY`ed into agent-runner. A Renovate bump there changes agent-runner's inputs, CI publishes a new digest, Renovate bumps it again, and so on forever. - `pm-dispatch` overwrites that field from `PM_WORKER_IMAGE` at runtime (`pm-dispatch.py:649`), so nothing is lost by leaving it all-zero. It is commented as deliberately untracked. - For the same loop reason, `conformance/versions.env` is **not** an input of the conformance-runner build, and a pipeline-only change does not republish on main. - **Changed from the brief: retention is a script, not a Gitea cleanup rule.** - Gitea 1.24.3 has **no API** for cleanup rules (none in `routers/api/v1/api.go`; web session+CSRF forms only, in `routers/web/web.go`). So a rule cannot be created, or drift-asserted, as code. **`ci/gitea-settings.sh` was therefore not extended.** A cheaper proxy (count versions per image) would need `read:package` on `SETTINGS_API_TOKEN`, which it lacks. - A built-in rule is also **unsafe here**. Gitea stores a manifest as a file of the TAG version it was pushed under (`routers/api/packages/container/manifest.go` `createPackageAndVersion`), and resolves a by-digest pull from those files (`models/packages/container/search.go`). Deleting the tag a deployed digest was pushed under breaks the `@sha256:` pull, and the rule's "newest N" knows nothing about what git pins. The script keeps every digest git pins. - **Auto-merge.** Digest bumps of these images auto-merge on a green gate (the existing `digest` rule). The exception is the conformance-runner bump: `conformance/versions.env` is a sensitive path, so it waits for an administrator's `/approve-override` every time. ## What I verified - **Local gate, pinned toolbox, from this worktree, committed HEAD.** Command: `ENGINE= DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`. The shim adds the `.git` mount. Result: see the PR body for the rc and stage list. They are recorded there rather than here because this entry is part of the tree being gated. - **Local podman builds through the CI driver**, run as `MODE=verify ENGINE=podman BASE_SHA=$(git merge-base origin/main HEAD) .gitea/workflows/scripts/images.sh`. It returned rc 0 and built and verified all three images: - toolbox: every tool on PATH, and the five pins matched; - agent-runner: `PM_SELFTEST=1 pm-dispatch` OK, and the stdin-fed `probe-validation.sh` gave "20 refusal(s), 0 failure(s)"; - conformance-runner: all seven `conformance-clone` refusals. - **Other driver paths, run locally:** - agent-runner alone builds the toolbox as its base; - an empty diff builds nothing; - publish mode without a token dies before doing anything; - `infra/agent/build-image.sh` with the zero placeholder dies with build-locally instructions. - **Renovate, with the pinned 44.167.0 image, locally:** - `renovate-config-validator` exits 0; - a `--platform=local` `extract` dry run finds all four marker references (custom regex manager, `currentValue: main`, `currentDigest: sha256:000…`); - a `lookup` dry run shows the `kubernetes`-manager duplicates as `skipReason: disabled` and the marker deps taking the "unsupported/unversioned value main → getDigest(main)" path. That lookup currently 404s because nothing is published yet. - **Renovate 39.264.0 rejects main's current `renovate.json5`.** Validator output reproduced above. - **`registry-retention.sh` dry run against the live Gitea (read-only):** zero versions per image, nothing to delete. The token exchange and the package listing both answered. - **Live, read-only measurements** (kubectl and the Gitea API): - the runner pod is on k3s-prod-1, which had **7 GB** nodefs available; - DinD holds only the 1.54 GB act image; buildx is v0.20.1; the daemon is 27.5.1 on overlay2 with cgroup v2; - Gitea `/data` is 9.7G, 1% used; - the Gateway's Envoy (Cilium) has `streamIdleTimeout: 300s`, `maxStreamDuration: 0s`, and **no route `timeout`** on the gitea route; - Gitea's `LIMIT_SIZE_CONTAINER` is unset (default −1 = unlimited, `modules/setting/packages.go`). ## What I did NOT verify - **Nothing has been pushed to the registry, by me or by CI.** The publish path has never run: - the registry HEAD/digest read-back; - the immutable-tag guard; - "already published at this commit" idempotence; - agent-runner `FROM @sha256:`; - `docker login` with a PAT. The first merge to main is the first test, and it needs `REGISTRY_PUSH_TOKEN` to exist first. - **Large-layer push through the Gateway is UNVERIFIED.** Envoy has no request-body cap and no stream duration cap here. But with no route `timeout` set, Envoy's **default 15 s upstream timeout** applies from the end of the request to the end of the response. That covers Gitea finalising a blob: copying the upload into package storage on Longhorn. A ~250 MB compressed layer should finish well inside 15 s, but I could not measure it without pushing. If the first publish shows 504s on blob PUTs, set `timeouts.request` on the gitea HTTPRoute. - **DinD fit is estimated, not measured** (until the PR run reports): - memory: 2Gi is shared by the job container and every build container; - disk: the toolbox, agent-runner, conformance-runner, Fedora base and build cache should total roughly 3 GB on top of the 1.5 GB act image, against 7 GB free on the node, with `dind-storage` having no sizeLimit. - **That act_runner hands the job container a Docker socket is inferred** from act_runner's DOCKER_HOST handling, not measured. The driver fails loudly if it does not. - **Renovate has still never completed a run here.** Three things block it: - #80, the `configurationFile` name; - `RENOVATE_TOKEN` gaining `read:package`; - this branch's version pin. `renovatebot/github-action@v41.0.9` running a 44.x image is untested. The action only passes the tag to `docker run`. - **Pods cannot pull these images yet regardless.** The kubelet pull credential is PR #61 (`pm-agent-registry` dockerconfigjson, `read:package`), still open. Until it merges, a correct digest still ends in ImagePullBackOff (401), as before. - **`registry-retention.sh --apply` has never run**, and its keep logic has only run against an empty registry. - **`git diff` change detection on Gitea `push` events relies on `github.event.before`.** A force-push or a first push falls back to building everything, which is intended but not observed.