feat(images): build, verify and publish the three images in CI; Renovate carries the digests
Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a workstation build/push. Nothing had ever been pushed to the Gitea registry. - .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as sha-<commit> + main, reading the digest back from the registry. Only changed images build. - Each image's own build-image.sh does the build and in-image verification, now engine-neutral (DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh. - Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex manager for them. The worker Job template is deliberately untracked (it would loop). - renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION. - Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed
1 parent
1301af9a4c
commit
0da34ddbc6
24 files changed
+1280
-210
No files matched your search
@@ -0,0 +1,222 @@
|
||||
---
|
||||
date: 2026-10-29
|
||||
branch: feat/image-pipeline
|
||||
title: "Images are built, verified and published by CI; Renovate carries the digests into git"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Operator rule (2026-10-29): container image creation, hosting and lifecycle belong to CI/CD and
|
||||
the codebase, never to a one-off workstation build/push. Before this branch the Home-Network org's
|
||||
package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and
|
||||
both `build-image.sh` scripts said "publishing is an operator action".
|
||||
|
||||
- **`.gitea/workflows/images.yaml`** + **`.gitea/workflows/scripts/images.sh`** (the logic).
|
||||
- Pull request: build and verify only. No push, and the step references no secret at all.
|
||||
- Push to `main`, or `workflow_dispatch` on `main`: build, verify, then push to
|
||||
`gitea.home.keithjackson.dev/home-network/{toolbox,agent-runner,conformance-runner}`.
|
||||
Tags are `sha-<12 hex>` (immutable: the driver refuses to overwrite one) and `main` (moving).
|
||||
The digest is read back from the registry with a HEAD on the manifest, compared with what the
|
||||
client printed, and written to the job summary.
|
||||
- Only images whose inputs changed are built. A re-run on the same commit skips images that are
|
||||
already published.
|
||||
- agent-runner builds FROM the toolbox built in the same run (on main, FROM its just-pushed
|
||||
digest). Otherwise:
|
||||
- in verify mode the toolbox is built anyway as its base, because a PR holds no credential;
|
||||
- in publish mode it uses `AGENT_BASE_IMAGE` from `infra/agent/versions.env`;
|
||||
- if that is still the zero placeholder, the toolbox is built and published first. This is
|
||||
how the first run on main bootstraps.
|
||||
- An always-run step prunes the DinD image store.
|
||||
- **One build recipe per image, reused by CI.**
|
||||
- New `infra/devcontainer/build-image.sh`. `dev.sh build` now delegates to it. It has an
|
||||
in-image verification step that the toolbox never had: every tool is on PATH, and
|
||||
promtool, kubectl, chainsaw, kyverno and sops report their pins.
|
||||
- `infra/agent/build-image.sh` and `conformance/incluster/build-image.sh` were made
|
||||
engine-neutral, because CI runs `ENGINE=docker` against DinD:
|
||||
- `podman image exists` became `image inspect`, with a pull fallback;
|
||||
- `{{.Digest}}` and `--digestfile` are gone. No local digest is printed, since only the
|
||||
registry's digest is meaningful;
|
||||
- `probe-validation.sh` is fed over **stdin** instead of a bind mount. Under DinD a bind
|
||||
source is resolved on the daemon's filesystem, which would silently mount an empty
|
||||
directory;
|
||||
- `probe-validation.sh` itself gained `${BASH_SOURCE[0]:-$0}` so it survives `bash -s`
|
||||
under `set -u`.
|
||||
- `AGENT_BASE_IMAGE` and `IMAGE_TAG` can be overridden from the environment, and
|
||||
`BUILD_REVISION` adds an `org.opencontainers.image.revision` label.
|
||||
- The "publishing is an operator action" text was replaced throughout.
|
||||
- **Digest references.** Each now carries a `# renovate-digest: image=<repo> tag=main` marker
|
||||
on the line above it:
|
||||
- `platform/pm-agent/deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`), which share
|
||||
one dependency, so they bump in one PR;
|
||||
- `infra/agent/versions.env` `AGENT_BASE_IMAGE`, which was `home-k3s-dev:local` and is now
|
||||
the published toolbox by digest (zero placeholder until the first publish);
|
||||
- `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to the
|
||||
zero placeholder because Renovate can only replace a digest it can match.
|
||||
|
||||
The admission policies are **untouched**; the `<repo>@sha256:<digest>` form is preserved.
|
||||
- **`renovate.json5`**:
|
||||
- a new `custom.regex` manager for the markers;
|
||||
- `enabledManagers` `regex` → `custom.regex`;
|
||||
- a packageRule that disables the `kubernetes` manager for these three repos. Otherwise it
|
||||
would also see the digest-only lines and look them up as `latest`;
|
||||
- an `own-image` label;
|
||||
- `ignorePaths` narrowed from `conformance/**` to its subdirectories. `ignorePaths` applies
|
||||
to custom managers too, so the blanket glob hid `conformance/versions.env`.
|
||||
- **`renovate.yaml`**: `renovate-version: 44.167.0` pinned, plus a `RENOVATE_HOST_RULES` docker
|
||||
rule for the Gitea registry, using the same `RENOVATE_TOKEN`, which now also needs
|
||||
`read:package`. See below for why the version pin is a real fix and not tidiness.
|
||||
- **`infra/gitea/registry-retention.sh`**: dry run by default, `--apply` to delete. This is an
|
||||
operator action. Per image it keeps:
|
||||
- `main` and `latest`;
|
||||
- the 10 newest `sha-` tags;
|
||||
- every tag whose digest is pinned anywhere in `origin/main`;
|
||||
- every digest-named version.
|
||||
- **`infra/gitea/conformance-trigger-kubeconfig.sh`**: the "runner image is digest-pinned"
|
||||
precondition grepped the literal line `CONFORMANCE_RUNNER_IMAGE="${…_REPO}@${…_DIGEST}"`
|
||||
for 64 hex digits, so it **could never pass**. It now sources the file, evaluates the value,
|
||||
and rejects the zero placeholder.
|
||||
- **`conformance/tests/l0-static/alert-rules-unit-tested/check.sh`**: its "the toolbox build
|
||||
passes `--build-arg PROMTOOL_VERSION`" assertion now reads `infra/devcontainer/build-image.sh`
|
||||
instead of `dev.sh`. The gate caught this on the first run, because the recipe had moved.
|
||||
- **Docs**: `platform/pm-agent/README.md`, `runbooks/pm-agent.md` (two rows),
|
||||
`docs/plans/pm-agent-issue-to-pr.md` (follow-up 1), `conformance/incluster/README.md`,
|
||||
`infra/devcontainer/README.md`, and a new "How images are built" section in
|
||||
`.gitea/workflows/README.md`.
|
||||
|
||||
### Decisions and pushback
|
||||
|
||||
- **The push credential is a PAT secret, `REGISTRY_PUSH_TOKEN`; the Actions token cannot push.**
|
||||
Checked against Gitea **v1.24.3** source:
|
||||
- `services/auth/basic.go` `Basic.Verify` turns a task token into the pseudo-user
|
||||
`gitea-actions` (ID −2) and sets no token scope;
|
||||
- `routers/api/packages/container/container.go` `Authenticate` (the `/v2/token` exchange every
|
||||
docker client performs) requires `read:package`/`write:package`/`all` scope, so it answers
|
||||
401. The Actions token cannot even *pull*;
|
||||
- `services/context/package.go` `determineAccessMode` has `// TODO: ActionUser permission check`
|
||||
and no branch for that user. Even past login it would get at most read on a public org;
|
||||
- `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3.
|
||||
|
||||
The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight
|
||||
pattern.
|
||||
- **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate:
|
||||
- `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` →
|
||||
`custom.regex`;
|
||||
- that migration runs on every repository run (`lib/workers/repository/init/merge.ts` →
|
||||
`migrateAndValidate`), not only in the validator;
|
||||
- `lib/modules/manager/index.ts` `getEnabledManagersList` accepts either spelling;
|
||||
- `custom.regex` exists since 37.51.1 (renovatebot/renovate#24079).
|
||||
- **The open question hid a bigger bug: the Renovate config on main is invalid for the Renovate
|
||||
the workflow actually runs.**
|
||||
- `renovatebot/[email protected]` defaults `renovate-version` to `39`
|
||||
(`action.yml`, `src/docker.ts`), which is 39.264.0.
|
||||
- The validator used in #79 was `ghcr.io/renovatebot/renovate` with no tag, i.e. `latest`,
|
||||
44.167.0.
|
||||
- **Measured:** `renovate:39.264.0 renovate-config-validator` on main's `renovate.json5` gives
|
||||
`ERROR: Found errors in configuration` with five errors, all `managerFilePatterns`.
|
||||
`fileMatch` was renamed in 40.2.0.
|
||||
- So once #80 fixes the filename, every run would still die at `CONFIG_VALIDATION`.
|
||||
- Fixed by pinning `renovate-version: 44.167.0`. This branch's config validates clean on it.
|
||||
The only `--strict` complaint is a pending migration of the pre-existing top-level
|
||||
`"docker": {"enabled": true}` block, which I left alone.
|
||||
- **Unworkable as briefed: tracking the worker Job template's digest.**
|
||||
- `infra/agent/worker-job-template.yaml` is `COPY`ed into agent-runner. A Renovate bump there
|
||||
changes agent-runner's inputs, CI publishes a new digest, Renovate bumps it again, and so on
|
||||
forever.
|
||||
- `pm-dispatch` overwrites that field from `PM_WORKER_IMAGE` at runtime
|
||||
(`pm-dispatch.py:649`), so nothing is lost by leaving it all-zero. It is commented as
|
||||
deliberately untracked.
|
||||
- For the same loop reason, `conformance/versions.env` is **not** an input of the
|
||||
conformance-runner build, and a pipeline-only change does not republish on main.
|
||||
- **Changed from the brief: retention is a script, not a Gitea cleanup rule.**
|
||||
- Gitea 1.24.3 has **no API** for cleanup rules (none in `routers/api/v1/api.go`; web
|
||||
session+CSRF forms only, in `routers/web/web.go`). So a rule cannot be created, or
|
||||
drift-asserted, as code. **`ci/gitea-settings.sh` was therefore not extended.** A cheaper
|
||||
proxy (count versions per image) would need `read:package` on `SETTINGS_API_TOKEN`, which
|
||||
it lacks.
|
||||
- A built-in rule is also **unsafe here**. Gitea stores a manifest as a file of the TAG
|
||||
version it was pushed under (`routers/api/packages/container/manifest.go`
|
||||
`createPackageAndVersion`), and resolves a by-digest pull from those files
|
||||
(`models/packages/container/search.go`). Deleting the tag a deployed digest was pushed under
|
||||
breaks the `@sha256:` pull, and the rule's "newest N" knows nothing about what git pins. The
|
||||
script keeps every digest git pins.
|
||||
- **Auto-merge.** Digest bumps of these images auto-merge on a green gate (the existing
|
||||
`digest` rule). The exception is the conformance-runner bump: `conformance/versions.env` is a
|
||||
sensitive path, so it waits for an administrator's `/approve-override` every time.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **Local gate, pinned toolbox, from this worktree, committed HEAD.** Command:
|
||||
`ENGINE=<shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`.
|
||||
The shim adds the `.git` mount. Result: see the PR body for the rc and stage list. They are
|
||||
recorded there rather than here because this entry is part of the tree being gated.
|
||||
- **Local podman builds through the CI driver**, run as
|
||||
`MODE=verify ENGINE=podman BASE_SHA=$(git merge-base origin/main HEAD) .gitea/workflows/scripts/images.sh`.
|
||||
It returned rc 0 and built and verified all three images:
|
||||
- toolbox: every tool on PATH, and the five pins matched;
|
||||
- agent-runner: `PM_SELFTEST=1 pm-dispatch` OK, and the stdin-fed `probe-validation.sh`
|
||||
gave "20 refusal(s), 0 failure(s)";
|
||||
- conformance-runner: all seven `conformance-clone` refusals.
|
||||
- **Other driver paths, run locally:**
|
||||
- agent-runner alone builds the toolbox as its base;
|
||||
- an empty diff builds nothing;
|
||||
- publish mode without a token dies before doing anything;
|
||||
- `infra/agent/build-image.sh` with the zero placeholder dies with build-locally
|
||||
instructions.
|
||||
- **Renovate, with the pinned 44.167.0 image, locally:**
|
||||
- `renovate-config-validator` exits 0;
|
||||
- a `--platform=local` `extract` dry run finds all four marker references (custom regex manager,
|
||||
`currentValue: main`, `currentDigest: sha256:000…`);
|
||||
- a `lookup` dry run shows the `kubernetes`-manager duplicates as `skipReason: disabled` and the
|
||||
marker deps taking the "unsupported/unversioned value main → getDigest(main)" path. That
|
||||
lookup currently 404s because nothing is published yet.
|
||||
- **Renovate 39.264.0 rejects main's current `renovate.json5`.** Validator output reproduced
|
||||
above.
|
||||
- **`registry-retention.sh` dry run against the live Gitea (read-only):** zero versions per image,
|
||||
nothing to delete. The token exchange and the package listing both answered.
|
||||
- **Live, read-only measurements** (kubectl and the Gitea API):
|
||||
- the runner pod is on k3s-prod-1, which had **7 GB** nodefs available;
|
||||
- DinD holds only the 1.54 GB act image; buildx is v0.20.1; the daemon is 27.5.1 on overlay2
|
||||
with cgroup v2;
|
||||
- Gitea `/data` is 9.7G, 1% used;
|
||||
- the Gateway's Envoy (Cilium) has `streamIdleTimeout: 300s`, `maxStreamDuration: 0s`, and
|
||||
**no route `timeout`** on the gitea route;
|
||||
- Gitea's `LIMIT_SIZE_CONTAINER` is unset (default −1 = unlimited, `modules/setting/packages.go`).
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **Nothing has been pushed to the registry, by me or by CI.** The publish path has never run:
|
||||
- the registry HEAD/digest read-back;
|
||||
- the immutable-tag guard;
|
||||
- "already published at this commit" idempotence;
|
||||
- agent-runner `FROM <toolbox>@sha256:`;
|
||||
- `docker login` with a PAT.
|
||||
|
||||
The first merge to main is the first test, and it needs `REGISTRY_PUSH_TOKEN` to exist first.
|
||||
- **Large-layer push through the Gateway is UNVERIFIED.** Envoy has no request-body cap and no
|
||||
stream duration cap here. But with no route `timeout` set, Envoy's **default 15 s upstream
|
||||
timeout** applies from the end of the request to the end of the response. That covers Gitea
|
||||
finalising a blob: copying the upload into package storage on Longhorn. A ~250 MB compressed
|
||||
layer should finish well inside 15 s, but I could not measure it without pushing. If the first
|
||||
publish shows 504s on blob PUTs, set `timeouts.request` on the gitea HTTPRoute.
|
||||
- **DinD fit is estimated, not measured** (until the PR run reports):
|
||||
- memory: 2Gi is shared by the job container and every build container;
|
||||
- disk: the toolbox, agent-runner, conformance-runner, Fedora base and build cache should
|
||||
total roughly 3 GB on top of the 1.5 GB act image, against 7 GB free on the node, with
|
||||
`dind-storage` having no sizeLimit.
|
||||
- **That act_runner hands the job container a Docker socket is inferred** from act_runner's
|
||||
DOCKER_HOST handling, not measured. The driver fails loudly if it does not.
|
||||
- **Renovate has still never completed a run here.** Three things block it:
|
||||
- #80, the `configurationFile` name;
|
||||
- `RENOVATE_TOKEN` gaining `read:package`;
|
||||
- this branch's version pin.
|
||||
|
||||
`renovatebot/[email protected]` running a 44.x image is untested. The action only passes
|
||||
the tag to `docker run`.
|
||||
- **Pods cannot pull these images yet regardless.** The kubelet pull credential is PR #61
|
||||
(`pm-agent-registry` dockerconfigjson, `read:package`), still open. Until it merges, a correct
|
||||
digest still ends in ImagePullBackOff (401), as before.
|
||||
- **`registry-retention.sh --apply` has never run**, and its keep logic has only run against an
|
||||
empty registry.
|
||||
- **`git diff` change detection on Gitea `push` events relies on `github.event.before`.** A
|
||||
force-push or a first push falls back to building everything, which is intended but not
|
||||
observed.
|
||||
Reference in new issue
Block a user