home-network-k3s-impl/decisions/2026-10-29-image-pipeline.md

date, branch, title

date	branch	title
2026-10-29	feat/image-pipeline	Images are built, verified and published by CI; Renovate carries the digests into git

What changed

Operator rule (2026-10-29): container image creation, hosting and lifecycle belong to CI/CD and the codebase, never to a one-off workstation build/push. Before this branch the Home-Network org's package list was empty, platform/pm-agent/ pinned an all-zero digest (ImagePullBackOff), and both build-image.sh scripts said "publishing is an operator action".

• .gitea/workflows/images.yaml + .gitea/workflows/scripts/images.sh (the logic).

  • Pull request: build and verify only. No push, and the step references no secret at all.
  • Push to main, or workflow_dispatch on main: build, verify, then push to gitea.home.keithjackson.dev/home-network/{toolbox,agent-runner,conformance-runner}. Tags are sha-<12 hex> (immutable: the driver refuses to overwrite one) and main (moving). The digest is read back from the registry with a HEAD on the manifest, compared with what the client printed, and written to the job summary.
  • Only images whose inputs changed are built. A re-run on the same commit skips images that are already published.
  • agent-runner builds FROM the toolbox built in the same run (on main, FROM its just-pushed digest). Otherwise:
    • in verify mode the toolbox is built anyway as its base, because a PR holds no credential;
    • in publish mode it uses AGENT_BASE_IMAGE from infra/agent/versions.env;
    • if that is still the zero placeholder, the toolbox is built and published first. This is how the first run on main bootstraps.
  • An always-run step prunes the DinD image store.

• One build recipe per image, reused by CI.

  • New infra/devcontainer/build-image.sh. dev.sh build now delegates to it. It has an in-image verification step that the toolbox never had: every tool is on PATH, and promtool, kubectl, chainsaw, kyverno and sops report their pins.
  • infra/agent/build-image.sh and conformance/incluster/build-image.sh were made engine-neutral, because CI runs ENGINE=docker against DinD:
    • podman image exists became image inspect, with a pull fallback;
    • {{.Digest}} and --digestfile are gone. No local digest is printed, since only the registry's digest is meaningful;
    • probe-validation.sh is fed over stdin instead of a bind mount. Under DinD a bind source is resolved on the daemon's filesystem, which would silently mount an empty directory;
    • probe-validation.sh itself gained ${BASH_SOURCE[0]:-$0} so it survives bash -s under set -u.
  • AGENT_BASE_IMAGE and IMAGE_TAG can be overridden from the environment, and BUILD_REVISION adds an org.opencontainers.image.revision label.
  • The "publishing is an operator action" text was replaced throughout.

• Digest references. Each now carries a # renovate-digest: image=<repo> tag=main marker on the line above it:

  • platform/pm-agent/deployment.yaml and configmap.yaml (PM_WORKER_IMAGE), which share one dependency, so they bump in one PR;
  • infra/agent/versions.env AGENT_BASE_IMAGE, which was home-k3s-dev:local and is now the published toolbox by digest (zero placeholder until the first publish);
  • conformance/versions.env CONFORMANCE_RUNNER_IMAGE_DIGEST, which went from "" to the zero placeholder because Renovate can only replace a digest it can match.

  The admission policies are untouched; the <repo>@sha256:<digest> form is preserved.

• renovate.json5:

  • a new custom.regex manager for the markers;
  • enabledManagers regex → custom.regex;
  • a packageRule that disables the kubernetes manager for these three repos. Otherwise it would also see the digest-only lines and look them up as latest;
  • an own-image label;
  • ignorePaths narrowed from conformance/** to its subdirectories. ignorePaths applies to custom managers too, so the blanket glob hid conformance/versions.env.

• renovate.yaml: renovate-version: 44.167.0 pinned, plus a RENOVATE_HOST_RULES docker rule for the Gitea registry, using the same RENOVATE_TOKEN, which now also needs read:package. See below for why the version pin is a real fix and not tidiness.

• infra/gitea/registry-retention.sh: dry run by default, --apply to delete. This is an operator action. Per image it keeps:

  • main and latest;
  • the 10 newest sha- tags;
  • every tag whose digest is pinned anywhere in origin/main;
  • every digest-named version.

• infra/gitea/conformance-trigger-kubeconfig.sh: the "runner image is digest-pinned" precondition grepped the literal line CONFORMANCE_RUNNER_IMAGE="${…_REPO}@${…_DIGEST}" for 64 hex digits, so it could never pass. It now sources the file, evaluates the value, and rejects the zero placeholder.

• conformance/tests/l0-static/alert-rules-unit-tested/check.sh: its "the toolbox build passes --build-arg PROMTOOL_VERSION" assertion now reads infra/devcontainer/build-image.sh instead of dev.sh. The gate caught this on the first run, because the recipe had moved.

• Docs: platform/pm-agent/README.md, runbooks/pm-agent.md (two rows), docs/plans/pm-agent-issue-to-pr.md (follow-up 1), conformance/incluster/README.md, infra/devcontainer/README.md, and a new "How images are built" section in .gitea/workflows/README.md.

BLOCKER — the runner cannot build images, and fixing that is a security decision

Measured on this PR's own CI run (images run 325): the job failed in 6 s with failed to connect to the docker API at unix:///var/run/docker.sock ... no such file. The job container is given no Docker daemon at all, for two independent reasons, both read from the exact versions deployed:

• act_runner v0.2.11 internal/app/cmd/daemon.go: when DOCKER_HOST has a scheme other than unix/npipe — here tcp://localhost:2376, the DinD sidecar — it sets cfg.Container.DockerHost = "-", which means "never mount into job containers".
• gitea/act v0.261.2 (act_runner's go.mod replace) pkg/container/docker_run.go sanitizeConfig: with valid_volumes empty (the runner has no config file, so the default []), hostConfig.Binds = []string{}. Every bind is dropped, the daemon socket included, and so is any volumes: a workflow asks for.

I assumed the opposite while designing (from act's getDockerDaemonSocketMountPath, which never gets the chance to run). That was wrong, and the PR run is what showed it.

Why I did not fix it in this PR. Any fix gives workflow code the DinD daemon. That daemon is privileged: true, so docker run --privileged -v /:/host from a job is root on the node. Today a malicious workflow — for example one pushed by a prompt-injected pm-agent worker, which holds a repo write token — runs in an unprivileged container and holds no cluster credential. After the fix it does not. That is a change to the security boundary, and it is the operator's call. It also cannot be tested from a PR anyway: the runner only changes after merge + Flux.

Options, for the operator:

1. Socket to every job (smallest change): in apps/wave-1/gitea-actions.yaml,

   • mount a shared emptyDir at /var/run in both the dind sidecar and the runner, so dockerd's own unix:///var/run/docker.sock is visible at the same path in both;
   • set the runner's DOCKER_HOST=unix:///var/run/docker.sock, and drop DOCKER_TLS_VERIFY and DOCKER_CERT_PATH, because TLS over a unix socket breaks the client;
   • add an act_runner config file (CONFIG_FILE) with container.valid_volumes: ["/var/run/docker.sock"] and runner.file: /data/.runner, so the existing registration is kept.

   Consequence: any workflow on any branch can reach node root.

2. Option 1 on a SECOND runner with its own label (e.g. image-builder) and its own DinD, with images.yaml runs-on that label. The gate runner stays socket-less, and image builds stop competing with the gate for the single slot and its 7 GB node disk. It is not a security boundary: Gitea cannot restrict which workflow may target a label, so any branch could still runs-on: image-builder.

3. A builder that is not a privileged daemon, e.g. a rootless BuildKit sidecar exposed to jobs. That bounds a job to BuildKit's rootless sandbox instead of node root. It needs the verification steps rewritten, because they run the image (as uid 65532, --read-only), which BuildKit alone does not do. This is a larger piece of work.

Nothing in this PR has to change for options 1 or 2 except runs-on for option 2. The driver already produced verified images with a stock docker 27 CLI (see What I verified).

Decisions and pushback

• The push credential is a PAT secret, REGISTRY_PUSH_TOKEN; the Actions token cannot push. Checked against Gitea v1.24.3 source:

  • services/auth/basic.go Basic.Verify turns a task token into the pseudo-user gitea-actions (ID −2) and sets no token scope;
  • routers/api/packages/container/container.go Authenticate (the /v2/token exchange every docker client performs) requires read:package/write:package/all scope, so it answers 401. The Actions token cannot even pull;
  • services/context/package.go determineAccessMode has // TODO: ActionUser permission check and no branch for that user. Even past login it would get at most read on a public org;
  • models/repo/repo_unit.go ActionsConfig has no token-permissions setting in 1.24.3.

  The workflow fails loudly when the secret is absent, copying the RENOVATE_TOKEN preflight pattern.

• regex vs custom.regex: both work. Switched to the canonical custom.regex. In Renovate:

  • lib/config/migrations/custom/enabled-managers-migration.ts rewrites regex → custom.regex;
  • that migration runs on every repository run (lib/workers/repository/init/merge.ts → migrateAndValidate), not only in the validator;
  • lib/modules/manager/index.ts getEnabledManagersList accepts either spelling;
  • custom.regex exists since 37.51.1 (renovatebot/renovate#24079).

• The open question hid a bigger bug: the Renovate config on main is invalid for the Renovate the workflow actually runs.

  • renovatebot/[email protected] defaults renovate-version to 39 (action.yml, src/docker.ts), which is 39.264.0.
  • The validator used in #79 was ghcr.io/renovatebot/renovate with no tag, i.e. latest, 44.167.0.
  • Measured: renovate:39.264.0 renovate-config-validator on main's renovate.json5 gives ERROR: Found errors in configuration with five errors, all managerFilePatterns. fileMatch was renamed in 40.2.0.
  • So once #80 fixes the filename, every run would still die at CONFIG_VALIDATION.
  • Fixed by pinning renovate-version: 44.167.0. This branch's config validates clean on it. The only --strict complaint is a pending migration of the pre-existing top-level "docker": {"enabled": true} block, which I left alone.

• Unworkable as briefed: tracking the worker Job template's digest.

  • infra/agent/worker-job-template.yaml is COPYed into agent-runner. A Renovate bump there changes agent-runner's inputs, CI publishes a new digest, Renovate bumps it again, and so on forever.
  • pm-dispatch overwrites that field from PM_WORKER_IMAGE at runtime (pm-dispatch.py:649), so nothing is lost by leaving it all-zero. It is commented as deliberately untracked.
  • For the same loop reason, conformance/versions.env is not an input of the conformance-runner build, and a pipeline-only change does not republish on main.

• Changed from the brief: retention is a script, not a Gitea cleanup rule.

  • Gitea 1.24.3 has no API for cleanup rules (none in routers/api/v1/api.go; web session+CSRF forms only, in routers/web/web.go). So a rule cannot be created, or drift-asserted, as code. ci/gitea-settings.sh was therefore not extended. A cheaper proxy (count versions per image) would need read:package on SETTINGS_API_TOKEN, which it lacks.
  • A built-in rule is also unsafe here. Gitea stores a manifest as a file of the TAG version it was pushed under (routers/api/packages/container/manifest.go createPackageAndVersion), and resolves a by-digest pull from those files (models/packages/container/search.go). Deleting the tag a deployed digest was pushed under breaks the @sha256: pull, and the rule's "newest N" knows nothing about what git pins. The script keeps every digest git pins.

• Auto-merge. Digest bumps of these images auto-merge on a green gate (the existing digest rule). The exception is the conformance-runner bump: conformance/versions.env is a sensitive path, so it waits for an administrator's /approve-override every time.

What I verified

• Local gate, pinned toolbox, from this worktree, committed HEAD. Command: ENGINE=<shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'. The shim adds the .git mount. Result: see the PR body for the rc and stage list. They are recorded there rather than here because this entry is part of the tree being gated.
• Local podman builds through the CI driver, run as MODE=verify ENGINE=podman BASE_SHA=$(git merge-base origin/main HEAD) .gitea/workflows/scripts/images.sh. It returned rc 0 and built and verified all three images:
  • toolbox: every tool on PATH, and the five pins matched;
  • agent-runner: PM_SELFTEST=1 pm-dispatch OK, and the stdin-fed probe-validation.sh gave "20 refusal(s), 0 failure(s)";
  • conformance-runner: all seven conformance-clone refusals.
• The docker-CLI code path, i.e. what CI runs. I ran the same driver inside docker:27-cli (ENGINE=docker, DOCKER_BUILDKIT=0) against podman's Docker-compatible API socket. Exit code 0; all three images built and verified (selftest OK; probe OK, 20 refusals). This covers the docker-specific calls (image inspect formats, run -i with stdin, --tmpfs, --read-only, --user). It does not cover BuildKit or a real DinD daemon.
• This PR's CI, run 325 (images): FAILED in 6 s, because there is no Docker daemon in the job container (see BLOCKER). Gate run 324 failed only on approval-policy (expected, sensitive paths). Every other stage passed there, including gitea-settings.
• Other driver paths, run locally:
  • agent-runner alone builds the toolbox as its base;
  • an empty diff builds nothing;
  • publish mode without a token dies before doing anything;
  • infra/agent/build-image.sh with the zero placeholder dies with build-locally instructions.
• Renovate, with the pinned 44.167.0 image, locally:
  • renovate-config-validator exits 0;
  • a --platform=local extract dry run finds all four marker references (custom regex manager, currentValue: main, currentDigest: sha256:000…);
  • a lookup dry run shows the kubernetes-manager duplicates as skipReason: disabled and the marker deps taking the "unsupported/unversioned value main → getDigest(main)" path. That lookup currently 404s because nothing is published yet.
• Renovate 39.264.0 rejects main's current renovate.json5. Validator output reproduced above.
• registry-retention.sh dry run against the live Gitea (read-only): zero versions per image, nothing to delete. The token exchange and the package listing both answered.
• Live, read-only measurements (kubectl and the Gitea API):
  • the runner pod is on k3s-prod-1, which had 7 GB nodefs available;
  • DinD holds only the 1.54 GB act image; buildx is v0.20.1; the daemon is 27.5.1 on overlay2 with cgroup v2;
  • Gitea /data is 9.7G, 1% used;
  • the Gateway's Envoy (Cilium) has streamIdleTimeout: 300s, maxStreamDuration: 0s, and no route timeout on the gitea route;
  • Gitea's LIMIT_SIZE_CONTAINER is unset (default −1 = unlimited, modules/setting/packages.go).

What I did NOT verify

• Nothing has been pushed to the registry, by me or by CI. The publish path has never run:

  • the registry HEAD/digest read-back;
  • the immutable-tag guard;
  • "already published at this commit" idempotence;
  • agent-runner FROM <toolbox>@sha256:;
  • docker login with a PAT.

  The first merge to main is the first test, and it needs REGISTRY_PUSH_TOKEN to exist first.

• Large-layer push through the Gateway is UNVERIFIED. Envoy has no request-body cap and no stream duration cap here. But with no route timeout set, Envoy's default 15 s upstream timeout applies from the end of the request to the end of the response. That covers Gitea finalising a blob: copying the upload into package storage on Longhorn. A ~250 MB compressed layer should finish well inside 15 s, but I could not measure it without pushing. If the first publish shows 504s on blob PUTs, set timeouts.request on the gitea HTTPRoute.

• DinD fit is estimated, not measured (no build has run there, see the BLOCKER):

  • memory: 2Gi is shared by the job container and every build container;
  • disk: the toolbox, agent-runner, conformance-runner, Fedora base and build cache should total roughly 3 GB on top of the 1.5 GB act image, against 7 GB free on the node, with dind-storage having no sizeLimit.

• No image has been built in this runner's DinD, because it cannot be done until the BLOCKER is resolved. Two things are therefore unmeasured until then:

  • DinD memory (2Gi) and disk fit;
  • BuildKit (buildx v0.20.1 is on the daemon) as opposed to the legacy builder I tested with.

• Renovate has still never completed a run here. Three things block it:

  • #80, the configurationFile name;
  • RENOVATE_TOKEN gaining read:package;
  • this branch's version pin.

  renovatebot/[email protected] running a 44.x image is untested. The action only passes the tag to docker run.

• Pods cannot pull these images yet regardless. The kubelet pull credential is PR #61 (pm-agent-registry dockerconfigjson, read:package), still open. Until it merges, a correct digest still ends in ImagePullBackOff (401), as before.

• registry-retention.sh --apply has never run, and its keep logic has only run against an empty registry.

• git diff change detection on Gitea push events relies on github.event.before. A force-push or a first push falls back to building everything, which is intended but not observed.