fix(images): name the measured runner blocker; the job container gets no Docker daemon

PR #81's first images run failed in 6s: act_runner 0.2.11 forces docker_host '-' for a tcp://
DOCKER_HOST, and gitea/act drops every bind when valid_volumes is empty. Giving jobs the
privileged DinD daemon is node-root for any workflow author, so it is left to the operator;
options are in the handoff entry. The driver's failure now names the cause.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed 2026-10-29 23:59:55 -05:00
1 parent e539109830
commit 8f35234fcc
3 files changed
+80 -5

No files matched your search

+9
View File
@@ -41,6 +41,15 @@
# not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER`
# is optional and only labels the login.
#
# ┌─ BLOCKER, MEASURED ON PR #81 (run 325): THIS RUNNER CANNOT BUILD IMAGES YET ───────────────┐
# │ The job container gets no Docker daemon. act_runner 0.2.11 forces container.docker_host to │
# │ "-" whenever DOCKER_HOST is tcp:// (internal/app/cmd/daemon.go), and with valid_volumes: [] │
# │ gitea/act strips every bind including the socket (pkg/container/docker_run.go │
# │ sanitizeConfig). So this job fails at its first check, loudly, until the operator chooses │
# │ how workflow code may reach a daemon — see decisions/2026-10-29-image-pipeline.md. It is │
# │ not a required check, so it does not block merges; it is red, which is the honest state. │
# └──────────────────────────────────────────────────────────────────────────────────────────────┘
#
# THE RUNNER (apps/wave-1/gitea-actions.yaml): act_runner 0.2.11, one job at a time, Docker
# executor on a privileged `docker:27-dind` sidecar whose image store is an emptyDir on the node
# (no sizeLimit) and whose memory limit is 2Gi — shared by the job container AND every build
+7 -2
View File
@@ -200,8 +200,13 @@ push_one() { # <image> -> sets DIGEST[image] to the registry-returned digest
log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}"
command -v "${ENGINE}" >/dev/null || die "${ENGINE} is not on PATH in the job container"
"${ENGINE}" info >/dev/null 2>&1 \
|| die "the ${ENGINE} daemon is not reachable from the job container — the runner did not
hand it a Docker socket (act_runner container.docker_host). Nothing can be built."
|| die "the ${ENGINE} daemon is not reachable from the job container, so nothing can be built.
MEASURED ON THIS RUNNER (PR #81, 2026-10-29): act_runner 0.2.11 never gives a job a
daemon. Its DOCKER_HOST is tcp:// (the DinD sidecar), and internal/app/cmd/daemon.go then
forces container.docker_host to '-' (\"can't be mounted\"); separately, with
valid_volumes: [] gitea/act's sanitizeConfig drops EVERY bind, the socket included.
Fixing it means handing workflow code the privileged DinD daemon — an operator security
decision, set out in decisions/2026-10-29-image-pipeline.md. Not a fault in this script."
"${ENGINE}" version 2>/dev/null | sed -n '1,12p' || true
"${ENGINE}" system df || true
+64 -3
View File
@@ -83,6 +83,56 @@ both `build-image.sh` scripts said "publishing is an operator action".
`infra/devcontainer/README.md`, and a new "How images are built" section in
`.gitea/workflows/README.md`.
### BLOCKER — the runner cannot build images, and fixing that is a security decision
**Measured on this PR's own CI run (images run 325): the job failed in 6 s** with
`failed to connect to the docker API at unix:///var/run/docker.sock ... no such file`. The job
container is given **no Docker daemon at all**, for two independent reasons, both read from the
exact versions deployed:
- act_runner **v0.2.11** `internal/app/cmd/daemon.go`: when `DOCKER_HOST` has a scheme other than
`unix`/`npipe` — here `tcp://localhost:2376`, the DinD sidecar — it sets
`cfg.Container.DockerHost = "-"`, which means "never mount into job containers".
- gitea/act **v0.261.2** (act_runner's `go.mod` replace) `pkg/container/docker_run.go`
`sanitizeConfig`: with `valid_volumes` empty (the runner has no config file, so the default
`[]`), `hostConfig.Binds = []string{}`. **Every** bind is dropped, the daemon socket included,
and so is any `volumes:` a workflow asks for.
I assumed the opposite while designing (from act's `getDockerDaemonSocketMountPath`, which never
gets the chance to run). That was wrong, and the PR run is what showed it.
**Why I did not fix it in this PR.** Any fix gives workflow code the DinD daemon. That daemon is
`privileged: true`, so `docker run --privileged -v /:/host` from a job is **root on the node**.
Today a malicious workflow — for example one pushed by a prompt-injected pm-agent worker, which
holds a repo write token — runs in an unprivileged container and holds no cluster credential.
After the fix it does not. That is a change to the security boundary, and it is the operator's
call. It also cannot be tested from a PR anyway: the runner only changes after merge + Flux.
**Options, for the operator:**
1. **Socket to every job (smallest change):** in `apps/wave-1/gitea-actions.yaml`,
- mount a shared `emptyDir` at `/var/run` in both the `dind` sidecar and the `runner`, so
dockerd's own `unix:///var/run/docker.sock` is visible at the same path in both;
- set the runner's `DOCKER_HOST=unix:///var/run/docker.sock`, and drop `DOCKER_TLS_VERIFY`
and `DOCKER_CERT_PATH`, because TLS over a unix socket breaks the client;
- add an act_runner config file (`CONFIG_FILE`) with `container.valid_volumes:
["/var/run/docker.sock"]` and `runner.file: /data/.runner`, so the existing registration
is kept.
Consequence: any workflow on any branch can reach node root.
2. **Option 1 on a SECOND runner** with its own label (e.g. `image-builder`) and its own DinD,
with `images.yaml` `runs-on` that label. The gate runner stays socket-less, and image builds
stop competing with the gate for the single slot and its 7 GB node disk. **It is not a
security boundary**: Gitea cannot restrict which workflow may target a label, so any branch
could still `runs-on: image-builder`.
3. **A builder that is not a privileged daemon**, e.g. a rootless BuildKit sidecar exposed to
jobs. That bounds a job to BuildKit's rootless sandbox instead of node root. It needs the
verification steps rewritten, because they *run* the image (as uid 65532, `--read-only`),
which BuildKit alone does not do. This is a larger piece of work.
Nothing in this PR has to change for options 1 or 2 except `runs-on` for option 2. The driver
already produced verified images with a stock docker 27 CLI (see *What I verified*).
### Decisions and pushback
- **The push credential is a PAT secret, `REGISTRY_PUSH_TOKEN`; the Actions token cannot push.**
@@ -156,6 +206,14 @@ both `build-image.sh` scripts said "publishing is an operator action".
- agent-runner: `PM_SELFTEST=1 pm-dispatch` OK, and the stdin-fed `probe-validation.sh`
gave "20 refusal(s), 0 failure(s)";
- conformance-runner: all seven `conformance-clone` refusals.
- **The docker-CLI code path**, i.e. what CI runs. I ran the same driver inside
`docker:27-cli` (`ENGINE=docker`, `DOCKER_BUILDKIT=0`) against podman's Docker-compatible API
socket. Exit code 0; all three images built and verified (selftest OK; probe OK, 20 refusals).
This covers the docker-specific calls (`image inspect` formats, `run -i` with stdin,
`--tmpfs`, `--read-only`, `--user`). It does **not** cover BuildKit or a real DinD daemon.
- **This PR's CI, run 325 (`images`): FAILED in 6 s**, because there is no Docker daemon in the
job container (see BLOCKER). Gate run 324 failed only on `approval-policy` (expected,
sensitive paths). Every other stage passed there, including `gitea-settings`.
- **Other driver paths, run locally:**
- agent-runner alone builds the toolbox as its base;
- an empty diff builds nothing;
@@ -198,13 +256,16 @@ both `build-image.sh` scripts said "publishing is an operator action".
finalising a blob: copying the upload into package storage on Longhorn. A ~250 MB compressed
layer should finish well inside 15 s, but I could not measure it without pushing. If the first
publish shows 504s on blob PUTs, set `timeouts.request` on the gitea HTTPRoute.
- **DinD fit is estimated, not measured** (until the PR run reports):
- **DinD fit is estimated, not measured** (no build has run there, see the BLOCKER):
- memory: 2Gi is shared by the job container and every build container;
- disk: the toolbox, agent-runner, conformance-runner, Fedora base and build cache should
total roughly 3 GB on top of the 1.5 GB act image, against 7 GB free on the node, with
`dind-storage` having no sizeLimit.
- **That act_runner hands the job container a Docker socket is inferred** from act_runner's
DOCKER_HOST handling, not measured. The driver fails loudly if it does not.
- **No image has been built in this runner's DinD**, because it cannot be done until the BLOCKER
is resolved. Two things are therefore unmeasured until then:
- DinD memory (2Gi) and disk fit;
- BuildKit (`buildx` v0.20.1 is on the daemon) as opposed to the legacy builder I tested
with.
- **Renovate has still never completed a run here.** Three things block it:
- #80, the `configurationFile` name;
- `RENOVATE_TOKEN` gaining `read:package`;