fix(images): name the measured runner blocker; the job container gets no Docker daemon
PR #81's first images run failed in 6s: act_runner 0.2.11 forces docker_host '-' for a tcp:// DOCKER_HOST, and gitea/act drops every bind when valid_volumes is empty. Giving jobs the privileged DinD daemon is node-root for any workflow author, so it is left to the operator; options are in the handoff entry. The driver's failure now names the cause. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
0da34ddbc6
commit
65a7769f55
3 files changed
+80
-5
No files matched your search
@@ -41,6 +41,15 @@
|
||||
# not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER`
|
||||
# is optional and only labels the login.
|
||||
#
|
||||
# ┌─ BLOCKER, MEASURED ON PR #81 (run 325): THIS RUNNER CANNOT BUILD IMAGES YET ───────────────┐
|
||||
# │ The job container gets no Docker daemon. act_runner 0.2.11 forces container.docker_host to │
|
||||
# │ "-" whenever DOCKER_HOST is tcp:// (internal/app/cmd/daemon.go), and with valid_volumes: [] │
|
||||
# │ gitea/act strips every bind including the socket (pkg/container/docker_run.go │
|
||||
# │ sanitizeConfig). So this job fails at its first check, loudly, until the operator chooses │
|
||||
# │ how workflow code may reach a daemon — see decisions/2026-10-29-image-pipeline.md. It is │
|
||||
# │ not a required check, so it does not block merges; it is red, which is the honest state. │
|
||||
# └──────────────────────────────────────────────────────────────────────────────────────────────┘
|
||||
#
|
||||
# THE RUNNER (apps/wave-1/gitea-actions.yaml): act_runner 0.2.11, one job at a time, Docker
|
||||
# executor on a privileged `docker:27-dind` sidecar whose image store is an emptyDir on the node
|
||||
# (no sizeLimit) and whose memory limit is 2Gi — shared by the job container AND every build
|
||||
|
||||
@@ -200,8 +200,13 @@ push_one() { # <image> -> sets DIGEST[image] to the registry-returned digest
|
||||
log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}"
|
||||
command -v "${ENGINE}" >/dev/null || die "${ENGINE} is not on PATH in the job container"
|
||||
"${ENGINE}" info >/dev/null 2>&1 \
|
||||
|| die "the ${ENGINE} daemon is not reachable from the job container — the runner did not
|
||||
hand it a Docker socket (act_runner container.docker_host). Nothing can be built."
|
||||
|| die "the ${ENGINE} daemon is not reachable from the job container, so nothing can be built.
|
||||
MEASURED ON THIS RUNNER (PR #81, 2026-10-29): act_runner 0.2.11 never gives a job a
|
||||
daemon. Its DOCKER_HOST is tcp:// (the DinD sidecar), and internal/app/cmd/daemon.go then
|
||||
forces container.docker_host to '-' (\"can't be mounted\"); separately, with
|
||||
valid_volumes: [] gitea/act's sanitizeConfig drops EVERY bind, the socket included.
|
||||
Fixing it means handing workflow code the privileged DinD daemon — an operator security
|
||||
decision, set out in decisions/2026-10-29-image-pipeline.md. Not a fault in this script."
|
||||
"${ENGINE}" version 2>/dev/null | sed -n '1,12p' || true
|
||||
"${ENGINE}" system df || true
|
||||
|
||||
|
||||
@@ -83,6 +83,56 @@ both `build-image.sh` scripts said "publishing is an operator action".
|
||||
`infra/devcontainer/README.md`, and a new "How images are built" section in
|
||||
`.gitea/workflows/README.md`.
|
||||
|
||||
### BLOCKER — the runner cannot build images, and fixing that is a security decision
|
||||
|
||||
**Measured on this PR's own CI run (images run 325): the job failed in 6 s** with
|
||||
`failed to connect to the docker API at unix:///var/run/docker.sock ... no such file`. The job
|
||||
container is given **no Docker daemon at all**, for two independent reasons, both read from the
|
||||
exact versions deployed:
|
||||
|
||||
- act_runner **v0.2.11** `internal/app/cmd/daemon.go`: when `DOCKER_HOST` has a scheme other than
|
||||
`unix`/`npipe` — here `tcp://localhost:2376`, the DinD sidecar — it sets
|
||||
`cfg.Container.DockerHost = "-"`, which means "never mount into job containers".
|
||||
- gitea/act **v0.261.2** (act_runner's `go.mod` replace) `pkg/container/docker_run.go`
|
||||
`sanitizeConfig`: with `valid_volumes` empty (the runner has no config file, so the default
|
||||
`[]`), `hostConfig.Binds = []string{}`. **Every** bind is dropped, the daemon socket included,
|
||||
and so is any `volumes:` a workflow asks for.
|
||||
|
||||
I assumed the opposite while designing (from act's `getDockerDaemonSocketMountPath`, which never
|
||||
gets the chance to run). That was wrong, and the PR run is what showed it.
|
||||
|
||||
**Why I did not fix it in this PR.** Any fix gives workflow code the DinD daemon. That daemon is
|
||||
`privileged: true`, so `docker run --privileged -v /:/host` from a job is **root on the node**.
|
||||
Today a malicious workflow — for example one pushed by a prompt-injected pm-agent worker, which
|
||||
holds a repo write token — runs in an unprivileged container and holds no cluster credential.
|
||||
After the fix it does not. That is a change to the security boundary, and it is the operator's
|
||||
call. It also cannot be tested from a PR anyway: the runner only changes after merge + Flux.
|
||||
|
||||
**Options, for the operator:**
|
||||
|
||||
1. **Socket to every job (smallest change):** in `apps/wave-1/gitea-actions.yaml`,
|
||||
- mount a shared `emptyDir` at `/var/run` in both the `dind` sidecar and the `runner`, so
|
||||
dockerd's own `unix:///var/run/docker.sock` is visible at the same path in both;
|
||||
- set the runner's `DOCKER_HOST=unix:///var/run/docker.sock`, and drop `DOCKER_TLS_VERIFY`
|
||||
and `DOCKER_CERT_PATH`, because TLS over a unix socket breaks the client;
|
||||
- add an act_runner config file (`CONFIG_FILE`) with `container.valid_volumes:
|
||||
["/var/run/docker.sock"]` and `runner.file: /data/.runner`, so the existing registration
|
||||
is kept.
|
||||
|
||||
Consequence: any workflow on any branch can reach node root.
|
||||
2. **Option 1 on a SECOND runner** with its own label (e.g. `image-builder`) and its own DinD,
|
||||
with `images.yaml` `runs-on` that label. The gate runner stays socket-less, and image builds
|
||||
stop competing with the gate for the single slot and its 7 GB node disk. **It is not a
|
||||
security boundary**: Gitea cannot restrict which workflow may target a label, so any branch
|
||||
could still `runs-on: image-builder`.
|
||||
3. **A builder that is not a privileged daemon**, e.g. a rootless BuildKit sidecar exposed to
|
||||
jobs. That bounds a job to BuildKit's rootless sandbox instead of node root. It needs the
|
||||
verification steps rewritten, because they *run* the image (as uid 65532, `--read-only`),
|
||||
which BuildKit alone does not do. This is a larger piece of work.
|
||||
|
||||
Nothing in this PR has to change for options 1 or 2 except `runs-on` for option 2. The driver
|
||||
already produced verified images with a stock docker 27 CLI (see *What I verified*).
|
||||
|
||||
### Decisions and pushback
|
||||
|
||||
- **The push credential is a PAT secret, `REGISTRY_PUSH_TOKEN`; the Actions token cannot push.**
|
||||
@@ -156,6 +206,14 @@ both `build-image.sh` scripts said "publishing is an operator action".
|
||||
- agent-runner: `PM_SELFTEST=1 pm-dispatch` OK, and the stdin-fed `probe-validation.sh`
|
||||
gave "20 refusal(s), 0 failure(s)";
|
||||
- conformance-runner: all seven `conformance-clone` refusals.
|
||||
- **The docker-CLI code path**, i.e. what CI runs. I ran the same driver inside
|
||||
`docker:27-cli` (`ENGINE=docker`, `DOCKER_BUILDKIT=0`) against podman's Docker-compatible API
|
||||
socket. Exit code 0; all three images built and verified (selftest OK; probe OK, 20 refusals).
|
||||
This covers the docker-specific calls (`image inspect` formats, `run -i` with stdin,
|
||||
`--tmpfs`, `--read-only`, `--user`). It does **not** cover BuildKit or a real DinD daemon.
|
||||
- **This PR's CI, run 325 (`images`): FAILED in 6 s**, because there is no Docker daemon in the
|
||||
job container (see BLOCKER). Gate run 324 failed only on `approval-policy` (expected,
|
||||
sensitive paths). Every other stage passed there, including `gitea-settings`.
|
||||
- **Other driver paths, run locally:**
|
||||
- agent-runner alone builds the toolbox as its base;
|
||||
- an empty diff builds nothing;
|
||||
@@ -198,13 +256,16 @@ both `build-image.sh` scripts said "publishing is an operator action".
|
||||
finalising a blob: copying the upload into package storage on Longhorn. A ~250 MB compressed
|
||||
layer should finish well inside 15 s, but I could not measure it without pushing. If the first
|
||||
publish shows 504s on blob PUTs, set `timeouts.request` on the gitea HTTPRoute.
|
||||
- **DinD fit is estimated, not measured** (until the PR run reports):
|
||||
- **DinD fit is estimated, not measured** (no build has run there, see the BLOCKER):
|
||||
- memory: 2Gi is shared by the job container and every build container;
|
||||
- disk: the toolbox, agent-runner, conformance-runner, Fedora base and build cache should
|
||||
total roughly 3 GB on top of the 1.5 GB act image, against 7 GB free on the node, with
|
||||
`dind-storage` having no sizeLimit.
|
||||
- **That act_runner hands the job container a Docker socket is inferred** from act_runner's
|
||||
DOCKER_HOST handling, not measured. The driver fails loudly if it does not.
|
||||
- **No image has been built in this runner's DinD**, because it cannot be done until the BLOCKER
|
||||
is resolved. Two things are therefore unmeasured until then:
|
||||
- DinD memory (2Gi) and disk fit;
|
||||
- BuildKit (`buildx` v0.20.1 is on the daemon) as opposed to the legacy builder I tested
|
||||
with.
|
||||
- **Renovate has still never completed a run here.** Three things block it:
|
||||
- #80, the `configurationFile` name;
|
||||
- `RENOVATE_TOKEN` gaining `read:package`;
|
||||
|
||||
Reference in new issue
Block a user