diff --git a/.gitea/workflows/images.yaml b/.gitea/workflows/images.yaml index 3d027bc..6170fe4 100644 --- a/.gitea/workflows/images.yaml +++ b/.gitea/workflows/images.yaml @@ -41,6 +41,15 @@ # not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER` # is optional and only labels the login. # +# ┌─ BLOCKER, MEASURED ON PR #81 (run 325): THIS RUNNER CANNOT BUILD IMAGES YET ───────────────┐ +# │ The job container gets no Docker daemon. act_runner 0.2.11 forces container.docker_host to │ +# │ "-" whenever DOCKER_HOST is tcp:// (internal/app/cmd/daemon.go), and with valid_volumes: [] │ +# │ gitea/act strips every bind including the socket (pkg/container/docker_run.go │ +# │ sanitizeConfig). So this job fails at its first check, loudly, until the operator chooses │ +# │ how workflow code may reach a daemon — see decisions/2026-10-29-image-pipeline.md. It is │ +# │ not a required check, so it does not block merges; it is red, which is the honest state. │ +# └──────────────────────────────────────────────────────────────────────────────────────────────┘ +# # THE RUNNER (apps/wave-1/gitea-actions.yaml): act_runner 0.2.11, one job at a time, Docker # executor on a privileged `docker:27-dind` sidecar whose image store is an emptyDir on the node # (no sizeLimit) and whose memory limit is 2Gi — shared by the job container AND every build diff --git a/.gitea/workflows/scripts/images.sh b/.gitea/workflows/scripts/images.sh index 2d83fb4..a36a950 100755 --- a/.gitea/workflows/scripts/images.sh +++ b/.gitea/workflows/scripts/images.sh @@ -200,8 +200,13 @@ push_one() { # -> sets DIGEST[image] to the registry-returned digest log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}" command -v "${ENGINE}" >/dev/null || die "${ENGINE} is not on PATH in the job container" "${ENGINE}" info >/dev/null 2>&1 \ - || die "the ${ENGINE} daemon is not reachable from the job container — the runner did not - hand it a Docker socket (act_runner container.docker_host). Nothing can be built." + || die "the ${ENGINE} daemon is not reachable from the job container, so nothing can be built. + MEASURED ON THIS RUNNER (PR #81, 2026-10-29): act_runner 0.2.11 never gives a job a + daemon. Its DOCKER_HOST is tcp:// (the DinD sidecar), and internal/app/cmd/daemon.go then + forces container.docker_host to '-' (\"can't be mounted\"); separately, with + valid_volumes: [] gitea/act's sanitizeConfig drops EVERY bind, the socket included. + Fixing it means handing workflow code the privileged DinD daemon — an operator security + decision, set out in decisions/2026-10-29-image-pipeline.md. Not a fault in this script." "${ENGINE}" version 2>/dev/null | sed -n '1,12p' || true "${ENGINE}" system df || true diff --git a/decisions/2026-10-29-image-pipeline.md b/decisions/2026-10-29-image-pipeline.md index 7fa2979..32e5a68 100644 --- a/decisions/2026-10-29-image-pipeline.md +++ b/decisions/2026-10-29-image-pipeline.md @@ -83,6 +83,56 @@ both `build-image.sh` scripts said "publishing is an operator action". `infra/devcontainer/README.md`, and a new "How images are built" section in `.gitea/workflows/README.md`. +### BLOCKER — the runner cannot build images, and fixing that is a security decision + +**Measured on this PR's own CI run (images run 325): the job failed in 6 s** with +`failed to connect to the docker API at unix:///var/run/docker.sock ... no such file`. The job +container is given **no Docker daemon at all**, for two independent reasons, both read from the +exact versions deployed: + +- act_runner **v0.2.11** `internal/app/cmd/daemon.go`: when `DOCKER_HOST` has a scheme other than + `unix`/`npipe` — here `tcp://localhost:2376`, the DinD sidecar — it sets + `cfg.Container.DockerHost = "-"`, which means "never mount into job containers". +- gitea/act **v0.261.2** (act_runner's `go.mod` replace) `pkg/container/docker_run.go` + `sanitizeConfig`: with `valid_volumes` empty (the runner has no config file, so the default + `[]`), `hostConfig.Binds = []string{}`. **Every** bind is dropped, the daemon socket included, + and so is any `volumes:` a workflow asks for. + +I assumed the opposite while designing (from act's `getDockerDaemonSocketMountPath`, which never +gets the chance to run). That was wrong, and the PR run is what showed it. + +**Why I did not fix it in this PR.** Any fix gives workflow code the DinD daemon. That daemon is +`privileged: true`, so `docker run --privileged -v /:/host` from a job is **root on the node**. +Today a malicious workflow — for example one pushed by a prompt-injected pm-agent worker, which +holds a repo write token — runs in an unprivileged container and holds no cluster credential. +After the fix it does not. That is a change to the security boundary, and it is the operator's +call. It also cannot be tested from a PR anyway: the runner only changes after merge + Flux. + +**Options, for the operator:** + +1. **Socket to every job (smallest change):** in `apps/wave-1/gitea-actions.yaml`, + - mount a shared `emptyDir` at `/var/run` in both the `dind` sidecar and the `runner`, so + dockerd's own `unix:///var/run/docker.sock` is visible at the same path in both; + - set the runner's `DOCKER_HOST=unix:///var/run/docker.sock`, and drop `DOCKER_TLS_VERIFY` + and `DOCKER_CERT_PATH`, because TLS over a unix socket breaks the client; + - add an act_runner config file (`CONFIG_FILE`) with `container.valid_volumes: + ["/var/run/docker.sock"]` and `runner.file: /data/.runner`, so the existing registration + is kept. + + Consequence: any workflow on any branch can reach node root. +2. **Option 1 on a SECOND runner** with its own label (e.g. `image-builder`) and its own DinD, + with `images.yaml` `runs-on` that label. The gate runner stays socket-less, and image builds + stop competing with the gate for the single slot and its 7 GB node disk. **It is not a + security boundary**: Gitea cannot restrict which workflow may target a label, so any branch + could still `runs-on: image-builder`. +3. **A builder that is not a privileged daemon**, e.g. a rootless BuildKit sidecar exposed to + jobs. That bounds a job to BuildKit's rootless sandbox instead of node root. It needs the + verification steps rewritten, because they *run* the image (as uid 65532, `--read-only`), + which BuildKit alone does not do. This is a larger piece of work. + +Nothing in this PR has to change for options 1 or 2 except `runs-on` for option 2. The driver +already produced verified images with a stock docker 27 CLI (see *What I verified*). + ### Decisions and pushback - **The push credential is a PAT secret, `REGISTRY_PUSH_TOKEN`; the Actions token cannot push.** @@ -156,6 +206,14 @@ both `build-image.sh` scripts said "publishing is an operator action". - agent-runner: `PM_SELFTEST=1 pm-dispatch` OK, and the stdin-fed `probe-validation.sh` gave "20 refusal(s), 0 failure(s)"; - conformance-runner: all seven `conformance-clone` refusals. +- **The docker-CLI code path**, i.e. what CI runs. I ran the same driver inside + `docker:27-cli` (`ENGINE=docker`, `DOCKER_BUILDKIT=0`) against podman's Docker-compatible API + socket. Exit code 0; all three images built and verified (selftest OK; probe OK, 20 refusals). + This covers the docker-specific calls (`image inspect` formats, `run -i` with stdin, + `--tmpfs`, `--read-only`, `--user`). It does **not** cover BuildKit or a real DinD daemon. +- **This PR's CI, run 325 (`images`): FAILED in 6 s**, because there is no Docker daemon in the + job container (see BLOCKER). Gate run 324 failed only on `approval-policy` (expected, + sensitive paths). Every other stage passed there, including `gitea-settings`. - **Other driver paths, run locally:** - agent-runner alone builds the toolbox as its base; - an empty diff builds nothing; @@ -198,13 +256,16 @@ both `build-image.sh` scripts said "publishing is an operator action". finalising a blob: copying the upload into package storage on Longhorn. A ~250 MB compressed layer should finish well inside 15 s, but I could not measure it without pushing. If the first publish shows 504s on blob PUTs, set `timeouts.request` on the gitea HTTPRoute. -- **DinD fit is estimated, not measured** (until the PR run reports): +- **DinD fit is estimated, not measured** (no build has run there, see the BLOCKER): - memory: 2Gi is shared by the job container and every build container; - disk: the toolbox, agent-runner, conformance-runner, Fedora base and build cache should total roughly 3 GB on top of the 1.5 GB act image, against 7 GB free on the node, with `dind-storage` having no sizeLimit. -- **That act_runner hands the job container a Docker socket is inferred** from act_runner's - DOCKER_HOST handling, not measured. The driver fails loudly if it does not. +- **No image has been built in this runner's DinD**, because it cannot be done until the BLOCKER + is resolved. Two things are therefore unmeasured until then: + - DinD memory (2Gi) and disk fit; + - BuildKit (`buildx` v0.20.1 is on the daemon) as opposed to the legacy builder I tested + with. - **Renovate has still never completed a run here.** Three things block it: - #80, the `configurationFile` name; - `RENOVATE_TOKEN` gaining `read:package`;