feat(images): build on rootless BuildKit over mTLS; verification moves into Containerfile verify stages
Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched. - platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined, read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP :1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it). No Kyverno exception: every applied policy matches app-namespace=true only. - Each Containerfile names its shipped stage and ends in a `verify` stage holding every former `docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now. - images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes. Fails loudly when the builder secrets are absent or buildkitd is unreachable. - infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed
1 parent
65a7769f55
commit
310deb0f7d
28 files changed
+1490
-608
No files matched your search
@@ -1,7 +1,7 @@
|
||||
---
|
||||
date: 2026-10-29
|
||||
branch: feat/image-pipeline
|
||||
title: "Images are built, verified and published by CI; Renovate carries the digests into git"
|
||||
title: "Images are built, verified and published by CI on rootless BuildKit; Renovate carries the digests into git"
|
||||
---
|
||||
|
||||
## What changed
|
||||
@@ -11,127 +11,177 @@ the codebase, never to a one-off workstation build/push. Before this branch the
|
||||
package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and
|
||||
both `build-image.sh` scripts said "publishing is an operator action".
|
||||
|
||||
- **`.gitea/workflows/images.yaml`** + **`.gitea/workflows/scripts/images.sh`** (the logic).
|
||||
- Pull request: build and verify only. No push, and the step references no secret at all.
|
||||
- Push to `main`, or `workflow_dispatch` on `main`: build, verify, then push to
|
||||
`gitea.home.keithjackson.dev/home-network/{toolbox,agent-runner,conformance-runner}`.
|
||||
Tags are `sha-<12 hex>` (immutable: the driver refuses to overwrite one) and `main` (moving).
|
||||
The digest is read back from the registry with a HEAD on the manifest, compared with what the
|
||||
client printed, and written to the job summary.
|
||||
- Only images whose inputs changed are built. A re-run on the same commit skips images that are
|
||||
already published.
|
||||
- agent-runner builds FROM the toolbox built in the same run (on main, FROM its just-pushed
|
||||
digest). Otherwise:
|
||||
- in verify mode the toolbox is built anyway as its base, because a PR holds no credential;
|
||||
- in publish mode it uses `AGENT_BASE_IMAGE` from `infra/agent/versions.env`;
|
||||
- if that is still the zero placeholder, the toolbox is built and published first. This is
|
||||
how the first run on main bootstraps.
|
||||
- An always-run step prunes the DinD image store.
|
||||
- **One build recipe per image, reused by CI.**
|
||||
- New `infra/devcontainer/build-image.sh`. `dev.sh build` now delegates to it. It has an
|
||||
in-image verification step that the toolbox never had: every tool is on PATH, and
|
||||
promtool, kubectl, chainsaw, kyverno and sops report their pins.
|
||||
- `infra/agent/build-image.sh` and `conformance/incluster/build-image.sh` were made
|
||||
engine-neutral, because CI runs `ENGINE=docker` against DinD:
|
||||
- `podman image exists` became `image inspect`, with a pull fallback;
|
||||
- `{{.Digest}}` and `--digestfile` are gone. No local digest is printed, since only the
|
||||
registry's digest is meaningful;
|
||||
- `probe-validation.sh` is fed over **stdin** instead of a bind mount. Under DinD a bind
|
||||
source is resolved on the daemon's filesystem, which would silently mount an empty
|
||||
directory;
|
||||
- `probe-validation.sh` itself gained `${BASH_SOURCE[0]:-$0}` so it survives `bash -s`
|
||||
under `set -u`.
|
||||
- `AGENT_BASE_IMAGE` and `IMAGE_TAG` can be overridden from the environment, and
|
||||
`BUILD_REVISION` adds an `org.opencontainers.image.revision` label.
|
||||
- The "publishing is an operator action" text was replaced throughout.
|
||||
- **Digest references.** Each now carries a `# renovate-digest: image=<repo> tag=main` marker
|
||||
on the line above it:
|
||||
- `platform/pm-agent/deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`), which share
|
||||
one dependency, so they bump in one PR;
|
||||
- `infra/agent/versions.env` `AGENT_BASE_IMAGE`, which was `home-k3s-dev:local` and is now
|
||||
the published toolbox by digest (zero placeholder until the first publish);
|
||||
- `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to the
|
||||
zero placeholder because Renovate can only replace a digest it can match.
|
||||
### The builder: rootless BuildKit — DECISION MADE (operator, on PR #81)
|
||||
|
||||
The admission policies are **untouched**; the `<repo>@sha256:<digest>` form is preserved.
|
||||
**The problem it answers, measured on this PR's first CI run (images run 325, failed in 6 s):** a
|
||||
CI job on this runner gets **no Docker daemon at all**, for two reasons, both in the deployed
|
||||
versions' source:
|
||||
- act_runner v0.2.11 `internal/app/cmd/daemon.go` sets `container.docker_host` to `-` ("never
|
||||
mount into jobs") whenever `DOCKER_HOST` is `tcp://`, which it is here (the DinD sidecar);
|
||||
- gitea/act v0.261.2 `pkg/container/docker_run.go` `sanitizeConfig` drops **every** bind,
|
||||
socket included, when `valid_volumes` is empty, which it is (the runner has no config file).
|
||||
|
||||
Giving jobs the DinD daemon would have made every branch push root on the node, because that
|
||||
daemon is `privileged: true`.
|
||||
|
||||
**The decision:** no job may reach a privileged daemon, and images are built by a **rootless
|
||||
BuildKit** daemon over mTLS. The runner (`apps/wave-1/gitea-actions.yaml`, DOCKER_HOST,
|
||||
valid_volumes) is **untouched**.
|
||||
|
||||
- **`platform/buildkit/`** — a new hand-managed component in its own namespace `buildkit`. The
|
||||
runner manifest is renderer-generated (`catalog/render.py` EXTRA_UNITS), and the builder has no
|
||||
business sitting next to a privileged container. Flux reconciles it as the new Kustomization
|
||||
`buildkit` (`clusters/base/buildkit.yaml`, `dependsOn: platform`).
|
||||
- **Deployment:** `moby/buildkit:v0.34.1-rootless@sha256:f68752a5…0c4f89`, pinned by index
|
||||
digest. The securityContext, every line measured (see *What I verified*):
|
||||
- `privileged: false`;
|
||||
- uid/gid 1000, `runAsNonRoot`;
|
||||
- `capabilities: drop ALL, add SETUID, SETGID`;
|
||||
- `allowPrivilegeEscalation: true`;
|
||||
- `seccompProfile: Unconfined` and `appArmorProfile: Unconfined`;
|
||||
- `readOnlyRootFilesystem: true`, with five emptyDirs.
|
||||
|
||||
Other settings:
|
||||
- `--oci-worker-no-process-sandbox`;
|
||||
- no ServiceAccount token;
|
||||
- memory limit 4Gi, which covers build steps too;
|
||||
- cache emptyDir with `sizeLimit: 6Gi`, and GC capped at 4 GB in `buildkitd.toml`
|
||||
(ConfigMap).
|
||||
- **mTLS via cert-manager** (`certificates.yaml`): a self-signed CA used only here, the server
|
||||
cert `buildkitd-tls`, and the client cert `buildkit-client-ci`. buildkitd verifies every
|
||||
client against that CA.
|
||||
- **Service:** ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`.
|
||||
- **NetworkPolicies:**
|
||||
- default-deny;
|
||||
- ingress on 1234 **only from the act-runner pod** (`gitea-actions`,
|
||||
`app.kubernetes.io/name=act-runner`);
|
||||
- egress: DNS, world :443/:80 for downloads, and the `ingress` entity (the Gateway) for the
|
||||
registry push.
|
||||
- `versions.env`: the CI-side client, `buildx` v0.38.1 with the sha256 from its
|
||||
`checksums.txt`, plus the address and certificate name CI dials.
|
||||
- **How DinD job containers appear to Cilium (measured, read-only).**
|
||||
`iptables -t nat -S POSTROUTING` in the dind container shows
|
||||
`-A POSTROUTING -s 172.18.0.0/16 ! -o br-… -j MASQUERADE` (and the same for docker0). So a job's
|
||||
connection leaves with the **act-runner pod's IP, and therefore that pod's Cilium identity**.
|
||||
That makes "only the act-runner pod" expressible and correct, but it cannot tell one job from
|
||||
another, or a job from act_runner itself. Per-job authentication is the client certificate.
|
||||
- **Admission: no Kyverno exception needed.**
|
||||
- Every ClusterPolicy actually applied (`platform-config/kustomization.yaml`; live:
|
||||
`conformance-trigger-job-shape`, `label-cdi-worker-pods`, `pm-agent-job-shape`,
|
||||
`require-auth-mode-label`, `require-default-deny-baseline`, `require-egress-class-label`,
|
||||
`restrict-host-namespaces`, `secret-provenance`, `validate-default-deny-shape`) matches only
|
||||
namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`, or only its own
|
||||
namespace in the job-shape policies' case. `buildkit` carries no such label.
|
||||
- There are no namespaced Kyverno Policies and no cluster-wide PSA default (both measured).
|
||||
- The namespace is deliberately left without a PSA label, because PSS `baseline` forbids
|
||||
`Unconfined` seccomp and AppArmor.
|
||||
- `validate-default-deny-shape` does apply to the namespace's default-deny NetworkPolicy, and
|
||||
its shape (both directions) satisfies it.
|
||||
- **Unprivileged user namespaces on the nodes (measured, read-only, via Ansible `sysctl`).** On
|
||||
all three Debian 13.7 nodes, kernel `7.1.8+deb13-amd64`:
|
||||
- `kernel.unprivileged_userns_clone = 1`;
|
||||
- `user.max_user_namespaces = 127206`;
|
||||
- no `kernel.apparmor_restrict_unprivileged_userns` (an Ubuntu-only key);
|
||||
- AppArmor enabled (`Y`).
|
||||
|
||||
So this is a satisfied prerequisite, not an open one.
|
||||
- **`infra/gitea/buildkit-client-secret.sh`** (operator action): copies `ca.crt`, `tls.crt` and
|
||||
`tls.key` from the Secret `buildkit/buildkit-client-ci` into the Actions secrets `BUILDKIT_CA`,
|
||||
`BUILDKIT_CLIENT_CERT` and `BUILDKIT_CLIENT_KEY`. Before writing anything it checks that:
|
||||
- the cert verifies against the CA;
|
||||
- the cert is a client-auth cert;
|
||||
- the key matches the cert.
|
||||
|
||||
It never prints a value. `--check` reports whether the stored copy is older than the current
|
||||
certificate.
|
||||
|
||||
### Verification moved into the Containerfiles
|
||||
|
||||
`docker run` cannot exist on BuildKit, so each Containerfile now names its shipped stage after the
|
||||
image (`toolbox`, `agent-runner`, `conformance-runner`) and ends in a **`verify` test stage**,
|
||||
`FROM <image> AS verify`. That stage runs every previous assertion at build time, as the image's
|
||||
non-root uid (65532 for agent-runner and conformance-runner), in `RUN --network=none`. It is never
|
||||
pushed.
|
||||
- **Every existing assertion is preserved**, and the toolbox gained one it never had: every tool
|
||||
on PATH, and five tools report their pinned versions.
|
||||
- `probe-validation.sh` is `COPY`ed into the verify stage only, so it never ships.
|
||||
- **Bug fixed on the way:** in the old `docker run` block, `cmd | grep -q … && echo ok` and
|
||||
`[ … ] && echo` could never fail the script under `set -e` + `pipefail`. The "entrypoints refuse
|
||||
arguments", "not root", "no API key baked in" and "HOME writable" checks asserted nothing. They
|
||||
are real assertions now.
|
||||
- **What a build step cannot reproduce, and how it is approximated** (also written in each stage's
|
||||
header):
|
||||
- **uid:** exact.
|
||||
- **`--network none`:** exact, and each stage *proves* it. Its first check fails if DNS or a
|
||||
TCP connection works. A negative control without `--network=none` fails that check.
|
||||
- **`--read-only` rootfs:** not reproducible, because a RUN step always has a writable layer.
|
||||
It is approximated by permission:
|
||||
- `touch /should-fail` must fail;
|
||||
- a `find / -xdev -writable ! -type l` inventory, run as the uid, must lie entirely under
|
||||
`/workspace`, `/tmp` or `/var/tmp`. `! -type l` was needed: Fedora's `/usr/tmp` symlink
|
||||
to `/var/tmp` tripped it.
|
||||
|
||||
The real control remains the pods' `readOnlyRootFilesystem`, which the admission policies
|
||||
enforce.
|
||||
- **`--tmpfs /workspace`:** approximated by the image's own `/workspace`, owned by 65532, which
|
||||
is exactly what the pod's emptyDir replaces.
|
||||
|
||||
The opt-in `--verify-clone` still `run`s the image, so it remains local-only.
|
||||
- **`build-image.sh` (all three):**
|
||||
- verify is `build --target verify` and the image is `build --target <image>`;
|
||||
- a new `--print-build-args` prints the pins as `KEY=VALUE`, which is what CI consumes. The
|
||||
arg list therefore exists once;
|
||||
- the local podman path still works.
|
||||
- The L0 check `alert-rules-unit-tested` now looks for `"PROMTOOL_VERSION=$PROMTOOL_VERSION"` in
|
||||
the toolbox's build-arg list.
|
||||
|
||||
### The pipeline (`.gitea/workflows/images.yaml` + `scripts/images.sh`)
|
||||
|
||||
- **The builder.** The driver downloads the pinned `buildx` and refuses a checksum mismatch. It
|
||||
writes the client cert from the `BUILDKIT_*` secrets to a 0600 temp dir that is removed on exit,
|
||||
creates a `remote` builder, and runs `buildx inspect --bootstrap`.
|
||||
- **It fails loudly, never skips.** It stops with a clear message if:
|
||||
- the secrets are absent;
|
||||
- buildkitd is unreachable. The message lists the causes in order: not deployed yet (Flux
|
||||
applies it only after merge), stale certificate, NetworkPolicy.
|
||||
- **Pull request:** `buildx bake <image>-verify` for changed images. Nothing is pushed and no
|
||||
registry credential is referenced. The `BUILDKIT_*` builder secrets *are* given to PR jobs,
|
||||
because they must build.
|
||||
- agent-runner is built FROM the toolbox built **in the same run** through a bake `target:`
|
||||
context. That toolbox exists in no registry, so the toolbox is built as its base even when
|
||||
its own inputs did not change.
|
||||
- **Push to main / dispatch on main:** for each image, the verify target first, and only then the
|
||||
image itself with `push=true`. Tags are `sha-<12 hex>` (immutable) and `main` (moving).
|
||||
- The digest is read back from the registry and compared with the client's metadata.
|
||||
- agent-runner builds FROM the toolbox digest just pushed, or else from the pinned
|
||||
`AGENT_BASE_IMAGE` (the zero placeholder bootstraps).
|
||||
- `BUILDX_NO_DEFAULT_ATTESTATIONS=1` keeps one manifest per tag, which is what admission,
|
||||
Renovate and the retention script assume.
|
||||
- **Unchanged from the first iteration:** change detection, publish idempotence,
|
||||
`REGISTRY_PUSH_TOKEN` and its preflight, the Renovate markers, and `registry-retention.sh`. The
|
||||
DinD prune step is gone, because nothing is built in DinD now.
|
||||
|
||||
### Unchanged from the first iteration (still true)
|
||||
|
||||
- **Digest references** carry `# renovate-digest: image=<repo> tag=main` markers:
|
||||
- `platform/pm-agent/deployment.yaml` and `configmap.yaml`;
|
||||
- `infra/agent/versions.env` `AGENT_BASE_IMAGE`, now the published toolbox by digest, zero
|
||||
until the first publish;
|
||||
- `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to zero.
|
||||
|
||||
The admission policies are untouched.
|
||||
- **`renovate.json5`**:
|
||||
- a new `custom.regex` manager for the markers;
|
||||
- the `custom.regex` marker manager;
|
||||
- `enabledManagers` `regex` → `custom.regex`;
|
||||
- a packageRule that disables the `kubernetes` manager for these three repos. Otherwise it
|
||||
would also see the digest-only lines and look them up as `latest`;
|
||||
- an `own-image` label;
|
||||
- `ignorePaths` narrowed from `conformance/**` to its subdirectories. `ignorePaths` applies
|
||||
to custom managers too, so the blanket glob hid `conformance/versions.env`.
|
||||
- **`renovate.yaml`**: `renovate-version: 44.167.0` pinned, plus a `RENOVATE_HOST_RULES` docker
|
||||
rule for the Gitea registry, using the same `RENOVATE_TOKEN`, which now also needs
|
||||
`read:package`. See below for why the version pin is a real fix and not tidiness.
|
||||
- **`infra/gitea/registry-retention.sh`**: dry run by default, `--apply` to delete. This is an
|
||||
operator action. Per image it keeps:
|
||||
- `main` and `latest`;
|
||||
- the 10 newest `sha-` tags;
|
||||
- every tag whose digest is pinned anywhere in `origin/main`;
|
||||
- every digest-named version.
|
||||
- **`infra/gitea/conformance-trigger-kubeconfig.sh`**: the "runner image is digest-pinned"
|
||||
precondition grepped the literal line `CONFORMANCE_RUNNER_IMAGE="${…_REPO}@${…_DIGEST}"`
|
||||
for 64 hex digits, so it **could never pass**. It now sources the file, evaluates the value,
|
||||
and rejects the zero placeholder.
|
||||
- **`conformance/tests/l0-static/alert-rules-unit-tested/check.sh`**: its "the toolbox build
|
||||
passes `--build-arg PROMTOOL_VERSION`" assertion now reads `infra/devcontainer/build-image.sh`
|
||||
instead of `dev.sh`. The gate caught this on the first run, because the recipe had moved.
|
||||
- **Docs**: `platform/pm-agent/README.md`, `runbooks/pm-agent.md` (two rows),
|
||||
`docs/plans/pm-agent-issue-to-pr.md` (follow-up 1), `conformance/incluster/README.md`,
|
||||
`infra/devcontainer/README.md`, and a new "How images are built" section in
|
||||
`.gitea/workflows/README.md`.
|
||||
- the `kubernetes` manager disabled for these three repos;
|
||||
- `ignorePaths` narrowed so `conformance/versions.env` is visible.
|
||||
|
||||
### BLOCKER — the runner cannot build images, and fixing that is a security decision
|
||||
|
||||
**Measured on this PR's own CI run (images run 325): the job failed in 6 s** with
|
||||
`failed to connect to the docker API at unix:///var/run/docker.sock ... no such file`. The job
|
||||
container is given **no Docker daemon at all**, for two independent reasons, both read from the
|
||||
exact versions deployed:
|
||||
|
||||
- act_runner **v0.2.11** `internal/app/cmd/daemon.go`: when `DOCKER_HOST` has a scheme other than
|
||||
`unix`/`npipe` — here `tcp://localhost:2376`, the DinD sidecar — it sets
|
||||
`cfg.Container.DockerHost = "-"`, which means "never mount into job containers".
|
||||
- gitea/act **v0.261.2** (act_runner's `go.mod` replace) `pkg/container/docker_run.go`
|
||||
`sanitizeConfig`: with `valid_volumes` empty (the runner has no config file, so the default
|
||||
`[]`), `hostConfig.Binds = []string{}`. **Every** bind is dropped, the daemon socket included,
|
||||
and so is any `volumes:` a workflow asks for.
|
||||
|
||||
I assumed the opposite while designing (from act's `getDockerDaemonSocketMountPath`, which never
|
||||
gets the chance to run). That was wrong, and the PR run is what showed it.
|
||||
|
||||
**Why I did not fix it in this PR.** Any fix gives workflow code the DinD daemon. That daemon is
|
||||
`privileged: true`, so `docker run --privileged -v /:/host` from a job is **root on the node**.
|
||||
Today a malicious workflow — for example one pushed by a prompt-injected pm-agent worker, which
|
||||
holds a repo write token — runs in an unprivileged container and holds no cluster credential.
|
||||
After the fix it does not. That is a change to the security boundary, and it is the operator's
|
||||
call. It also cannot be tested from a PR anyway: the runner only changes after merge + Flux.
|
||||
|
||||
**Options, for the operator:**
|
||||
|
||||
1. **Socket to every job (smallest change):** in `apps/wave-1/gitea-actions.yaml`,
|
||||
- mount a shared `emptyDir` at `/var/run` in both the `dind` sidecar and the `runner`, so
|
||||
dockerd's own `unix:///var/run/docker.sock` is visible at the same path in both;
|
||||
- set the runner's `DOCKER_HOST=unix:///var/run/docker.sock`, and drop `DOCKER_TLS_VERIFY`
|
||||
and `DOCKER_CERT_PATH`, because TLS over a unix socket breaks the client;
|
||||
- add an act_runner config file (`CONFIG_FILE`) with `container.valid_volumes:
|
||||
["/var/run/docker.sock"]` and `runner.file: /data/.runner`, so the existing registration
|
||||
is kept.
|
||||
|
||||
Consequence: any workflow on any branch can reach node root.
|
||||
2. **Option 1 on a SECOND runner** with its own label (e.g. `image-builder`) and its own DinD,
|
||||
with `images.yaml` `runs-on` that label. The gate runner stays socket-less, and image builds
|
||||
stop competing with the gate for the single slot and its 7 GB node disk. **It is not a
|
||||
security boundary**: Gitea cannot restrict which workflow may target a label, so any branch
|
||||
could still `runs-on: image-builder`.
|
||||
3. **A builder that is not a privileged daemon**, e.g. a rootless BuildKit sidecar exposed to
|
||||
jobs. That bounds a job to BuildKit's rootless sandbox instead of node root. It needs the
|
||||
verification steps rewritten, because they *run* the image (as uid 65532, `--read-only`),
|
||||
which BuildKit alone does not do. This is a larger piece of work.
|
||||
|
||||
Nothing in this PR has to change for options 1 or 2 except `runs-on` for option 2. The driver
|
||||
already produced verified images with a stock docker 27 CLI (see *What I verified*).
|
||||
`renovate.yaml` pins `renovate-version: 44.167.0` and adds a docker `RENOVATE_HOST_RULES` entry
|
||||
using `RENOVATE_TOKEN`, which now needs `read:package`.
|
||||
- **`infra/gitea/registry-retention.sh`** (dry run by default; operator action); the
|
||||
`conformance-trigger-kubeconfig.sh` precondition fix; and the docs (pm-agent README, runbook,
|
||||
plan, conformance README, devcontainer README, workflows README). The docs are updated again
|
||||
for BuildKit, and there is a new `platform/buildkit/README.md`.
|
||||
|
||||
### Decisions and pushback
|
||||
|
||||
@@ -147,7 +197,8 @@ already produced verified images with a stock docker 27 CLI (see *What I verifie
|
||||
- `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3.
|
||||
|
||||
The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight
|
||||
pattern.
|
||||
pattern. With BuildKit the credential reaches the daemon through the buildx session (a
|
||||
0600 `config.json` in the job's temp dir), never through the cluster.
|
||||
- **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate:
|
||||
- `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` →
|
||||
`custom.regex`;
|
||||
@@ -195,89 +246,112 @@ already produced verified images with a stock docker 27 CLI (see *What I verifie
|
||||
|
||||
## What I verified
|
||||
|
||||
- **Local gate, pinned toolbox, from this worktree, committed HEAD.** Command:
|
||||
`ENGINE=<shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`.
|
||||
The shim adds the `.git` mount. Result: see the PR body for the rc and stage list. They are
|
||||
recorded there rather than here because this entry is part of the tree being gated.
|
||||
- **Local podman builds through the CI driver**, run as
|
||||
`MODE=verify ENGINE=podman BASE_SHA=$(git merge-base origin/main HEAD) .gitea/workflows/scripts/images.sh`.
|
||||
It returned rc 0 and built and verified all three images:
|
||||
- toolbox: every tool on PATH, and the five pins matched;
|
||||
- agent-runner: `PM_SELFTEST=1 pm-dispatch` OK, and the stdin-fed `probe-validation.sh`
|
||||
gave "20 refusal(s), 0 failure(s)";
|
||||
- conformance-runner: all seven `conformance-clone` refusals.
|
||||
- **The docker-CLI code path**, i.e. what CI runs. I ran the same driver inside
|
||||
`docker:27-cli` (`ENGINE=docker`, `DOCKER_BUILDKIT=0`) against podman's Docker-compatible API
|
||||
socket. Exit code 0; all three images built and verified (selftest OK; probe OK, 20 refusals).
|
||||
This covers the docker-specific calls (`image inspect` formats, `run -i` with stdin,
|
||||
`--tmpfs`, `--read-only`, `--user`). It does **not** cover BuildKit or a real DinD daemon.
|
||||
- **This PR's CI, run 325 (`images`): FAILED in 6 s**, because there is no Docker daemon in the
|
||||
job container (see BLOCKER). Gate run 324 failed only on `approval-policy` (expected,
|
||||
sensitive paths). Every other stage passed there, including `gitea-settings`.
|
||||
- **Other driver paths, run locally:**
|
||||
- agent-runner alone builds the toolbox as its base;
|
||||
- an empty diff builds nothing;
|
||||
- publish mode without a token dies before doing anything;
|
||||
- `infra/agent/build-image.sh` with the zero placeholder dies with build-locally
|
||||
instructions.
|
||||
- **Renovate, with the pinned 44.167.0 image, locally:**
|
||||
- `renovate-config-validator` exits 0;
|
||||
- a `--platform=local` `extract` dry run finds all four marker references (custom regex manager,
|
||||
`currentValue: main`, `currentDigest: sha256:000…`);
|
||||
- a `lookup` dry run shows the `kubernetes`-manager duplicates as `skipReason: disabled` and the
|
||||
marker deps taking the "unsupported/unversioned value main → getDigest(main)" path. That
|
||||
lookup currently 404s because nothing is published yet.
|
||||
- **Renovate 39.264.0 rejects main's current `renovate.json5`.** Validator output reproduced
|
||||
above.
|
||||
- **`registry-retention.sh` dry run against the live Gitea (read-only):** zero versions per image,
|
||||
nothing to delete. The token exchange and the package listing both answered.
|
||||
- **Live, read-only measurements** (kubectl and the Gitea API):
|
||||
- the runner pod is on k3s-prod-1, which had **7 GB** nodefs available;
|
||||
- DinD holds only the 1.54 GB act image; buildx is v0.20.1; the daemon is 27.5.1 on overlay2
|
||||
with cgroup v2;
|
||||
- Gitea `/data` is 9.7G, 1% used;
|
||||
- the Gateway's Envoy (Cilium) has `streamIdleTimeout: 300s`, `maxStreamDuration: 0s`, and
|
||||
**no route `timeout`** on the gitea route;
|
||||
- Gitea's `LIMIT_SIZE_CONTAINER` is unset (default −1 = unlimited, `modules/setting/packages.go`).
|
||||
**Proved locally, against the exact pinned images:**
|
||||
|
||||
- **Rootless buildkitd, configured as the manifest configures it.**
|
||||
- Setup: the pinned `moby/buildkit:v0.34.1-rootless` digest, under rootless podman, with
|
||||
`--cap-drop ALL --cap-add SETUID --cap-add SETGID`, seccomp unconfined and `--read-only`;
|
||||
tmpfs at the same five paths; the ConfigMap's `buildkitd.toml`; mTLS on :1234.
|
||||
- The image's `/etc/subuid` was overridden with `user:1:999` and `user:1001:64535`, because
|
||||
rootless podman maps only 65,537 uids. rootlesskit's real `newuidmap` user-namespace path
|
||||
still ran.
|
||||
- It started, `buildctl debug workers` listed the OCI worker, and `debug workers -v` showed the
|
||||
GC config applied (max 4.29 GB, reserved 1.07 GB).
|
||||
- **The securityContext, item by item.** Each requirement was measured by taking it away:
|
||||
- **default seccomp:** daemon starts, every RUN step fails → `Unconfined` is required;
|
||||
- **no `SETUID`/`SETGID`:** rootlesskit dies with
|
||||
`newuidmap … operation not permitted` → both capabilities are required;
|
||||
- **`--read-only` rootfs:** works once the five emptyDirs exist. Its only failure was the
|
||||
probe client's `~/.docker`, which is now an emptyDir;
|
||||
- **minimal caps + Unconfined seccomp:** works, and a RUN step executes.
|
||||
- **The full CI path, in the actual CI job image.** Setup: `ghcr.io/catthehacker/ubuntu:act-22.04`
|
||||
running `.gitea/workflows/scripts/images.sh` in verify mode. It downloaded and sha256-checked
|
||||
buildx v0.38.1, connected over mTLS, and bake-built all three `verify` targets from an empty
|
||||
cache. rc 0, 1m32s total:
|
||||
- `TOOLBOX VERIFIED` (42 s);
|
||||
- `AGENT-RUNNER VERIFIED` (26 s): selftest OK, probe "20 refusal(s), 0 failure(s)", and built
|
||||
FROM the in-run toolbox through the `target:` context, with the log showing
|
||||
`[agent-runner-verify toolbox 1/6]` and `base=toolbox-in-this-run`;
|
||||
- `CONFORMANCE-RUNNER VERIFIED` (21 s): all seven refusals;
|
||||
- every stage printed `confirmed: no network`.
|
||||
- **mTLS rejects what it should.** Tested with a real BuildKit API call (`buildctl debug workers`):
|
||||
- no client cert → `remote error: tls: certificate required`;
|
||||
- a client cert from another CA → rejected the same way;
|
||||
- the valid cert → accepted.
|
||||
|
||||
Through the driver:
|
||||
- another CA's client cert → rc 1;
|
||||
- the wrong CA for the server → rc 1;
|
||||
- an unreachable host → rc 1 in about 20 s, with the explanatory message;
|
||||
- missing secrets → rc 1, naming the three secrets.
|
||||
- **The local podman path**, `build-image.sh` for all three (rc 0).
|
||||
- Negative control: a verify stage expecting a different promtool version printed
|
||||
`DRIFT promtool want 9.9.9` and failed the build.
|
||||
- Negative control: the `--network=none` check fails when the flag is removed.
|
||||
- **`buildkit-client-secret.sh`'s sanity checks**, run offline on local certs. They tell apart:
|
||||
- a good client cert;
|
||||
- a server cert (not client-auth);
|
||||
- a mismatched key;
|
||||
- a cert from another CA.
|
||||
|
||||
**Measured live, read-only:**
|
||||
- node user namespaces, Debian 13.7 and AppArmor (via an Ansible `sysctl`/`cat`);
|
||||
- DinD's MASQUERADE rules;
|
||||
- the live ClusterPolicies, namespaced Policies, PSA labels, and cert-manager v1.21.3 with its
|
||||
ClusterIssuers;
|
||||
- the act-runner pod labels.
|
||||
|
||||
**Renovate (unchanged since the first iteration):**
|
||||
- the validator with 44.167.0: exit 0;
|
||||
- `extract` finds all four markers;
|
||||
- the `kubernetes` duplicates are `skipReason: disabled`;
|
||||
- 39.264.0 rejects `main`'s config.
|
||||
|
||||
**Local gate, CI results:** see the PR body. They are recorded there because this entry is part of
|
||||
the tree being gated.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **Nothing has been pushed to the registry, by me or by CI.** The publish path has never run:
|
||||
- the registry HEAD/digest read-back;
|
||||
- **platform/buildkit has never run in the cluster.** It can't until it merges: agents may not
|
||||
`kubectl apply`, and Flux reconciles only `main`. So this PR's own `images` job **cannot build
|
||||
and is expected to fail at the builder preflight**, which is the designed loud failure. The
|
||||
following are therefore all unproven in-cluster:
|
||||
- kubelet admitting the pod;
|
||||
- containerd honouring `appArmorProfile: Unconfined`;
|
||||
- the five emptyDirs;
|
||||
- cert-manager issuing the three certs;
|
||||
- the CiliumNetworkPolicies.
|
||||
- **AppArmor `Unconfined` is reasoned, not measured.** My test host runs SELinux, so the AppArmor
|
||||
requirement comes from upstream's example plus the nodes having AppArmor enabled. If the pod
|
||||
were to work under `RuntimeDefault` AppArmor, that line could be tightened; I did not test it.
|
||||
- **The local userns mapping is narrower than the cluster's.** Locally it was 65,535 ids across
|
||||
two ranges; in the cluster it is the image's own `100000:65536` against real host subuids. That
|
||||
is the normal case, but it is the cluster's first test.
|
||||
- **The NetworkPolicies are untested.** In particular:
|
||||
- that `toEntities: [ingress]` admits the push to the Gateway VIP. The comment explains the
|
||||
reasoning, but it is unmeasured;
|
||||
- that world :443/:80 covers every download the builds make. A dnf mirror on another port
|
||||
would fail a build visibly.
|
||||
- **No push, anywhere.** The whole publish path is unexercised:
|
||||
- `REGISTRY_PUSH_TOKEN` reaching buildkitd through the buildx session;
|
||||
- `push=true`;
|
||||
- the metadata digest compared with the registry read-back;
|
||||
- agent-runner FROM `toolbox@sha256:`;
|
||||
- the immutable-tag guard;
|
||||
- "already published at this commit" idempotence;
|
||||
- agent-runner `FROM <toolbox>@sha256:`;
|
||||
- `docker login` with a PAT.
|
||||
- idempotence.
|
||||
|
||||
The first merge to main is the first test, and it needs `REGISTRY_PUSH_TOKEN` to exist first.
|
||||
- **Large-layer push through the Gateway is UNVERIFIED.** Envoy has no request-body cap and no
|
||||
stream duration cap here. But with no route `timeout` set, Envoy's **default 15 s upstream
|
||||
timeout** applies from the end of the request to the end of the response. That covers Gitea
|
||||
finalising a blob: copying the upload into package storage on Longhorn. A ~250 MB compressed
|
||||
layer should finish well inside 15 s, but I could not measure it without pushing. If the first
|
||||
publish shows 504s on blob PUTs, set `timeouts.request` on the gitea HTTPRoute.
|
||||
- **DinD fit is estimated, not measured** (no build has run there, see the BLOCKER):
|
||||
- memory: 2Gi is shared by the job container and every build container;
|
||||
- disk: the toolbox, agent-runner, conformance-runner, Fedora base and build cache should
|
||||
total roughly 3 GB on top of the 1.5 GB act image, against 7 GB free on the node, with
|
||||
`dind-storage` having no sizeLimit.
|
||||
- **No image has been built in this runner's DinD**, because it cannot be done until the BLOCKER
|
||||
is resolved. Two things are therefore unmeasured until then:
|
||||
- DinD memory (2Gi) and disk fit;
|
||||
- BuildKit (`buildx` v0.20.1 is on the daemon) as opposed to the legacy builder I tested
|
||||
with.
|
||||
- **Renovate has still never completed a run here.** Three things block it:
|
||||
- #80, the `configurationFile` name;
|
||||
- `RENOVATE_TOKEN` gaining `read:package`;
|
||||
- this branch's version pin.
|
||||
|
||||
`renovatebot/github-action@v41.0.9` running a 44.x image is untested. The action only passes
|
||||
the tag to `docker run`.
|
||||
- **Pods cannot pull these images yet regardless.** The kubelet pull credential is PR #61
|
||||
(`pm-agent-registry` dockerconfigjson, `read:package`), still open. Until it merges, a correct
|
||||
digest still ends in ImagePullBackOff (401), as before.
|
||||
- **`registry-retention.sh --apply` has never run**, and its keep logic has only run against an
|
||||
empty registry.
|
||||
- **`git diff` change detection on Gitea `push` events relies on `github.event.before`.** A
|
||||
force-push or a first push falls back to building everything, which is intended but not
|
||||
observed.
|
||||
So are the 15 s Envoy route-timeout risk on blob finalisation, and the Gitea PVC (10Gi, shared
|
||||
with repositories) absorbing ~2 GB per toolbox publish until retention runs.
|
||||
- **Node disk under real builds.** The cache emptyDir (6Gi limit, 4 GB GC cap) sits on nodes that
|
||||
had 7–9 GB free. It is untested under concurrent pressure.
|
||||
- **Certificate rotation is designed, not exercised.** The server cert renews 30 days early but
|
||||
buildkitd only reads it at start; the client copy needs the script re-run.
|
||||
- **`--oci-worker-no-process-sandbox` is an accepted risk, not a mitigated one.** A build step can
|
||||
ptrace the daemon and read its TLS key and cache. Anyone with the client cert, which means every
|
||||
PR job, can run build steps.
|
||||
- **Renovate has still never completed a run.** It is blocked on #80, on `RENOVATE_TOKEN` gaining
|
||||
`read:package`, and on this branch. The `[email protected]` + 44.x image pairing is
|
||||
untested.
|
||||
- **Pods cannot pull these images until PR #61** (the kubelet pull secret) merges.
|
||||
- **`registry-retention.sh --apply` has never run**; its keep logic has only run against an empty
|
||||
registry.
|
||||
Reference in new issue
Block a user