platform/buildkit — rootless BuildKit, the CI image builder
The builder behind .gitea/workflows/images.yaml. It
builds toolbox, agent-runner and conformance-runner, and on main pushes them to the
Gitea registry. Flux reconciles it as the Kustomization buildkit
(clusters/base/buildkit.yaml, dependsOn: platform).
Why it exists (operator decision on PR #81, 2026-10-29)
CI jobs run as containers on the act runner's privileged DinD sidecar, and cannot reach any
Docker daemon. act_runner 0.2.11 forces docker_host: "-" for a tcp:// DinD, and gitea/act
strips every bind mount when valid_volumes is empty. Measured on PR #81's first run. Giving jobs
that daemon would make every branch push root on the node. So the rule is: no job may reach a
privileged daemon. Builds go to this rootless daemon over mTLS instead.
Shape
| Piece | What it is |
|---|---|
deployment.yaml |
moby/buildkit:v0.34.1-rootless, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The securityContext is commented line by line with how each item was measured |
certificates.yaml |
cert-manager: a self-signed CA used only here, plus the server cert buildkitd-tls and the CI client cert buildkit-client-ci |
service.yaml |
ClusterIP buildkitd.buildkit.svc.cluster.local:1234, mTLS only. Never on the Gateway |
networkpolicy.yaml |
Default-deny, plus two narrow allows. Ingress is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. Egress is DNS, world :443/:80 for downloads, and, for the registry push, the Gateway (ingress entity) plus Gitea's backend identity on :3000 — Cilium's Gateway re-checks this pod's egress against the backend it proxies to, and without that rule answers 403 Access denied itself (measured 2026-10-30, decisions/2026-10-30-gitea-admit-buildkit.md) |
configmap.yaml |
buildkitd.toml: cache GC capped at 4 GB, because the cache is an emptyDir on node disk |
versions.env |
The CI side: pinned buildx (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource |
Admission
No Kyverno exception is needed.
- Every ClusterPolicy that is actually applied (the list in
platform-config/kustomization.yaml) matches only namespaces labelledhomelab.keithjackson.dev/app-namespace: "true". This namespace, likegitea-actions, does not carry that label. - The job-shape policies (
pm-agent-job-shape,conformance-trigger-job-shape) match only their own namespaces. - There is no cluster-wide Pod Security default. This namespace is deliberately left without a
pod-securitylabel, because PSSbaselineforbids theUnconfinedseccomp and AppArmor profiles that rootless BuildKit needs.
Prerequisites (operator, in order)
- Unprivileged user namespaces on the nodes. Already true. Measured 2026-10-29 on all three
Debian 13.7 nodes (kernel 7.1.8):
kernel.unprivileged_userns_clone = 1,user.max_user_namespaces = 127206, and no Ubuntu-style AppArmor userns restriction. - Merge. Flux applies this directory. Check that the Kustomization
buildkitis Ready and that all three Certificates in namespacebuildkitare READY. - Copy the client certificate to CI. Run
infra/gitea/buildkit-client-secret.sh. Re-run it yearly, after renewal;--checkreports whether the copy is stale. - Restart buildkitd after the server certificate renews. It reads the certificate at start, and cert-manager renews it 30 days before expiry, so a restart inside that window is needed.
Until step 3 is done, the images workflow fails at its builder preflight with a message naming
the cause. It never skips.
What a build step can do
The daemon runs with --oci-worker-no-process-sandbox, so build steps share its PID namespace.
buildkitd warns about this itself at startup.
- A build step CAN signal or ptrace the daemon and read what it can read, including this pod's TLS key and the build cache.
- A build step CANNOT reach the node. Everything runs as uid 1000 in a user namespace, unprivileged, with no host mounts.
- Who can run one: anyone holding the client certificate, i.e. every CI job, including pull requests.
Local equivalent
The pinned image runs under rootless podman with the same restrictions:
--cap-drop ALL --cap-add SETUID --cap-add SETGID --security-opt seccomp=unconfined --read-only,
plus tmpfs at the same five paths. That is how the securityContext was measured. Rootless podman
maps only 65,537 uids, so the image's /etc/subuid (user:100000:65536) must be overridden with
two ranges that fit: user:1:999 and user:1001:64535.