jakman85andClaude Opus 5.5
310deb0f7d
Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched. - platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined, read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP :1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it). No Kyverno exception: every applied policy matches app-namespace=true only. - Each Containerfile names its shipped stage and ends in a `verify` stage holding every former `docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now. - images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes. Fails loudly when the builder secrets are absent or buildkitd is unreachable. - infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets. Co-Authored-By: Claude Opus 5.5 <[email protected]>
platform/buildkit — rootless BuildKit, the CI image builder
The builder behind .gitea/workflows/images.yaml. It
builds toolbox, agent-runner and conformance-runner, and on main pushes them to the
Gitea registry. Flux reconciles it as the Kustomization buildkit
(clusters/base/buildkit.yaml, dependsOn: platform).
Why it exists (operator decision on PR #81, 2026-10-29)
CI jobs run as containers on the act runner's privileged DinD sidecar, and cannot reach any
Docker daemon. act_runner 0.2.11 forces docker_host: "-" for a tcp:// DinD, and gitea/act
strips every bind mount when valid_volumes is empty. Measured on PR #81's first run. Giving jobs
that daemon would make every branch push root on the node. So the rule is: no job may reach a
privileged daemon. Builds go to this rootless daemon over mTLS instead.
Shape
| Piece | What it is |
|---|---|
deployment.yaml |
moby/buildkit:v0.34.1-rootless, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The securityContext is commented line by line with how each item was measured |
certificates.yaml |
cert-manager: a self-signed CA used only here, plus the server cert buildkitd-tls and the CI client cert buildkit-client-ci |
service.yaml |
ClusterIP buildkitd.buildkit.svc.cluster.local:1234, mTLS only. Never on the Gateway |
networkpolicy.yaml |
Default-deny, plus two narrow allows. Ingress is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. Egress is DNS, world :443/:80 for downloads, and the Gateway (ingress entity) for the registry push |
configmap.yaml |
buildkitd.toml: cache GC capped at 4 GB, because the cache is an emptyDir on node disk |
versions.env |
The CI side: pinned buildx (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource |
Admission
No Kyverno exception is needed.
- Every ClusterPolicy that is actually applied (the list in
platform-config/kustomization.yaml) matches only namespaces labelledhomelab.keithjackson.dev/app-namespace: "true". This namespace, likegitea-actions, does not carry that label. - The job-shape policies (
pm-agent-job-shape,conformance-trigger-job-shape) match only their own namespaces. - There is no cluster-wide Pod Security default. This namespace is deliberately left without a
pod-securitylabel, because PSSbaselineforbids theUnconfinedseccomp and AppArmor profiles that rootless BuildKit needs.
Prerequisites (operator, in order)
- Unprivileged user namespaces on the nodes. Already true. Measured 2026-10-29 on all three
Debian 13.7 nodes (kernel 7.1.8):
kernel.unprivileged_userns_clone = 1,user.max_user_namespaces = 127206, and no Ubuntu-style AppArmor userns restriction. - Merge. Flux applies this directory. Check that the Kustomization
buildkitis Ready and that all three Certificates in namespacebuildkitare READY. - Copy the client certificate to CI. Run
infra/gitea/buildkit-client-secret.sh. Re-run it yearly, after renewal;--checkreports whether the copy is stale. - Restart buildkitd after the server certificate renews. It reads the certificate at start, and cert-manager renews it 30 days before expiry, so a restart inside that window is needed.
Until step 3 is done, the images workflow fails at its builder preflight with a message naming
the cause. It never skips.
What a build step can do
The daemon runs with --oci-worker-no-process-sandbox, so build steps share its PID namespace.
buildkitd warns about this itself at startup.
- A build step CAN signal or ptrace the daemon and read what it can read, including this pod's TLS key and the build cache.
- A build step CANNOT reach the node. Everything runs as uid 1000 in a user namespace, unprivileged, with no host mounts.
- Who can run one: anyone holding the client certificate, i.e. every CI job, including pull requests.
Local equivalent
The pinned image runs under rootless podman with the same restrictions:
--cap-drop ALL --cap-add SETUID --cap-add SETGID --security-opt seccomp=unconfined --read-only,
plus tmpfs at the same five paths. That is how the securityContext was measured. Rootless podman
maps only 65,537 uids, so the image's /etc/subuid (user:100000:65536) must be overridden with
two ranges that fit: user:1:999 and user:1001:64535.