home-network-k3s-impl/platform/buildkit/README.md

platform/buildkit — rootless BuildKit, the CI image builder

The builder behind .gitea/workflows/images.yaml. It builds toolbox, agent-runner and conformance-runner, and on main pushes them to the Gitea registry. Flux reconciles it as the Kustomization buildkit (clusters/base/buildkit.yaml, dependsOn: platform).

Why it exists (operator decision on PR #81, 2026-10-29)

CI jobs run as containers on the act runner's privileged DinD sidecar, and cannot reach any Docker daemon. act_runner 0.2.11 forces docker_host: "-" for a tcp:// DinD, and gitea/act strips every bind mount when valid_volumes is empty. Measured on PR #81's first run. Giving jobs that daemon would make every branch push root on the node. So the rule is: no job may reach a privileged daemon. Builds go to this rootless daemon over mTLS instead.

Shape

Piece	What it is
deployment.yaml	moby/buildkit:v0.34.1-rootless, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The securityContext is commented line by line with how each item was measured
certificates.yaml	cert-manager: a self-signed CA used only here, plus the server cert buildkitd-tls and the CI client cert buildkit-client-ci
service.yaml	ClusterIP buildkitd.buildkit.svc.cluster.local:1234, mTLS only. Never on the Gateway
networkpolicy.yaml	Default-deny, plus two narrow allows. Ingress is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. Egress is DNS, world :443/:80 for downloads, and, for the registry push, the Gateway (ingress entity) plus Gitea's backend identity on :3000 — Cilium's Gateway re-checks this pod's egress against the backend it proxies to, and without that rule answers 403 Access denied itself (measured 2026-10-30, decisions/2026-10-30-gitea-admit-buildkit.md)
configmap.yaml	buildkitd.toml: cache GC capped at 4 GB, because the cache is an emptyDir on node disk
versions.env	The CI side: pinned buildx (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource

Admission

No Kyverno exception is needed.

• Every ClusterPolicy that is actually applied (the list in platform-config/kustomization.yaml) matches only namespaces labelled homelab.keithjackson.dev/app-namespace: "true". This namespace, like gitea-actions, does not carry that label.
• The job-shape policies (pm-agent-job-shape, conformance-trigger-job-shape) match only their own namespaces.
• There is no cluster-wide Pod Security default. This namespace is deliberately left without a pod-security label, because PSS baseline forbids the Unconfined seccomp and AppArmor profiles that rootless BuildKit needs.

Prerequisites (operator, in order)

1. Unprivileged user namespaces on the nodes. Already true. Measured 2026-10-29 on all three Debian 13.7 nodes (kernel 7.1.8): kernel.unprivileged_userns_clone = 1, user.max_user_namespaces = 127206, and no Ubuntu-style AppArmor userns restriction.
2. Merge. Flux applies this directory. Check that the Kustomization buildkit is Ready and that all three Certificates in namespace buildkit are READY.
3. Copy the client certificate to CI. Run infra/gitea/buildkit-client-secret.sh. Re-run it yearly, after renewal; --check reports whether the copy is stale.
4. Restart buildkitd after the server certificate renews. It reads the certificate at start, and cert-manager renews it 30 days before expiry, so a restart inside that window is needed.

Until step 3 is done, the images workflow fails at its builder preflight with a message naming the cause. It never skips.

What a build step can do

The daemon runs with --oci-worker-no-process-sandbox, so build steps share its PID namespace. buildkitd warns about this itself at startup.

• A build step CAN signal or ptrace the daemon and read what it can read, including this pod's TLS key and the build cache.
• A build step CANNOT reach the node. Everything runs as uid 1000 in a user namespace, unprivileged, with no host mounts.
• Who can run one: anyone holding the client certificate, i.e. every CI job, including pull requests.

Local equivalent

The pinned image runs under rootless podman with the same restrictions: --cap-drop ALL --cap-add SETUID --cap-add SETGID --security-opt seccomp=unconfined --read-only, plus tmpfs at the same five paths. That is how the securityContext was measured. Rootless podman maps only 65,537 uids, so the image's /etc/subuid (user:100000:65536) must be overridden with two ranges that fit: user:1:999 and user:1001:64535.