# platform/buildkit — rootless BuildKit, the CI image builder

The builder behind [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). It
builds `toolbox`, `agent-runner` and `conformance-runner`, and on `main` pushes them to the
Gitea registry. Flux reconciles it as the Kustomization `buildkit`
([`clusters/base/buildkit.yaml`](../../clusters/base/buildkit.yaml), `dependsOn: platform`).

## Why it exists (operator decision on PR #81, 2026-10-29)

CI jobs run as containers on the act runner's privileged DinD sidecar, and **cannot reach any
Docker daemon**. act_runner 0.2.11 forces `docker_host: "-"` for a `tcp://` DinD, and gitea/act
strips every bind mount when `valid_volumes` is empty. Measured on PR #81's first run. Giving jobs
that daemon would make every branch push root on the node. So the rule is: **no job may reach a
privileged daemon.** Builds go to this rootless daemon over mTLS instead.

## Shape

| Piece | What it is |
|---|---|
| `deployment.yaml` | `moby/buildkit:v0.34.1-rootless`, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The `securityContext` is commented line by line with how each item was measured |
| `certificates.yaml` | cert-manager: a self-signed CA used only here, plus the server cert `buildkitd-tls` and the CI client cert `buildkit-client-ci` |
| `service.yaml` | ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`, mTLS only. Never on the Gateway |
| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and, for the registry push, the Gateway (`ingress` entity) **plus** Gitea's backend identity on :3000 — Cilium's Gateway re-checks this pod's egress against the backend it proxies to, and without that rule answers `403 Access denied` itself (measured 2026-10-30, `decisions/2026-10-30-gitea-admit-buildkit.md`) |
| `configmap.yaml` | `buildkitd.toml`: cache GC capped at 4 GB, because the cache is an `emptyDir` on node disk |
| `versions.env` | The CI side: pinned `buildx` (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource |

## Admission

**No Kyverno exception is needed.**
- Every ClusterPolicy that is actually applied (the list in
  [`platform-config/kustomization.yaml`](../../platform-config/kustomization.yaml)) matches only
  namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`. This namespace, like
  `gitea-actions`, does not carry that label.
- The job-shape policies (`pm-agent-job-shape`, `conformance-trigger-job-shape`) match only their
  own namespaces.
- There is no cluster-wide Pod Security default. This namespace is deliberately left without a
  `pod-security` label, because PSS `baseline` forbids the `Unconfined` seccomp and AppArmor
  profiles that rootless BuildKit needs.

## Prerequisites (operator, in order)

1. **Unprivileged user namespaces on the nodes.** Already true. Measured 2026-10-29 on all three
   Debian 13.7 nodes (kernel 7.1.8): `kernel.unprivileged_userns_clone = 1`,
   `user.max_user_namespaces = 127206`, and no Ubuntu-style AppArmor userns restriction.
2. **Merge.** Flux applies this directory. Check that the Kustomization `buildkit` is Ready and
   that all three Certificates in namespace `buildkit` are READY.
3. **Copy the client certificate to CI.** Run
   [`infra/gitea/buildkit-client-secret.sh`](../../infra/gitea/buildkit-client-secret.sh).
   Re-run it yearly, after renewal; `--check` reports whether the copy is stale.
4. **Restart buildkitd after the server certificate renews.** It reads the certificate at start,
   and cert-manager renews it 30 days before expiry, so a restart inside that window is needed.

Until step 3 is done, the `images` workflow fails at its builder preflight with a message naming
the cause. It never skips.

## What a build step can do

The daemon runs with `--oci-worker-no-process-sandbox`, so build steps share its PID namespace.
buildkitd warns about this itself at startup.

- **A build step CAN** signal or ptrace the daemon and read what it can read, including this
  pod's TLS key and the build cache.
- **A build step CANNOT** reach the node. Everything runs as uid 1000 in a user namespace,
  unprivileged, with no host mounts.
- **Who can run one:** anyone holding the client certificate, i.e. every CI job, including pull
  requests.

## Local equivalent

The pinned image runs under rootless podman with the same restrictions:
`--cap-drop ALL --cap-add SETUID --cap-add SETGID --security-opt seccomp=unconfined --read-only`,
plus tmpfs at the same five paths. That is how the `securityContext` was measured. Rootless podman
maps only 65,537 uids, so the image's `/etc/subuid` (`user:100000:65536`) must be overridden with
two ranges that fit: `user:1:999` and `user:1001:64535`.