jakman85andClaude Opus 5.5
f65c12b395
Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched. - platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined, read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP :1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it). No Kyverno exception: every applied policy matches app-namespace=true only. - Each Containerfile names its shipped stage and ends in a `verify` stage holding every former `docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now. - images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes. Fails loudly when the builder secrets are absent or buildkitd is unreachable. - infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets. Co-Authored-By: Claude Opus 5.5 <[email protected]>
75 lines
4.6 KiB
Markdown
75 lines
4.6 KiB
Markdown
# platform/buildkit — rootless BuildKit, the CI image builder
|
|
|
|
The builder behind [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). It
|
|
builds `toolbox`, `agent-runner` and `conformance-runner`, and on `main` pushes them to the
|
|
Gitea registry. Flux reconciles it as the Kustomization `buildkit`
|
|
([`clusters/base/buildkit.yaml`](../../clusters/base/buildkit.yaml), `dependsOn: platform`).
|
|
|
|
## Why it exists (operator decision on PR #81, 2026-10-29)
|
|
|
|
CI jobs run as containers on the act runner's privileged DinD sidecar, and **cannot reach any
|
|
Docker daemon**. act_runner 0.2.11 forces `docker_host: "-"` for a `tcp://` DinD, and gitea/act
|
|
strips every bind mount when `valid_volumes` is empty. Measured on PR #81's first run. Giving jobs
|
|
that daemon would make every branch push root on the node. So the rule is: **no job may reach a
|
|
privileged daemon.** Builds go to this rootless daemon over mTLS instead.
|
|
|
|
## Shape
|
|
|
|
| Piece | What it is |
|
|
|---|---|
|
|
| `deployment.yaml` | `moby/buildkit:v0.34.1-rootless`, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The `securityContext` is commented line by line with how each item was measured |
|
|
| `certificates.yaml` | cert-manager: a self-signed CA used only here, plus the server cert `buildkitd-tls` and the CI client cert `buildkit-client-ci` |
|
|
| `service.yaml` | ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`, mTLS only. Never on the Gateway |
|
|
| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and the Gateway (`ingress` entity) for the registry push |
|
|
| `configmap.yaml` | `buildkitd.toml`: cache GC capped at 4 GB, because the cache is an `emptyDir` on node disk |
|
|
| `versions.env` | The CI side: pinned `buildx` (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource |
|
|
|
|
## Admission
|
|
|
|
**No Kyverno exception is needed.**
|
|
- Every ClusterPolicy that is actually applied (the list in
|
|
[`platform-config/kustomization.yaml`](../../platform-config/kustomization.yaml)) matches only
|
|
namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`. This namespace, like
|
|
`gitea-actions`, does not carry that label.
|
|
- The job-shape policies (`pm-agent-job-shape`, `conformance-trigger-job-shape`) match only their
|
|
own namespaces.
|
|
- There is no cluster-wide Pod Security default. This namespace is deliberately left without a
|
|
`pod-security` label, because PSS `baseline` forbids the `Unconfined` seccomp and AppArmor
|
|
profiles that rootless BuildKit needs.
|
|
|
|
## Prerequisites (operator, in order)
|
|
|
|
1. **Unprivileged user namespaces on the nodes.** Already true. Measured 2026-10-29 on all three
|
|
Debian 13.7 nodes (kernel 7.1.8): `kernel.unprivileged_userns_clone = 1`,
|
|
`user.max_user_namespaces = 127206`, and no Ubuntu-style AppArmor userns restriction.
|
|
2. **Merge.** Flux applies this directory. Check that the Kustomization `buildkit` is Ready and
|
|
that all three Certificates in namespace `buildkit` are READY.
|
|
3. **Copy the client certificate to CI.** Run
|
|
[`infra/gitea/buildkit-client-secret.sh`](../../infra/gitea/buildkit-client-secret.sh).
|
|
Re-run it yearly, after renewal; `--check` reports whether the copy is stale.
|
|
4. **Restart buildkitd after the server certificate renews.** It reads the certificate at start,
|
|
and cert-manager renews it 30 days before expiry, so a restart inside that window is needed.
|
|
|
|
Until step 3 is done, the `images` workflow fails at its builder preflight with a message naming
|
|
the cause. It never skips.
|
|
|
|
## What a build step can do
|
|
|
|
The daemon runs with `--oci-worker-no-process-sandbox`, so build steps share its PID namespace.
|
|
buildkitd warns about this itself at startup.
|
|
|
|
- **A build step CAN** signal or ptrace the daemon and read what it can read, including this
|
|
pod's TLS key and the build cache.
|
|
- **A build step CANNOT** reach the node. Everything runs as uid 1000 in a user namespace,
|
|
unprivileged, with no host mounts.
|
|
- **Who can run one:** anyone holding the client certificate, i.e. every CI job, including pull
|
|
requests.
|
|
|
|
## Local equivalent
|
|
|
|
The pinned image runs under rootless podman with the same restrictions:
|
|
`--cap-drop ALL --cap-add SETUID --cap-add SETGID --security-opt seccomp=unconfined --read-only`,
|
|
plus tmpfs at the same five paths. That is how the `securityContext` was measured. Rootless podman
|
|
maps only 65,537 uids, so the image's `/etc/subuid` (`user:100000:65536`) must be overridden with
|
|
two ranges that fit: `user:1:999` and `user:1001:64535`.
|