jakman85andClaude Opus 5.5
d2c4bd5194
Images run 337's push failed with "403 Forbidden: Access denied". Measured
read-only: Gitea never logged the request; an unauthenticated GET /v2/ via the
Gateway gets 401 from Gitea when sent from act-runner (egress unrestricted) but
403 from buildkitd; Hubble logs it as an L7 http-request DROPPED with
traffic_direction EGRESS. Cilium >= 1.16.8 (CVE-2025-30162) re-checks the
source pod's egress against the proxied backend identity, and buildkitd's
egress only allowed world and the ingress entity. Same fix as 68690e5 for
Keycloak. Gitea's ingress already admits reserved:ingress and is unchanged.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
75 lines
4.9 KiB
Markdown
75 lines
4.9 KiB
Markdown
# platform/buildkit — rootless BuildKit, the CI image builder
|
|
|
|
The builder behind [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). It
|
|
builds `toolbox`, `agent-runner` and `conformance-runner`, and on `main` pushes them to the
|
|
Gitea registry. Flux reconciles it as the Kustomization `buildkit`
|
|
([`clusters/base/buildkit.yaml`](../../clusters/base/buildkit.yaml), `dependsOn: platform`).
|
|
|
|
## Why it exists (operator decision on PR #81, 2026-10-29)
|
|
|
|
CI jobs run as containers on the act runner's privileged DinD sidecar, and **cannot reach any
|
|
Docker daemon**. act_runner 0.2.11 forces `docker_host: "-"` for a `tcp://` DinD, and gitea/act
|
|
strips every bind mount when `valid_volumes` is empty. Measured on PR #81's first run. Giving jobs
|
|
that daemon would make every branch push root on the node. So the rule is: **no job may reach a
|
|
privileged daemon.** Builds go to this rootless daemon over mTLS instead.
|
|
|
|
## Shape
|
|
|
|
| Piece | What it is |
|
|
|---|---|
|
|
| `deployment.yaml` | `moby/buildkit:v0.34.1-rootless`, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The `securityContext` is commented line by line with how each item was measured |
|
|
| `certificates.yaml` | cert-manager: a self-signed CA used only here, plus the server cert `buildkitd-tls` and the CI client cert `buildkit-client-ci` |
|
|
| `service.yaml` | ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`, mTLS only. Never on the Gateway |
|
|
| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and, for the registry push, the Gateway (`ingress` entity) **plus** Gitea's backend identity on :3000 — Cilium's Gateway re-checks this pod's egress against the backend it proxies to, and without that rule answers `403 Access denied` itself (measured 2026-10-30, `decisions/2026-10-30-gitea-admit-buildkit.md`) |
|
|
| `configmap.yaml` | `buildkitd.toml`: cache GC capped at 4 GB, because the cache is an `emptyDir` on node disk |
|
|
| `versions.env` | The CI side: pinned `buildx` (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource |
|
|
|
|
## Admission
|
|
|
|
**No Kyverno exception is needed.**
|
|
- Every ClusterPolicy that is actually applied (the list in
|
|
[`platform-config/kustomization.yaml`](../../platform-config/kustomization.yaml)) matches only
|
|
namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`. This namespace, like
|
|
`gitea-actions`, does not carry that label.
|
|
- The job-shape policies (`pm-agent-job-shape`, `conformance-trigger-job-shape`) match only their
|
|
own namespaces.
|
|
- There is no cluster-wide Pod Security default. This namespace is deliberately left without a
|
|
`pod-security` label, because PSS `baseline` forbids the `Unconfined` seccomp and AppArmor
|
|
profiles that rootless BuildKit needs.
|
|
|
|
## Prerequisites (operator, in order)
|
|
|
|
1. **Unprivileged user namespaces on the nodes.** Already true. Measured 2026-10-29 on all three
|
|
Debian 13.7 nodes (kernel 7.1.8): `kernel.unprivileged_userns_clone = 1`,
|
|
`user.max_user_namespaces = 127206`, and no Ubuntu-style AppArmor userns restriction.
|
|
2. **Merge.** Flux applies this directory. Check that the Kustomization `buildkit` is Ready and
|
|
that all three Certificates in namespace `buildkit` are READY.
|
|
3. **Copy the client certificate to CI.** Run
|
|
[`infra/gitea/buildkit-client-secret.sh`](../../infra/gitea/buildkit-client-secret.sh).
|
|
Re-run it yearly, after renewal; `--check` reports whether the copy is stale.
|
|
4. **Restart buildkitd after the server certificate renews.** It reads the certificate at start,
|
|
and cert-manager renews it 30 days before expiry, so a restart inside that window is needed.
|
|
|
|
Until step 3 is done, the `images` workflow fails at its builder preflight with a message naming
|
|
the cause. It never skips.
|
|
|
|
## What a build step can do
|
|
|
|
The daemon runs with `--oci-worker-no-process-sandbox`, so build steps share its PID namespace.
|
|
buildkitd warns about this itself at startup.
|
|
|
|
- **A build step CAN** signal or ptrace the daemon and read what it can read, including this
|
|
pod's TLS key and the build cache.
|
|
- **A build step CANNOT** reach the node. Everything runs as uid 1000 in a user namespace,
|
|
unprivileged, with no host mounts.
|
|
- **Who can run one:** anyone holding the client certificate, i.e. every CI job, including pull
|
|
requests.
|
|
|
|
## Local equivalent
|
|
|
|
The pinned image runs under rootless podman with the same restrictions:
|
|
`--cap-drop ALL --cap-add SETUID --cap-add SETGID --security-opt seccomp=unconfined --read-only`,
|
|
plus tmpfs at the same five paths. That is how the `securityContext` was measured. Rootless podman
|
|
maps only 65,537 uids, so the image's `/etc/subuid` (`user:100000:65536`) must be overridden with
|
|
two ranges that fit: `user:1:999` and `user:1001:64535`.
|