feat(images): build on rootless BuildKit over mTLS; verification moves into Containerfile verify stages

Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched.

- platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by
  digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined,
  read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP
  :1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it).
  No Kyverno exception: every applied policy matches app-namespace=true only.
- Each Containerfile names its shipped stage and ends in a `verify` stage holding every former
  `docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for
  the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now.
- images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify
  targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes.
  Fails loudly when the builder secrets are absent or buildkitd is unreachable.
- infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed 2026-10-29 22:21:03 -05:00
1 parent 65a7769f55
commit 310deb0f7d
28 files changed
+1490 -608

No files matched your search

+74
View File
@@ -0,0 +1,74 @@
# platform/buildkit — rootless BuildKit, the CI image builder
The builder behind [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). It
builds `toolbox`, `agent-runner` and `conformance-runner`, and on `main` pushes them to the
Gitea registry. Flux reconciles it as the Kustomization `buildkit`
([`clusters/base/buildkit.yaml`](../../clusters/base/buildkit.yaml), `dependsOn: platform`).
## Why it exists (operator decision on PR #81, 2026-10-29)
CI jobs run as containers on the act runner's privileged DinD sidecar, and **cannot reach any
Docker daemon**. act_runner 0.2.11 forces `docker_host: "-"` for a `tcp://` DinD, and gitea/act
strips every bind mount when `valid_volumes` is empty. Measured on PR #81's first run. Giving jobs
that daemon would make every branch push root on the node. So the rule is: **no job may reach a
privileged daemon.** Builds go to this rootless daemon over mTLS instead.
## Shape
| Piece | What it is |
|---|---|
| `deployment.yaml` | `moby/buildkit:v0.34.1-rootless`, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The `securityContext` is commented line by line with how each item was measured |
| `certificates.yaml` | cert-manager: a self-signed CA used only here, plus the server cert `buildkitd-tls` and the CI client cert `buildkit-client-ci` |
| `service.yaml` | ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`, mTLS only. Never on the Gateway |
| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and the Gateway (`ingress` entity) for the registry push |
| `configmap.yaml` | `buildkitd.toml`: cache GC capped at 4 GB, because the cache is an `emptyDir` on node disk |
| `versions.env` | The CI side: pinned `buildx` (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource |
## Admission
**No Kyverno exception is needed.**
- Every ClusterPolicy that is actually applied (the list in
[`platform-config/kustomization.yaml`](../../platform-config/kustomization.yaml)) matches only
namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`. This namespace, like
`gitea-actions`, does not carry that label.
- The job-shape policies (`pm-agent-job-shape`, `conformance-trigger-job-shape`) match only their
own namespaces.
- There is no cluster-wide Pod Security default. This namespace is deliberately left without a
`pod-security` label, because PSS `baseline` forbids the `Unconfined` seccomp and AppArmor
profiles that rootless BuildKit needs.
## Prerequisites (operator, in order)
1. **Unprivileged user namespaces on the nodes.** Already true. Measured 2026-10-29 on all three
Debian 13.7 nodes (kernel 7.1.8): `kernel.unprivileged_userns_clone = 1`,
`user.max_user_namespaces = 127206`, and no Ubuntu-style AppArmor userns restriction.
2. **Merge.** Flux applies this directory. Check that the Kustomization `buildkit` is Ready and
that all three Certificates in namespace `buildkit` are READY.
3. **Copy the client certificate to CI.** Run
[`infra/gitea/buildkit-client-secret.sh`](../../infra/gitea/buildkit-client-secret.sh).
Re-run it yearly, after renewal; `--check` reports whether the copy is stale.
4. **Restart buildkitd after the server certificate renews.** It reads the certificate at start,
and cert-manager renews it 30 days before expiry, so a restart inside that window is needed.
Until step 3 is done, the `images` workflow fails at its builder preflight with a message naming
the cause. It never skips.
## What a build step can do
The daemon runs with `--oci-worker-no-process-sandbox`, so build steps share its PID namespace.
buildkitd warns about this itself at startup.
- **A build step CAN** signal or ptrace the daemon and read what it can read, including this
pod's TLS key and the build cache.
- **A build step CANNOT** reach the node. Everything runs as uid 1000 in a user namespace,
unprivileged, with no host mounts.
- **Who can run one:** anyone holding the client certificate, i.e. every CI job, including pull
requests.
## Local equivalent
The pinned image runs under rootless podman with the same restrictions:
`--cap-drop ALL --cap-add SETUID --cap-add SETGID --security-opt seccomp=unconfined --read-only`,
plus tmpfs at the same five paths. That is how the `securityContext` was measured. Rootless podman
maps only 65,537 uids, so the image's `/etc/subuid` (`user:100000:65536`) must be overridden with
two ranges that fit: `user:1:999` and `user:1001:64535`.
+100
View File
@@ -0,0 +1,100 @@
---
# mTLS for buildkitd, issued by cert-manager from a CA that exists ONLY for this purpose.
#
# buildkit-selfsigned (Issuer) --signs--> buildkit-ca (CA Certificate, 10y)
# buildkit-ca (Issuer) --signs--> buildkitd-tls (server, the daemon presents it)
# --signs--> buildkit-client-ci (client, CI presents it)
#
# buildkitd verifies EVERY client certificate against this CA (--tlscacert), so holding a client
# certificate signed by it is what "may build" means. Nothing else in the cluster trusts this CA,
# and it trusts nothing else.
#
# THE CLIENT CERTIFICATE REACHES CI AS ACTIONS SECRETS, not through the cluster: CI jobs hold no
# cluster credential (R-SEC-02). infra/gitea/buildkit-client-secret.sh copies ca.crt/tls.crt/
# tls.key from the Secret below into the repository secrets BUILDKIT_CA, BUILDKIT_CLIENT_CERT and
# BUILDKIT_CLIENT_KEY without printing them. Re-run it after every renewal (see renewBefore).
#
# ROTATION. cert-manager renews both leaves `renewBefore` their expiry and rewrites the Secrets.
# * server: buildkitd reads its certificate at START, so the pod must restart inside the 30-day
# renewal window (a node reboot by kured, a Deployment rollout, or a manual restart).
# * client: the Actions secrets are a COPY; re-run the script. Until then CI fails loudly at
# its TLS preflight (.gitea/workflows/scripts/images.sh), never silently.
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: buildkit-selfsigned
namespace: buildkit
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: buildkit-ca
namespace: buildkit
spec:
isCA: true
commonName: buildkit-ca
secretName: buildkit-ca
duration: 87600h # 10 years: rotating the CA means re-issuing everything and re-copying
privateKey:
algorithm: ECDSA
size: 256
issuerRef:
name: buildkit-selfsigned
kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: buildkit-ca
namespace: buildkit
spec:
ca:
secretName: buildkit-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: buildkitd-tls
namespace: buildkit
spec:
secretName: buildkitd-tls
commonName: buildkitd.buildkit.svc.cluster.local
dnsNames:
- buildkitd.buildkit.svc.cluster.local # what CI dials (platform/buildkit/versions.env)
- buildkitd.buildkit.svc
- buildkitd
duration: 8760h
renewBefore: 720h
usages:
- server auth
- digital signature
privateKey:
algorithm: ECDSA
size: 256
rotationPolicy: Always
issuerRef:
name: buildkit-ca
kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: buildkit-client-ci
namespace: buildkit
spec:
secretName: buildkit-client-ci
commonName: gitea-actions-image-pipeline
duration: 8760h
renewBefore: 720h
usages:
- client auth
- digital signature
privateKey:
algorithm: ECDSA
size: 256
rotationPolicy: Always
issuerRef:
name: buildkit-ca
kind: Issuer
+24
View File
@@ -0,0 +1,24 @@
---
# buildkitd.toml. Validated against the pinned image (v0.34.1-rootless): `buildctl debug workers
# -v` reports "Maximum used space: 4.29GB / Reserved space: 1.07GB" for the default policy.
#
# WHY 4GB. The cache lives in an emptyDir on the NODE's disk (deployment.yaml), and the nodes had
# 7-9 GB free when this was written. The largest image (toolbox) is ~1.4 GB uncompressed; the
# three images share their Fedora base. The emptyDir's sizeLimit sits above this cap, so GC
# trims the cache well before the kubelet would evict the pod for exceeding it.
apiVersion: v1
kind: ConfigMap
metadata:
name: buildkitd-config
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
data:
buildkitd.toml: |
[worker.oci]
enabled = true
gc = true
maxUsedSpace = "4GB"
reservedSpace = "1GB"
[worker.containerd]
enabled = false
+142
View File
@@ -0,0 +1,142 @@
---
# buildkitd, ROOTLESS. The builder behind .gitea/workflows/images.yaml; see README.md.
#
# THE SECURITY CONTEXT IS THE BOUNDARY, AND EVERY LINE OF IT WAS MEASURED (2026-10-29, the pinned
# image under podman with the same restrictions; decisions/2026-10-29-image-pipeline.md):
# * privileged: false — never. This is the whole point of the operator's decision on PR #81:
# the alternative was handing CI the privileged DinD daemon, i.e. root on the node.
# * runAsUser/Group 1000 — the image's `user`; rootlesskit maps it to uid 0 INSIDE a user
# namespace only. The nodes allow that (measured: kernel.unprivileged_userns_clone = 1,
# user.max_user_namespaces = 127206 on all three Debian 13 nodes).
# * capabilities: drop ALL, add SETUID + SETGID — the image's newuidmap/newgidmap are NOT
# setuid; they carry FILE capabilities, which need SETUID/SETGID in the bounding set.
# Measured: with drop ALL and nothing added, rootlesskit fails "newuidmap ... operation not
# permitted" and the daemon never starts.
# * allowPrivilegeEscalation: true — required for the same reason: no_new_privs (what `false`
# sets) stops a file capability from ever taking effect. It does NOT grant privileges the
# bounding set above does not contain.
# * seccompProfile: Unconfined — measured: with the default seccomp profile the daemon starts
# but every RUN step fails (the runtime cannot create the build step's namespaces/mounts).
# * appArmorProfile: Unconfined — upstream's documented requirement (buildkit
# examples/kubernetes, *.rootless.yaml); the nodes run AppArmor (enabled=Y) and containerd's
# default profile denies mount. NOT measured here: the test host runs SELinux, not AppArmor.
# * readOnlyRootFilesystem: true — measured to work with the five emptyDirs below.
#
# --oci-worker-no-process-sandbox: build steps share the daemon's PID namespace instead of getting
# their own (creating one needs an unmasked /proc, i.e. more privilege). Consequence, stated by
# buildkitd itself at startup: a build step can signal or ptrace the daemon and read what the
# daemon can read — including this pod's TLS key and the build cache. So a build step is trusted
# at the level of "may impersonate this builder", which is the level anyone holding the client
# certificate already has. It can NOT reach the node: everything still runs as uid 1000 in a user
# namespace, unprivileged, with no host mounts.
apiVersion: apps/v1
kind: Deployment
metadata:
name: buildkitd
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
spec:
replicas: 1
strategy:
type: Recreate # one builder, one cache; never two daemons fighting over a node's disk
selector:
matchLabels:
app.kubernetes.io/name: buildkitd
template:
metadata:
labels:
app.kubernetes.io/name: buildkitd
spec:
automountServiceAccountToken: false # the builder needs no Kubernetes API at all
enableServiceLinks: false
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000 # makes the 0440 TLS Secret below readable by uid 1000, and nobody else
containers:
- name: buildkitd
# Pinned by digest (the multi-arch index). The kubernetes manager in renovate.json5
# tracks platform/**, so Renovate proposes bumps; the CI client (buildx, pinned in
# versions.env) talks to any recent daemon.
image: docker.io/moby/buildkit:v0.34.1-rootless@sha256:f68752a5a94acfcdad2d07867b7075601a421638e17fc5f28efc1c814d0c4f89
args:
- --addr
- unix:///run/user/1000/buildkit/buildkitd.sock # local only: the probes below
- --addr
- tcp://0.0.0.0:1234 # CI, mTLS only
- --tlscacert
- /certs/ca.crt # client certs are VERIFIED against this CA: no cert, no build
- --tlscert
- /certs/tls.crt
- --tlskey
- /certs/tls.key
- --config
- /etc/buildkit/buildkitd.toml
- --oci-worker-no-process-sandbox
ports:
- name: buildkit
containerPort: 1234
protocol: TCP
readinessProbe:
exec:
command: ["buildctl", "debug", "workers"]
initialDelaySeconds: 5
periodSeconds: 30
livenessProbe:
exec:
command: ["buildctl", "debug", "workers"]
initialDelaySeconds: 30
periodSeconds: 60
timeoutSeconds: 10
securityContext:
privileged: false
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
allowPrivilegeEscalation: true
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
add: ["SETUID", "SETGID"]
seccompProfile:
type: Unconfined
appArmorProfile:
type: Unconfined
resources:
# Build steps (dnf, npm, pip) run INSIDE this container's cgroup, so the limit covers
# the daemon and the build together.
requests:
cpu: 250m
memory: 512Mi
ephemeral-storage: 2Gi
limits:
memory: 4Gi
volumeMounts:
- { name: certs, mountPath: /certs, readOnly: true }
- { name: config, mountPath: /etc/buildkit, readOnly: true }
- { name: cache, mountPath: /home/user/.local/share/buildkit }
- { name: run, mountPath: /run/user }
- { name: tmpdir, mountPath: /home/user/.local/tmp } # the image's TMPDIR
- { name: tmp, mountPath: /tmp }
- { name: clicfg, mountPath: /home/user/.docker } # buildctl (the probes) writes here
volumes:
- name: certs
secret:
secretName: buildkitd-tls # cert-manager: ca.crt + tls.crt + tls.key
defaultMode: 0440
- name: config
configMap:
name: buildkitd-config
- name: cache
emptyDir:
sizeLimit: 6Gi # GC caps use at 4GB (configmap.yaml); this is the backstop
- name: run
emptyDir: {}
- name: tmpdir
emptyDir: {}
- name: tmp
emptyDir: {}
- name: clicfg
emptyDir: {}
+13
View File
@@ -0,0 +1,13 @@
---
# platform/buildkit — rootless BuildKit, the CI image builder. Reconciled by the Flux
# Kustomization `buildkit` (clusters/base/buildkit.yaml). See README.md.
# versions.env is NOT a resource: it pins the CI-side client (buildx) and the address CI dials.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- certificates.yaml
- configmap.yaml
- deployment.yaml
- service.yaml
- networkpolicy.yaml
+26
View File
@@ -0,0 +1,26 @@
---
# buildkit — the image builder for CI (.gitea/workflows/images.yaml). Operator decision on PR #81
# (2026-10-29): images are built by a ROOTLESS BuildKit daemon reached over mTLS, and NO CI job
# may reach a privileged daemon. See README.md for the whole design.
#
# A NAMESPACE OF ITS OWN, deliberately NOT `gitea-actions`:
# * The act-runner namespace exists to hold a PRIVILEGED DinD sidecar. Putting the builder
# beside it would put a TLS server key next to a privileged container for no benefit.
# * Its NetworkPolicies, Secrets and quota then belong to one component, and the ingress rule
# ("only the act-runner pod may connect") can be written from the outside.
#
# NO `app-namespace` LABEL, like gitea-actions: this is CI plumbing, not a user-facing app. That
# is also why NO Kyverno exception is needed — every applied ClusterPolicy
# (platform-config/kustomization.yaml) matches only namespaces labelled
# homelab.keithjackson.dev/app-namespace: "true", and none of the job-shape policies match here.
#
# NO pod-security.kubernetes.io LABEL, on purpose. Rootless BuildKit needs seccomp AND AppArmor
# `Unconfined` (measured — see deployment.yaml), which PSS `baseline` forbids; there is no
# cluster-wide PSA default here, so an unlabelled namespace admits it. It is NOT privileged and
# the deployment's securityContext is the boundary — read it before relaxing anything.
apiVersion: v1
kind: Namespace
metadata:
name: buildkit
labels:
homelab.keithjackson.dev/ci: buildkit
+88
View File
@@ -0,0 +1,88 @@
---
# Default-deny for the namespace, then two narrow allows.
#
# WHO MAY CONNECT — AND WHY THE ACT-RUNNER POD IS THE RIGHT SOURCE (measured 2026-10-29).
# CI jobs do not run as pods. act_runner starts each job as a container on its DinD sidecar, on a
# docker bridge network INSIDE the act-runner pod's network namespace, and dockerd masquerades it:
# kubectl -n gitea-actions exec deploy/act-runner -c dind -- iptables -t nat -S POSTROUTING
# -A POSTROUTING -s 172.18.0.0/16 ! -o br-c0f13d041ad6 -j MASQUERADE
# -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
# So a job's connection leaves with the act-runner POD's IP and therefore Cilium's identity for
# that pod (namespace gitea-actions, app.kubernetes.io/name=act-runner). That is exactly what the
# ingress rule names. It cannot distinguish one job from another, or a job from act_runner
# itself — per-job authentication is the mTLS client certificate, not this policy. What the
# policy adds is that NOTHING ELSE in the cluster can even open the port.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: buildkit
labels:
homelab.keithjackson.dev/baseline: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress: []
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: buildkitd-allow-ci-ingress
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: buildkitd
ingress:
- fromEndpoints:
- matchLabels:
io.kubernetes.pod.namespace: gitea-actions
app.kubernetes.io/name: act-runner
toPorts:
- ports:
- { port: "1234", protocol: TCP }
---
# WHAT A BUILD NEEDS TO REACH. Build steps download (dnf mirrors, GitHub releases, dl.k8s.io,
# npm, PyPI, Fedora/Docker registries) and the result is pushed to the Gitea registry.
# * world 443/80: the downloads. 80 because Fedora's mirror list includes plain-http mirrors
# (dnf verifies package GPG signatures, not transport).
# * ingress (the Gateway's Envoy, reserved:ingress): the push. gitea.home.keithjackson.dev
# resolves in-cluster to the Gateway VIP (split-horizon CoreDNS), and Cilium hands that
# traffic to the Gateway, whose identity is `ingress` — the same identity gitea's own
# ingress allow admits.
# Nothing else in the cluster: not the API server, not other pods. A build step is code from a
# pull request.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: buildkitd-allow-egress
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: buildkitd
egress:
- toEntities:
- world
toPorts:
- ports:
- { port: "443", protocol: TCP }
- { port: "80", protocol: TCP }
- toEntities:
- ingress
+19
View File
@@ -0,0 +1,19 @@
---
# The address CI dials: tcp://buildkitd.buildkit.svc.cluster.local:1234 (platform/buildkit/
# versions.env), mTLS only. ClusterIP — never exposed on the Gateway.
apiVersion: v1
kind: Service
metadata:
name: buildkitd
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: buildkitd
ports:
- name: buildkit
port: 1234
targetPort: buildkit
protocol: TCP
+15
View File
@@ -0,0 +1,15 @@
# platform/buildkit/versions.env — the CI side of the builder. Sourced by
# .gitea/workflows/scripts/images.sh. NOT a Kubernetes resource (kustomization.yaml omits it).
#
# The CLIENT. CI drives the daemon with `buildx` (remote driver) rather than bare `buildctl`
# because the image pipeline needs `bake`'s `target:` contexts: on a pull request agent-runner is
# built FROM the toolbox built in the same run, which never exists in any registry.
# Pinned with the sha256 from the release's own checksums.txt
# (https://github.com/docker/buildx/releases/download/v0.38.1/checksums.txt); the pipeline
# refuses a download that does not match. Bump BOTH lines together.
BUILDX_VERSION=0.38.1
BUILDX_SHA256=e25d5cd675092bb5d1d03632fdc46cdf3fede7b514dc5f023a105e00d3ce5ad4
# The DAEMON's address and the name its certificate carries (platform/buildkit/certificates.yaml).
BUILDKIT_HOST=tcp://buildkitd.buildkit.svc.cluster.local:1234
BUILDKIT_SERVERNAME=buildkitd.buildkit.svc.cluster.local
+3 -3
View File
@@ -47,9 +47,9 @@ Both are deliberate, and both fail **visibly** rather than silently:
1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
`PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder until the first
publish. `.gitea/workflows/images.yaml` builds `agent-runner` with
`infra/agent/build-image.sh` (the same script, and the same in-image verification, you can
run locally), and on a push to `main` publishes it to
publish. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
(`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that
`infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to
`gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
Renovate then follows the `main` tag through the `# renovate-digest:` marker above **both**
lines and moves them together in one pull request. Nobody builds-and-pushes this image from