feat(images): build on rootless BuildKit over mTLS; verification moves into Containerfile verify stages
Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched. - platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined, read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP :1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it). No Kyverno exception: every applied policy matches app-namespace=true only. - Each Containerfile names its shipped stage and ends in a `verify` stage holding every former `docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now. - images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes. Fails loudly when the builder secrets are absent or buildkitd is unreachable. - infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
8f35234fcc
commit
f65c12b395
28 files changed
+1490
-608
No files matched your search
@@ -120,10 +120,20 @@ CI. That is a real coverage gap, not a phase.
|
|||||||
This repo's three images — `toolbox` (`infra/devcontainer/`), `agent-runner` (`infra/agent/`,
|
This repo's three images — `toolbox` (`infra/devcontainer/`), `agent-runner` (`infra/agent/`,
|
||||||
built `FROM` the toolbox) and `conformance-runner` (`conformance/incluster/`) — are built,
|
built `FROM` the toolbox) and `conformance-runner` (`conformance/incluster/`) — are built,
|
||||||
verified and published by [`images.yaml`](images.yaml), never from a workstation
|
verified and published by [`images.yaml`](images.yaml), never from a workstation
|
||||||
(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh); each
|
(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh).
|
||||||
image is built by **its own** `build-image.sh`, the same script a developer runs locally, so the
|
|
||||||
in-image verification (non-root uid, read-only rootfs, tool presence, `PM_SELFTEST=1
|
**The builder is rootless BuildKit** ([`platform/buildkit/`](../../platform/buildkit/README.md)),
|
||||||
pm-dispatch`, the `conformance-clone` refusals) exists once.
|
reached over mTLS with a pinned `buildx` — no job here touches a Docker daemon, privileged or
|
||||||
|
otherwise (operator decision on PR #81). The client certificate reaches jobs as the Actions
|
||||||
|
secrets `BUILDKIT_CA`, `BUILDKIT_CLIENT_CERT`, `BUILDKIT_CLIENT_KEY`.
|
||||||
|
|
||||||
|
**Verification is a build target.** Each Containerfile ends in a `verify` stage that runs every
|
||||||
|
in-image assertion at build time as the non-root uid with `RUN --network=none` (tool presence and
|
||||||
|
pinned versions, `PM_SELFTEST=1 pm-dispatch`, the pm-worker probe, the `conformance-clone`
|
||||||
|
refusals, no writable path outside the workspace). The pipeline builds `<image>-verify` before it
|
||||||
|
builds, let alone pushes, the image. Build args come from each image's own
|
||||||
|
`build-image.sh --print-build-args`, which is also the local podman path — so the recipe and the
|
||||||
|
proof exist once.
|
||||||
|
|
||||||
| Trigger | What happens |
|
| Trigger | What happens |
|
||||||
|---|---|
|
|---|---|
|
||||||
@@ -143,8 +153,10 @@ account with package write on the org). The automatic Actions token cannot push
|
|||||||
[`infra/gitea/registry-retention.sh`](../../infra/gitea/registry-retention.sh), an operator
|
[`infra/gitea/registry-retention.sh`](../../infra/gitea/registry-retention.sh), an operator
|
||||||
action, keeps every digest git pins plus the newest tags.
|
action, keeps every digest git pins plus the newest tags.
|
||||||
|
|
||||||
Cost on this runner: the job shares the single act_runner slot with the gate, and the DinD store
|
Cost: the job shares the single act_runner slot with the gate, but the build itself runs in the
|
||||||
is a node `emptyDir`; the job prunes it on exit.
|
`buildkitd` pod, not in DinD, so the runner's disk and memory are untouched. Until
|
||||||
|
`platform/buildkit` is deployed and its client certificate copied
|
||||||
|
(`infra/gitea/buildkit-client-secret.sh`), the job fails loudly at its builder preflight.
|
||||||
|
|
||||||
## Files
|
## Files
|
||||||
|
|
||||||
|
|||||||
@@ -41,21 +41,22 @@
|
|||||||
# not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER`
|
# not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER`
|
||||||
# is optional and only labels the login.
|
# is optional and only labels the login.
|
||||||
#
|
#
|
||||||
# ┌─ BLOCKER, MEASURED ON PR #81 (run 325): THIS RUNNER CANNOT BUILD IMAGES YET ───────────────┐
|
# THE BUILDER — rootless BuildKit (operator decision on PR #81, 2026-10-29).
|
||||||
# │ The job container gets no Docker daemon. act_runner 0.2.11 forces container.docker_host to │
|
# The act runner's job containers get NO Docker daemon (act_runner 0.2.11 forces docker_host "-"
|
||||||
# │ "-" whenever DOCKER_HOST is tcp:// (internal/app/cmd/daemon.go), and with valid_volumes: [] │
|
# for a tcp:// DinD, and gitea/act strips every bind when valid_volumes is empty — measured on
|
||||||
# │ gitea/act strips every bind including the socket (pkg/container/docker_run.go │
|
# this PR's first run), and handing them the privileged DinD daemon would be root on the node for
|
||||||
# │ sanitizeConfig). So this job fails at its first check, loudly, until the operator chooses │
|
# anyone who can push a branch. So images are built by platform/buildkit/: a ROOTLESS buildkitd
|
||||||
# │ how workflow code may reach a daemon — see decisions/2026-10-29-image-pipeline.md. It is │
|
# (not privileged; uid 1000 in a user namespace), reached over mTLS at
|
||||||
# │ not a required check, so it does not block merges; it is red, which is the honest state. │
|
# buildkitd.buildkit.svc.cluster.local:1234 with a pinned `buildx` (remote driver). Its client
|
||||||
# └──────────────────────────────────────────────────────────────────────────────────────────────┘
|
# certificate reaches jobs as three Actions secrets — BUILDKIT_CA, BUILDKIT_CLIENT_CERT,
|
||||||
|
# BUILDKIT_CLIENT_KEY — copied from cert-manager by infra/gitea/buildkit-client-secret.sh.
|
||||||
|
# Pull requests get the client certificate (they must build) but never a registry credential.
|
||||||
|
#
|
||||||
|
# UNTIL platform/buildkit IS DEPLOYED (Flux applies it only after merge) THIS JOB FAILS, LOUDLY,
|
||||||
|
# at its builder preflight. A skipped Gitea job reports `success`, so it is red on purpose.
|
||||||
|
#
|
||||||
|
# VERIFICATION is each Containerfile's `verify` stage, built before anything is pushed.
|
||||||
#
|
#
|
||||||
# THE RUNNER (apps/wave-1/gitea-actions.yaml): act_runner 0.2.11, one job at a time, Docker
|
|
||||||
# executor on a privileged `docker:27-dind` sidecar whose image store is an emptyDir on the node
|
|
||||||
# (no sizeLimit) and whose memory limit is 2Gi — shared by the job container AND every build
|
|
||||||
# container. The toolbox is ~1.4 GB uncompressed and agent-runner adds ~0.3 GB on top of it, so the
|
|
||||||
# cleanup step below is not optional: without it each run would leave ~3 GB on a node that had
|
|
||||||
# 7 GB free when this was written, and the next eviction would take the runner with it.
|
|
||||||
name: images
|
name: images
|
||||||
|
|
||||||
on:
|
on:
|
||||||
@@ -71,6 +72,7 @@ on:
|
|||||||
- conformance/incluster/build-image.sh
|
- conformance/incluster/build-image.sh
|
||||||
- .gitea/workflows/images.yaml
|
- .gitea/workflows/images.yaml
|
||||||
- .gitea/workflows/scripts/images.sh
|
- .gitea/workflows/scripts/images.sh
|
||||||
|
- platform/buildkit/versions.env
|
||||||
push:
|
push:
|
||||||
branches: [main]
|
branches: [main]
|
||||||
paths:
|
paths:
|
||||||
@@ -83,6 +85,7 @@ on:
|
|||||||
- conformance/incluster/build-image.sh
|
- conformance/incluster/build-image.sh
|
||||||
- .gitea/workflows/images.yaml
|
- .gitea/workflows/images.yaml
|
||||||
- .gitea/workflows/scripts/images.sh
|
- .gitea/workflows/scripts/images.sh
|
||||||
|
- platform/buildkit/versions.env
|
||||||
workflow_dispatch:
|
workflow_dispatch:
|
||||||
inputs:
|
inputs:
|
||||||
images:
|
images:
|
||||||
@@ -115,6 +118,11 @@ jobs:
|
|||||||
MODE: verify
|
MODE: verify
|
||||||
IMAGES: auto
|
IMAGES: auto
|
||||||
PR_BASE_REF: ${{ github.base_ref }}
|
PR_BASE_REF: ${{ github.base_ref }}
|
||||||
|
# The BUILDER's client certificate (platform/buildkit). Not a registry credential:
|
||||||
|
# a pull request can build, never push.
|
||||||
|
BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
|
||||||
|
BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
|
||||||
|
BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
|
||||||
run: |
|
run: |
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
BASE_SHA="$(git merge-base "origin/${PR_BASE_REF}" HEAD)"
|
BASE_SHA="$(git merge-base "origin/${PR_BASE_REF}" HEAD)"
|
||||||
@@ -127,6 +135,9 @@ jobs:
|
|||||||
env:
|
env:
|
||||||
MODE: verify
|
MODE: verify
|
||||||
IMAGES: ${{ inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images }}
|
IMAGES: ${{ inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images }}
|
||||||
|
BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
|
||||||
|
BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
|
||||||
|
BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
|
||||||
run: ./.gitea/workflows/scripts/images.sh
|
run: ./.gitea/workflows/scripts/images.sh
|
||||||
|
|
||||||
# ── main: publish ───────────────────────────────────────────────────────────────────────
|
# ── main: publish ───────────────────────────────────────────────────────────────────────
|
||||||
@@ -168,16 +179,7 @@ jobs:
|
|||||||
BASE_SHA: ${{ github.event.before }}
|
BASE_SHA: ${{ github.event.before }}
|
||||||
REGISTRY_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
|
REGISTRY_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
|
||||||
REGISTRY_USER: ${{ vars.REGISTRY_PUSH_USER }}
|
REGISTRY_USER: ${{ vars.REGISTRY_PUSH_USER }}
|
||||||
|
BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
|
||||||
|
BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
|
||||||
|
BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
|
||||||
run: ./.gitea/workflows/scripts/images.sh
|
run: ./.gitea/workflows/scripts/images.sh
|
||||||
|
|
||||||
# ── always: give the DinD store back ────────────────────────────────────────────────────
|
|
||||||
# The runner's image store is a node emptyDir with no sizeLimit (see header). Everything this
|
|
||||||
# job built or pulled is removed; `prune -a` cannot remove the job's own image, because the
|
|
||||||
# job container is still running on it.
|
|
||||||
- name: Clean the DinD image store
|
|
||||||
if: always()
|
|
||||||
run: |
|
|
||||||
docker logout gitea.home.keithjackson.dev >/dev/null 2>&1 || true
|
|
||||||
docker image prune -af >/dev/null || true
|
|
||||||
docker builder prune -af >/dev/null || true
|
|
||||||
docker system df || true
|
|
||||||
+196
-119
@@ -4,35 +4,41 @@
|
|||||||
#
|
#
|
||||||
# Builds, verifies and (on main) PUBLISHES the three images this repo ships:
|
# Builds, verifies and (on main) PUBLISHES the three images this repo ships:
|
||||||
#
|
#
|
||||||
# toolbox infra/devcontainer/ the pinned dev/gate toolchain
|
# toolbox infra/devcontainer/ the pinned dev/gate toolchain
|
||||||
# agent-runner infra/agent/ FROM toolbox; the PM dispatcher + worker
|
# agent-runner infra/agent/ FROM toolbox; the PM dispatcher + worker
|
||||||
# conformance-runner conformance/incluster/ the in-cluster L2/L3 Chainsaw runner
|
# conformance-runner conformance/incluster/ the in-cluster L2/L3 Chainsaw runner
|
||||||
#
|
#
|
||||||
# Each image is built by ITS OWN build-image.sh — the same script a developer runs locally —
|
# THE BUILDER (operator decision on PR #81): a ROOTLESS BuildKit daemon, platform/buildkit/,
|
||||||
# so the in-image verification (non-root uid, read-only rootfs, tool presence, pm-dispatch
|
# reached over mTLS. No CI job may reach a privileged daemon, so there is no `docker build` and no
|
||||||
# selftest, conformance-clone refusals, ...) is written once and runs in both places. This
|
# `docker run` here: this script drives buildkitd with a pinned `buildx` (remote driver) and
|
||||||
# file only decides WHAT to build, in which order, FROM which base, and where it goes.
|
# nothing else. The client certificate arrives as Actions secrets (BUILDKIT_CA,
|
||||||
|
# BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY; infra/gitea/buildkit-client-secret.sh puts them
|
||||||
|
# there). If the daemon is unreachable this FAILS LOUDLY — it never skips.
|
||||||
|
#
|
||||||
|
# VERIFICATION IS A BUILD TARGET. Each Containerfile ends in a `verify` stage that runs every
|
||||||
|
# in-image assertion at build time (as the non-root uid, with RUN --network=none); this script
|
||||||
|
# builds `<image>-verify` and only then the image. Build args come from each image's own
|
||||||
|
# `build-image.sh --print-build-args`, the same list the local podman path uses.
|
||||||
#
|
#
|
||||||
# MODES (env MODE):
|
# MODES (env MODE):
|
||||||
# verify build + verify, never push, holds NO credential. Pull requests.
|
# verify build the verify targets, never push, hold no registry credential. Pull requests.
|
||||||
# publish build + verify + push to ${REGISTRY}/${NAMESPACE}/<image> with tags
|
# publish verify, then build + push to ${REGISTRY}/${NAMESPACE}/<image> with tags
|
||||||
# `sha-<12 hex>` (immutable: never re-pushed) and `main` (moving; Renovate follows
|
# `sha-<12 hex>` (immutable: never re-pushed) and `main` (moving; Renovate follows
|
||||||
# it). The digest recorded is the one the REGISTRY returns, read back after the push.
|
# it). The digest recorded is the one the REGISTRY returns, read back after the push.
|
||||||
#
|
#
|
||||||
# WHAT GETS BUILT (env IMAGES): `auto` (default) diffs BASE_SHA..HEAD_SHA against each image's
|
# WHAT GETS BUILT (env IMAGES): `auto` (default) diffs BASE_SHA..HEAD_SHA against each image's
|
||||||
# inputs; otherwise a space-separated list. A change to the pipeline itself (images.yaml, this
|
# inputs; otherwise a space-separated list. A change to the pipeline itself (images.yaml, this
|
||||||
# file) builds everything in verify mode, and nothing extra in publish mode: re-publishing
|
# file, platform/buildkit/versions.env) builds everything in verify mode, and nothing extra in
|
||||||
# unchanged inputs would only mint new digests and churn Renovate PRs.
|
# publish mode: re-publishing unchanged inputs would only mint new digests and churn Renovate PRs.
|
||||||
#
|
#
|
||||||
# THE agent-runner BASE — the one ordering rule, and the reason this is not three jobs:
|
# THE agent-runner BASE:
|
||||||
# * toolbox built in this run -> agent-runner builds FROM it (in publish mode, FROM the
|
# * verify mode -> FROM the toolbox built in this run, linked with a bake `target:` context —
|
||||||
# digest the registry just returned for it).
|
# a pull request holds no registry credential and that toolbox exists in no registry. If the
|
||||||
# * verify mode, toolbox not changed -> the toolbox is built anyway as a base. A pull request
|
# toolbox's inputs did not change it is built anyway, as the base.
|
||||||
# holds no registry credential, so it cannot pull the published one.
|
# * publish mode, toolbox published in this run -> FROM the digest the registry just returned.
|
||||||
# * publish mode, toolbox not changed -> AGENT_BASE_IMAGE from infra/agent/versions.env,
|
# * publish mode otherwise -> AGENT_BASE_IMAGE from infra/agent/versions.env (the published
|
||||||
# i.e. the published toolbox digest that Renovate keeps current. If that is still the
|
# toolbox digest Renovate keeps current). If that is still the all-zero placeholder the
|
||||||
# all-zero placeholder the toolbox has never been published, so it is built and published
|
# toolbox has never been published, so it is built and published first (the bootstrap).
|
||||||
# first (this is how the very first run bootstraps).
|
|
||||||
#
|
#
|
||||||
# LOOP-FREEDOM. Renovate turns each publish into a digest-bump PR. Those bumps land in
|
# LOOP-FREEDOM. Renovate turns each publish into a digest-bump PR. Those bumps land in
|
||||||
# platform/pm-agent/ (no image input), conformance/versions.env (no image input) and
|
# platform/pm-agent/ (no image input), conformance/versions.env (no image input) and
|
||||||
@@ -46,9 +52,14 @@ set -euo pipefail
|
|||||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
|
||||||
cd "${REPO_ROOT}"
|
cd "${REPO_ROOT}"
|
||||||
|
|
||||||
|
# shellcheck disable=SC1091
|
||||||
|
BK_HOST_OVERRIDE="${BUILDKIT_HOST:-}"; BK_NAME_OVERRIDE="${BUILDKIT_SERVERNAME:-}"
|
||||||
|
. ./platform/buildkit/versions.env
|
||||||
|
BUILDKIT_HOST="${BK_HOST_OVERRIDE:-${BUILDKIT_HOST}}"
|
||||||
|
BUILDKIT_SERVERNAME="${BK_NAME_OVERRIDE:-${BUILDKIT_SERVERNAME}}"
|
||||||
|
|
||||||
MODE="${MODE:-verify}"
|
MODE="${MODE:-verify}"
|
||||||
IMAGES="${IMAGES:-auto}"
|
IMAGES="${IMAGES:-auto}"
|
||||||
ENGINE="${ENGINE:-docker}"
|
|
||||||
REGISTRY="${REGISTRY:-gitea.home.keithjackson.dev}"
|
REGISTRY="${REGISTRY:-gitea.home.keithjackson.dev}"
|
||||||
NAMESPACE="${NAMESPACE:-home-network}"
|
NAMESPACE="${NAMESPACE:-home-network}"
|
||||||
HEAD_SHA="${HEAD_SHA:-$(git rev-parse HEAD)}"
|
HEAD_SHA="${HEAD_SHA:-$(git rev-parse HEAD)}"
|
||||||
@@ -63,13 +74,15 @@ MOVING_TAG="main"
|
|||||||
log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
||||||
info() { printf ' %s\n' "$*"; }
|
info() { printf ' %s\n' "$*"; }
|
||||||
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; printf -- '- **ERROR**: %s\n' "$*" >>"${SUMMARY}"; exit 1; }
|
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; printf -- '- **ERROR**: %s\n' "$*" >>"${SUMMARY}"; exit 1; }
|
||||||
|
yn() { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; }
|
||||||
|
|
||||||
case "${MODE}" in verify|publish) ;; *) die "MODE must be verify or publish, got '${MODE}'" ;; esac
|
case "${MODE}" in verify|publish) ;; *) die "MODE must be verify or publish, got '${MODE}'" ;; esac
|
||||||
|
for t in git jq curl sha256sum; do command -v "$t" >/dev/null || die "the job image lacks '$t'"; done
|
||||||
|
|
||||||
# --- inputs per image: a change under any of these rebuilds that image ---------------------
|
# --- per-image facts ------------------------------------------------------------------------
|
||||||
# conformance-runner takes its chainsaw/kubectl/fedora pins from infra/devcontainer/versions.env.
|
# Inputs: a change under any of these rebuilds that image. conformance-runner takes its pins from
|
||||||
# conformance/versions.env is NOT an input (it only RECORDS the published digest), and must not
|
# infra/devcontainer/versions.env. conformance/versions.env is NOT an input (it only RECORDS the
|
||||||
# become one: Renovate writes it after every publish, and an input there would loop.
|
# published digest), and must not become one: Renovate writes it after every publish -> a loop.
|
||||||
inputs() {
|
inputs() {
|
||||||
case "$1" in
|
case "$1" in
|
||||||
toolbox) echo "infra/devcontainer/Containerfile infra/devcontainer/versions.env infra/devcontainer/build-image.sh" ;;
|
toolbox) echo "infra/devcontainer/Containerfile infra/devcontainer/versions.env infra/devcontainer/build-image.sh" ;;
|
||||||
@@ -77,16 +90,17 @@ inputs() {
|
|||||||
conformance-runner) echo "conformance/incluster/Containerfile conformance/incluster/conformance-clone conformance/incluster/build-image.sh infra/devcontainer/versions.env" ;;
|
conformance-runner) echo "conformance/incluster/Containerfile conformance/incluster/conformance-clone conformance/incluster/build-image.sh infra/devcontainer/versions.env" ;;
|
||||||
esac
|
esac
|
||||||
}
|
}
|
||||||
PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh"
|
context_of() { case "$1" in toolbox) echo infra/devcontainer ;; agent-runner) echo infra/agent ;; conformance-runner) echo conformance/incluster ;; esac; }
|
||||||
|
PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh platform/buildkit/versions.env"
|
||||||
|
|
||||||
declare -A WANT=() BUILT=() DIGEST=() SECS=() NOTE=()
|
declare -A WANT=() BUILT=() DIGEST=() SECS=() NOTE=()
|
||||||
|
|
||||||
changed_any() { # <paths...> -> 0 if BASE_SHA..HEAD_SHA touches any
|
changed_any() { # <paths...> -> 0 if BASE_SHA..HEAD_SHA touches any
|
||||||
# shellcheck disable=SC2086
|
|
||||||
[ -n "$(git diff --name-only "${BASE_SHA}" "${HEAD_SHA}" -- "$@")" ]
|
[ -n "$(git diff --name-only "${BASE_SHA}" "${HEAD_SHA}" -- "$@")" ]
|
||||||
}
|
}
|
||||||
|
|
||||||
select_images() {
|
select_images() {
|
||||||
|
local i
|
||||||
if [ "${IMAGES}" != "auto" ]; then
|
if [ "${IMAGES}" != "auto" ]; then
|
||||||
for i in ${IMAGES}; do
|
for i in ${IMAGES}; do
|
||||||
case " ${ALL_IMAGES[*]} " in *" ${i} "*) WANT[$i]=1; NOTE[$i]="requested" ;; *) die "unknown image '${i}'" ;; esac
|
case " ${ALL_IMAGES[*]} " in *" ${i} "*) WANT[$i]=1; NOTE[$i]="requested" ;; *) die "unknown image '${i}'" ;; esac
|
||||||
@@ -99,7 +113,6 @@ select_images() {
|
|||||||
return
|
return
|
||||||
fi
|
fi
|
||||||
info "diff base ${BASE_SHA} .. ${HEAD_SHA}"
|
info "diff base ${BASE_SHA} .. ${HEAD_SHA}"
|
||||||
local i
|
|
||||||
for i in "${ALL_IMAGES[@]}"; do
|
for i in "${ALL_IMAGES[@]}"; do
|
||||||
# shellcheck disable=SC2046
|
# shellcheck disable=SC2046
|
||||||
if changed_any $(inputs "$i"); then WANT[$i]=1; NOTE[$i]="inputs changed"; fi
|
if changed_any $(inputs "$i"); then WANT[$i]=1; NOTE[$i]="inputs changed"; fi
|
||||||
@@ -110,6 +123,114 @@ select_images() {
|
|||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# --- the builder: pinned buildx + mTLS to rootless buildkitd ---------------------------------
|
||||||
|
WORK="$(mktemp -d)"
|
||||||
|
trap 'rm -rf "${WORK}"' EXIT # client key + registry auth live only here, only this long
|
||||||
|
export BUILDX_CONFIG="${WORK}/buildx" DOCKER_CONFIG="${WORK}/docker"
|
||||||
|
# Single-manifest images: the admission policies, Renovate and registry-retention.sh all reason
|
||||||
|
# about one manifest per tag. (Provenance attestations would turn every push into an index.)
|
||||||
|
export BUILDX_NO_DEFAULT_ATTESTATIONS=1
|
||||||
|
mkdir -p "${BUILDX_CONFIG}" "${DOCKER_CONFIG}" "${WORK}/certs"
|
||||||
|
echo '{}' > "${DOCKER_CONFIG}/config.json"
|
||||||
|
|
||||||
|
setup_builder() {
|
||||||
|
log "Builder: rootless buildkitd at ${BUILDKIT_HOST} (mTLS)"
|
||||||
|
BUILDX="${BUILDX_BIN:-}"
|
||||||
|
if [ -z "${BUILDX}" ]; then
|
||||||
|
BUILDX="${WORK}/buildx-bin"
|
||||||
|
curl -fsSL --retry 3 -o "${BUILDX}" \
|
||||||
|
"https://github.com/docker/buildx/releases/download/v${BUILDX_VERSION}/buildx-v${BUILDX_VERSION}.linux-amd64" \
|
||||||
|
|| die "could not download buildx v${BUILDX_VERSION}"
|
||||||
|
echo "${BUILDX_SHA256} ${BUILDX}" | sha256sum -c --quiet - \
|
||||||
|
|| die "buildx v${BUILDX_VERSION} does not match the sha256 pinned in platform/buildkit/versions.env — refusing to run it"
|
||||||
|
chmod +x "${BUILDX}"
|
||||||
|
fi
|
||||||
|
info "client: $("${BUILDX}" version)"
|
||||||
|
|
||||||
|
local missing=()
|
||||||
|
[ -n "${BUILDKIT_CA:-}" ] || missing+=(BUILDKIT_CA)
|
||||||
|
[ -n "${BUILDKIT_CLIENT_CERT:-}" ] || missing+=(BUILDKIT_CLIENT_CERT)
|
||||||
|
[ -n "${BUILDKIT_CLIENT_KEY:-}" ] || missing+=(BUILDKIT_CLIENT_KEY)
|
||||||
|
[ "${#missing[@]}" -eq 0 ] || die "the builder's client certificate is missing: ${missing[*]} not set.
|
||||||
|
These Actions secrets are copied from the cert-manager Secret buildkit/buildkit-client-ci by
|
||||||
|
infra/gitea/buildkit-client-secret.sh (an operator action, after platform/buildkit is
|
||||||
|
deployed). Without them nothing can be built — this fails rather than skipping ON PURPOSE."
|
||||||
|
umask 077
|
||||||
|
printf '%s\n' "${BUILDKIT_CA}" > "${WORK}/certs/ca.pem"
|
||||||
|
printf '%s\n' "${BUILDKIT_CLIENT_CERT}" > "${WORK}/certs/cert.pem"
|
||||||
|
printf '%s\n' "${BUILDKIT_CLIENT_KEY}" > "${WORK}/certs/key.pem"
|
||||||
|
umask 022
|
||||||
|
|
||||||
|
"${BUILDX}" create --name ci-remote --driver remote \
|
||||||
|
--driver-opt "cacert=${WORK}/certs/ca.pem,cert=${WORK}/certs/cert.pem,key=${WORK}/certs/key.pem,servername=${BUILDKIT_SERVERNAME}" \
|
||||||
|
"${BUILDKIT_HOST}" >/dev/null
|
||||||
|
if ! out="$(timeout 90 "${BUILDX}" inspect --bootstrap --builder ci-remote 2>&1)"; then
|
||||||
|
printf '%s\n' "${out}" >&2
|
||||||
|
die "buildkitd is UNREACHABLE at ${BUILDKIT_HOST} — nothing can be built, and this job fails
|
||||||
|
rather than skipping. In order of likelihood:
|
||||||
|
1. platform/buildkit is not deployed yet. Flux applies it only after it merges to main
|
||||||
|
(Kustomization 'buildkit'); a pull request that ADDS it cannot reach it.
|
||||||
|
2. the BUILDKIT_* secrets are stale (the client cert was renewed) or from another CA:
|
||||||
|
re-run infra/gitea/buildkit-client-secret.sh.
|
||||||
|
3. the NetworkPolicy (platform/buildkit/networkpolicy.yaml) does not admit this runner."
|
||||||
|
fi
|
||||||
|
printf '%s\n' "${out}" | grep -E '^(Name|Driver|Status|BuildKit version|Platforms):' | sed 's/^/ /' || true
|
||||||
|
}
|
||||||
|
|
||||||
|
# KEY=VALUE lines from an image's build-image.sh --print-build-args -> a JSON object.
|
||||||
|
build_args_json() {
|
||||||
|
local ctx; ctx="$(context_of "$1")"
|
||||||
|
"./${ctx}/build-image.sh" --print-build-args | jq -Rn '[inputs | select(length>0) | capture("^(?<k>[^=]+)=(?<v>.*)$")] | map({(.k): .v}) | add'
|
||||||
|
}
|
||||||
|
|
||||||
|
# bake_run <bakefile> <targets...>
|
||||||
|
bake_run() {
|
||||||
|
local f="$1"; shift
|
||||||
|
"${BUILDX}" bake --builder ci-remote --progress plain -f "$f" \
|
||||||
|
--metadata-file "${WORK}/meta.json" "$@"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Write a bake file with a verify target and a ship target for one image.
|
||||||
|
# $1 image, $2 agent base mode: "" | "link" | <image ref>
|
||||||
|
write_bake() {
|
||||||
|
local img="$1" base="${2:-}" ctx args f
|
||||||
|
ctx="$(context_of "$img")"
|
||||||
|
args="$(build_args_json "$img")"
|
||||||
|
f="${WORK}/bake-${img}.json"
|
||||||
|
local contexts='{}'
|
||||||
|
if [ "${img}" = agent-runner ]; then
|
||||||
|
if [ "${base}" = link ]; then
|
||||||
|
# FROM ${AGENT_BASE_IMAGE} resolves to the named context below, i.e. the toolbox target
|
||||||
|
# of this same bake file — built here, never pushed anywhere.
|
||||||
|
args="$(jq -c '. + {AGENT_BASE_IMAGE: "toolbox-in-this-run"}' <<<"${args}")"
|
||||||
|
contexts='{"toolbox-in-this-run": "target:toolbox"}'
|
||||||
|
else
|
||||||
|
args="$(jq -c --arg b "${base}" '. + {AGENT_BASE_IMAGE: $b}' <<<"${args}")"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
jq -n --arg img "$img" --arg ctx "$ctx" --argjson args "$args" --argjson contexts "$contexts" \
|
||||||
|
--arg tbargs "$( [ "${base}" = link ] && build_args_json toolbox || echo '{}')" \
|
||||||
|
--arg rev "${HEAD_SHA}" --arg repo "${REGISTRY}/${NAMESPACE}/${img}" \
|
||||||
|
--arg sha "${SHA_TAG}" --arg moving "${MOVING_TAG}" --arg mode "${MODE}" '
|
||||||
|
{target: (
|
||||||
|
{
|
||||||
|
($img + "-verify"): {context: $ctx, dockerfile: "Containerfile", target: "verify",
|
||||||
|
args: $args, contexts: $contexts, output: ["type=cacheonly"]},
|
||||||
|
($img): ({context: $ctx, dockerfile: "Containerfile", target: $img, args: $args,
|
||||||
|
contexts: $contexts,
|
||||||
|
labels: {"org.opencontainers.image.revision": $rev}}
|
||||||
|
+ (if $mode == "publish"
|
||||||
|
then {tags: [($repo + ":" + $sha), ($repo + ":" + $moving)], output: ["type=image,push=true"]}
|
||||||
|
else {output: ["type=cacheonly"]} end))
|
||||||
|
}
|
||||||
|
+ (if ($contexts | has("toolbox-in-this-run"))
|
||||||
|
then {toolbox: {context: "infra/devcontainer", dockerfile: "Containerfile", target: "toolbox",
|
||||||
|
args: ($tbargs | fromjson), output: ["type=cacheonly"]}}
|
||||||
|
else {} end)
|
||||||
|
)}' > "$f"
|
||||||
|
echo "$f"
|
||||||
|
}
|
||||||
|
|
||||||
# --- registry helpers (publish mode only) ----------------------------------------------------
|
# --- registry helpers (publish mode only) ----------------------------------------------------
|
||||||
REG_TOKEN=""
|
REG_TOKEN=""
|
||||||
registry_token() {
|
registry_token() {
|
||||||
@@ -120,8 +241,8 @@ registry_token() {
|
|||||||
[ -n "${REG_TOKEN}" ] || die "registry returned no bearer token"
|
[ -n "${REG_TOKEN}" ] || die "registry returned no bearer token"
|
||||||
}
|
}
|
||||||
# Prints the manifest digest the REGISTRY serves for <image>:<tag>; nothing if it is absent (404).
|
# Prints the manifest digest the REGISTRY serves for <image>:<tag>; nothing if it is absent (404).
|
||||||
# Any other answer (401, 5xx, no digest header) is fatal: "cannot tell" must never be read as
|
# Any other answer is fatal: "cannot tell" must never be read as "not published", or the
|
||||||
# "not published", or the immutable-tag guard would be bypassed by an auth hiccup.
|
# immutable-tag guard would be bypassed by an auth hiccup.
|
||||||
registry_digest() {
|
registry_digest() {
|
||||||
registry_token
|
registry_token
|
||||||
local hdrs code digest
|
local hdrs code digest
|
||||||
@@ -142,87 +263,50 @@ registry_digest() {
|
|||||||
esac
|
esac
|
||||||
}
|
}
|
||||||
|
|
||||||
local_ref() { echo "${REGISTRY}/${NAMESPACE}/$1:ci-${HEAD_SHA:0:12}"; }
|
# build_one <image> [agent base] — verify, then (publish) build + push + read back.
|
||||||
|
build_one() {
|
||||||
build_one() { # <image> [agent base]
|
local img="$1" base="${2:-}" t0 f pushed got existing
|
||||||
local img="$1" base="${2:-}" t0 tag
|
|
||||||
tag="ci-${HEAD_SHA:0:12}"
|
|
||||||
t0=$(date +%s)
|
t0=$(date +%s)
|
||||||
log "BUILD + VERIFY ${img}"
|
f="$(write_bake "$img" "$base")"
|
||||||
case "${img}" in
|
log "VERIFY ${img} (Containerfile target: verify, on rootless BuildKit)"
|
||||||
toolbox)
|
bake_run "$f" "${img}-verify"
|
||||||
ENGINE="${ENGINE}" IMAGE="$(local_ref toolbox)" BUILD_REVISION="${HEAD_SHA}" \
|
|
||||||
./infra/devcontainer/build-image.sh ;;
|
|
||||||
agent-runner)
|
|
||||||
ENGINE="${ENGINE}" IMAGE_TAG="${tag}" AGENT_BASE_IMAGE="${base}" BUILD_REVISION="${HEAD_SHA}" \
|
|
||||||
./infra/agent/build-image.sh ;;
|
|
||||||
conformance-runner)
|
|
||||||
ENGINE="${ENGINE}" IMAGE_TAG="${tag}" BUILD_REVISION="${HEAD_SHA}" \
|
|
||||||
./conformance/incluster/build-image.sh ;;
|
|
||||||
esac
|
|
||||||
BUILT[$img]=1
|
BUILT[$img]=1
|
||||||
SECS[$img]=$(( $(date +%s) - t0 ))
|
if [ "${MODE}" = publish ]; then
|
||||||
info "${img}: built and verified in ${SECS[$img]}s"
|
existing="$(registry_digest "${img}" "${SHA_TAG}")"
|
||||||
}
|
|
||||||
|
|
||||||
push_one() { # <image> -> sets DIGEST[image] to the registry-returned digest
|
|
||||||
local img="$1" repo src existing pushed got t0
|
|
||||||
repo="${REGISTRY}/${NAMESPACE}/${img}"
|
|
||||||
src="$(local_ref "${img}")"
|
|
||||||
t0=$(date +%s)
|
|
||||||
log "PUSH ${img} -> ${repo}:{${SHA_TAG},${MOVING_TAG}}"
|
|
||||||
existing="$(registry_digest "${img}" "${SHA_TAG}")"
|
|
||||||
if [ -n "${existing}" ]; then
|
|
||||||
# Immutable tag: never overwrite. A rebuild is not byte-reproducible, so re-pushing would
|
# Immutable tag: never overwrite. A rebuild is not byte-reproducible, so re-pushing would
|
||||||
# silently change what `sha-<commit>` means after something may already have pinned it.
|
# silently change what `sha-<commit>` means after something may already have pinned it.
|
||||||
die "${repo}:${SHA_TAG} already exists (${existing}). Tags here are immutable; this run must not
|
[ -z "${existing}" ] || die "${REGISTRY}/${NAMESPACE}/${img}:${SHA_TAG} already exists (${existing}); tags are immutable"
|
||||||
replace it. (publish skips images that already exist at this commit BEFORE building them,
|
log "BUILD + PUSH ${img} -> ${REGISTRY}/${NAMESPACE}/${img}:{${SHA_TAG},${MOVING_TAG}}"
|
||||||
so reaching this line means the tag appeared mid-run.)"
|
bake_run "$f" "${img}"
|
||||||
|
pushed="$(jq -r --arg t "$img" '.[$t]["containerimage.digest"] // empty' "${WORK}/meta.json")"
|
||||||
|
# The AUTHORITATIVE digest is what the registry serves, read back — not what the client
|
||||||
|
# reported. They should agree; if they do not, nothing downstream may trust either.
|
||||||
|
got="$(registry_digest "${img}" "${SHA_TAG}")"
|
||||||
|
[ -n "${got}" ] || die "pushed ${img}:${SHA_TAG} but the registry does not serve it"
|
||||||
|
[ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] || die "${img}: client reported ${pushed} but the registry serves ${got}"
|
||||||
|
[ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \
|
||||||
|
|| die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})"
|
||||||
|
DIGEST[$img]="${got}"
|
||||||
|
info "${img}: registry digest ${got}"
|
||||||
fi
|
fi
|
||||||
"${ENGINE}" tag "${src}" "${repo}:${SHA_TAG}"
|
SECS[$img]=$(( $(date +%s) - t0 ))
|
||||||
"${ENGINE}" tag "${src}" "${repo}:${MOVING_TAG}"
|
info "${img}: done in ${SECS[$img]}s"
|
||||||
pushed="$("${ENGINE}" push "${repo}:${SHA_TAG}" | tee /dev/stderr | grep -oE 'digest: sha256:[0-9a-f]{64}' | tail -1 | cut -d' ' -f2)"
|
|
||||||
"${ENGINE}" push "${repo}:${MOVING_TAG}" >/dev/null
|
|
||||||
# The AUTHORITATIVE digest is what the registry serves, read back — not what the client
|
|
||||||
# printed. They should agree; if they do not, nothing downstream may trust either.
|
|
||||||
got="$(registry_digest "${img}" "${SHA_TAG}")"
|
|
||||||
[ -n "${got}" ] || die "pushed ${repo}:${SHA_TAG} but the registry does not serve it"
|
|
||||||
[ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] \
|
|
||||||
|| die "${img}: client reported ${pushed} but the registry serves ${got}"
|
|
||||||
[ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \
|
|
||||||
|| die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})"
|
|
||||||
DIGEST[$img]="${got}"
|
|
||||||
SECS[$img]=$(( ${SECS[$img]:-0} + $(date +%s) - t0 ))
|
|
||||||
info "${img}: registry digest ${got}"
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# --- main ---------------------------------------------------------------------------------
|
# --- main ---------------------------------------------------------------------------------
|
||||||
log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}"
|
log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}"
|
||||||
command -v "${ENGINE}" >/dev/null || die "${ENGINE} is not on PATH in the job container"
|
select_images
|
||||||
"${ENGINE}" info >/dev/null 2>&1 \
|
|
||||||
|| die "the ${ENGINE} daemon is not reachable from the job container, so nothing can be built.
|
|
||||||
MEASURED ON THIS RUNNER (PR #81, 2026-10-29): act_runner 0.2.11 never gives a job a
|
|
||||||
daemon. Its DOCKER_HOST is tcp:// (the DinD sidecar), and internal/app/cmd/daemon.go then
|
|
||||||
forces container.docker_host to '-' (\"can't be mounted\"); separately, with
|
|
||||||
valid_volumes: [] gitea/act's sanitizeConfig drops EVERY bind, the socket included.
|
|
||||||
Fixing it means handing workflow code the privileged DinD daemon — an operator security
|
|
||||||
decision, set out in decisions/2026-10-29-image-pipeline.md. Not a fault in this script."
|
|
||||||
"${ENGINE}" version 2>/dev/null | sed -n '1,12p' || true
|
|
||||||
"${ENGINE}" system df || true
|
|
||||||
|
|
||||||
if [ "${MODE}" = publish ]; then
|
if [ "${MODE}" = publish ]; then
|
||||||
: "${REGISTRY_TOKEN:?REGISTRY_TOKEN is required in publish mode}"
|
: "${REGISTRY_TOKEN:?REGISTRY_TOKEN is required in publish mode}"
|
||||||
REGISTRY_USER="${REGISTRY_USER:-ci-image-publisher}"
|
REGISTRY_USER="${REGISTRY_USER:-ci-image-publisher}"
|
||||||
printf '%s' "${REGISTRY_TOKEN}" | "${ENGINE}" login "${REGISTRY}" -u "${REGISTRY_USER}" --password-stdin >/dev/null \
|
# buildx (and through it buildkitd, which does the push) reads registry auth from this file.
|
||||||
|| die "docker login ${REGISTRY} failed — check REGISTRY_PUSH_TOKEN (scope write:package, owner has package write on the org)"
|
jq -n --arg r "${REGISTRY}" --arg a "$(printf '%s:%s' "${REGISTRY_USER}" "${REGISTRY_TOKEN}" | base64 -w0)" \
|
||||||
fi
|
'{auths: {($r): {auth: $a}}}' > "${DOCKER_CONFIG}/config.json"
|
||||||
|
chmod 600 "${DOCKER_CONFIG}/config.json"
|
||||||
select_images
|
registry_token
|
||||||
|
# Publish idempotence: an image already published at THIS commit is not rebuilt. That is what
|
||||||
# Publish idempotence: an image already published at THIS commit is not rebuilt. This is what
|
# makes a re-run after a half-finished publish safe — the immutable tag is never pushed twice.
|
||||||
# makes a re-run after a half-finished publish safe — and it is the only way a rerun can
|
|
||||||
# finish, because the immutable sha tag must never be pushed twice.
|
|
||||||
if [ "${MODE}" = publish ]; then
|
|
||||||
for i in "${ALL_IMAGES[@]}"; do
|
for i in "${ALL_IMAGES[@]}"; do
|
||||||
[ -n "${WANT[$i]:-}" ] || continue
|
[ -n "${WANT[$i]:-}" ] || continue
|
||||||
d="$(registry_digest "${i}" "${SHA_TAG}")"
|
d="$(registry_digest "${i}" "${SHA_TAG}")"
|
||||||
@@ -236,11 +320,11 @@ fi
|
|||||||
# Decide the agent-runner base (see header).
|
# Decide the agent-runner base (see header).
|
||||||
AGENT_BASE=""
|
AGENT_BASE=""
|
||||||
if [ -n "${WANT[agent-runner]:-}" ]; then
|
if [ -n "${WANT[agent-runner]:-}" ]; then
|
||||||
# shellcheck disable=SC1091
|
|
||||||
PINNED_BASE="$(. ./infra/agent/versions.env; echo "${AGENT_BASE_IMAGE}")"
|
PINNED_BASE="$(. ./infra/agent/versions.env; echo "${AGENT_BASE_IMAGE}")"
|
||||||
if [ "${MODE}" = verify ] && [ -z "${WANT[toolbox]:-}" ]; then
|
if [ "${MODE}" = verify ]; then
|
||||||
WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)"
|
AGENT_BASE=link
|
||||||
elif [ "${MODE}" = publish ] && [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then
|
[ -n "${WANT[toolbox]:-}" ] || { WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)"; }
|
||||||
|
elif [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then
|
||||||
case "${PINNED_BASE}" in
|
case "${PINNED_BASE}" in
|
||||||
*@"${ZERO_DIGEST}") WANT[toolbox]=1; NOTE[toolbox]="never published; required as agent-runner's base" ;;
|
*@"${ZERO_DIGEST}") WANT[toolbox]=1; NOTE[toolbox]="never published; required as agent-runner's base" ;;
|
||||||
*@sha256:*) AGENT_BASE="${PINNED_BASE}" ;;
|
*@sha256:*) AGENT_BASE="${PINNED_BASE}" ;;
|
||||||
@@ -249,37 +333,30 @@ if [ -n "${WANT[agent-runner]:-}" ]; then
|
|||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
yn() { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; }
|
|
||||||
for i in "${ALL_IMAGES[@]}"; do info "plan: ${i}: $(yn "${WANT[$i]:-}" BUILD skip) (${NOTE[$i]:-inputs unchanged})"; done
|
for i in "${ALL_IMAGES[@]}"; do info "plan: ${i}: $(yn "${WANT[$i]:-}" BUILD skip) (${NOTE[$i]:-inputs unchanged})"; done
|
||||||
|
if [ -z "${WANT[toolbox]:-}${WANT[agent-runner]:-}${WANT[conformance-runner]:-}" ]; then
|
||||||
if [ -n "${WANT[toolbox]:-}" ]; then
|
info "nothing to build"
|
||||||
build_one toolbox
|
else
|
||||||
if [ "${MODE}" = publish ]; then push_one toolbox; fi
|
setup_builder
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
if [ -n "${WANT[toolbox]:-}" ]; then build_one toolbox; fi
|
||||||
if [ -n "${WANT[agent-runner]:-}" ]; then
|
if [ -n "${WANT[agent-runner]:-}" ]; then
|
||||||
if [ -z "${AGENT_BASE}" ]; then
|
if [ -z "${AGENT_BASE}" ]; then
|
||||||
if [ "${MODE}" = publish ]; then
|
AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}"
|
||||||
AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}"
|
|
||||||
else
|
|
||||||
AGENT_BASE="$(local_ref toolbox)"
|
|
||||||
fi
|
|
||||||
fi
|
fi
|
||||||
info "agent-runner base: ${AGENT_BASE}"
|
info "agent-runner base: $( [ "${AGENT_BASE}" = link ] && echo "the toolbox built in this run (bake target: context)" || echo "${AGENT_BASE}")"
|
||||||
build_one agent-runner "${AGENT_BASE}"
|
build_one agent-runner "${AGENT_BASE}"
|
||||||
if [ "${MODE}" = publish ]; then push_one agent-runner; fi
|
|
||||||
fi
|
|
||||||
if [ -n "${WANT[conformance-runner]:-}" ]; then
|
|
||||||
build_one conformance-runner
|
|
||||||
if [ "${MODE}" = publish ]; then push_one conformance-runner; fi
|
|
||||||
fi
|
fi
|
||||||
|
if [ -n "${WANT[conformance-runner]:-}" ]; then build_one conformance-runner; fi
|
||||||
|
|
||||||
# --- summary ------------------------------------------------------------------------------
|
# --- summary ------------------------------------------------------------------------------
|
||||||
{
|
{
|
||||||
echo "## Images (${MODE})"
|
echo "## Images (${MODE})"
|
||||||
echo
|
echo
|
||||||
echo "commit \`${HEAD_SHA}\`"
|
echo "commit \`${HEAD_SHA}\` — builder: rootless BuildKit \`${BUILDKIT_HOST}\` (mTLS)"
|
||||||
echo
|
echo
|
||||||
echo "| image | built + verified | seconds | published digest (registry-returned) | note |"
|
echo "| image | verify target built | seconds | published digest (registry-returned) | note |"
|
||||||
echo "|---|---|---|---|---|"
|
echo "|---|---|---|---|---|"
|
||||||
for i in "${ALL_IMAGES[@]}"; do
|
for i in "${ALL_IMAGES[@]}"; do
|
||||||
d="${DIGEST[$i]:-}"
|
d="${DIGEST[$i]:-}"
|
||||||
|
|||||||
@@ -0,0 +1,21 @@
|
|||||||
|
---
|
||||||
|
# Flux Kustomization -> ./platform/buildkit — rootless BuildKit, the image builder CI uses
|
||||||
|
# (.gitea/workflows/images.yaml). Its own Kustomization so a stuck Certificate or a failing
|
||||||
|
# builder can never stall any other tier (Flux health-checks Certificates under wait:true).
|
||||||
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||||
|
kind: Kustomization
|
||||||
|
metadata:
|
||||||
|
name: buildkit
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
interval: 10m
|
||||||
|
retryInterval: 2m
|
||||||
|
timeout: 5m
|
||||||
|
prune: true
|
||||||
|
wait: true
|
||||||
|
dependsOn:
|
||||||
|
- name: platform # cert-manager CRDs (Issuer/Certificate) come from its HelmRelease
|
||||||
|
sourceRef:
|
||||||
|
kind: GitRepository
|
||||||
|
name: flux-system
|
||||||
|
path: ./platform/buildkit
|
||||||
@@ -98,4 +98,5 @@ resources:
|
|||||||
# deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon.
|
# deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon.
|
||||||
# See clusters/base/kured.yaml and platform/kured/helmrelease.yaml.
|
# See clusters/base/kured.yaml and platform/kured/helmrelease.yaml.
|
||||||
- kured.yaml # Flux Kustomization -> ./platform/kured (dependsOn platform)
|
- kured.yaml # Flux Kustomization -> ./platform/kured (dependsOn platform)
|
||||||
|
- buildkit.yaml # Flux Kustomization -> ./platform/buildkit (dependsOn platform)
|
||||||
- cluster-vars.yaml # overlay knob surface (base defaults)
|
- cluster-vars.yaml # overlay knob surface (base defaults)
|
||||||
@@ -98,7 +98,10 @@ RUN set -eux; \
|
|||||||
|
|
||||||
# --- runtime -----------------------------------------------------------------
|
# --- runtime -----------------------------------------------------------------
|
||||||
ARG FEDORA_VERSION=44
|
ARG FEDORA_VERSION=44
|
||||||
FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION}
|
# Stage `conformance-runner` is the image that ships; stage `verify` (end of file) is a TEST stage
|
||||||
|
# built FROM it that only runs assertions and is never pushed. build-image.sh and the CI
|
||||||
|
# pipeline build `--target verify` to prove it and `--target conformance-runner` for the image.
|
||||||
|
FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION} AS conformance-runner
|
||||||
|
|
||||||
ARG CHAINSAW_VERSION=
|
ARG CHAINSAW_VERSION=
|
||||||
ARG KUBECTL_VERSION=
|
ARG KUBECTL_VERSION=
|
||||||
@@ -198,3 +201,86 @@ WORKDIR /workspace
|
|||||||
# prepend something the policy cannot see. `bash` is the interactive fallback for a human
|
# prepend something the policy cannot see. `bash` is the interactive fallback for a human
|
||||||
# debugging the image locally.
|
# debugging the image locally.
|
||||||
CMD ["/bin/bash"]
|
CMD ["/bin/bash"]
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# verify — a TEST stage, never shipped. Every assertion conformance/incluster/build-image.sh used
|
||||||
|
# to make with `run --user 65532 --read-only --tmpfs /workspace` now runs here, at build time,
|
||||||
|
# because CI builds on rootless BuildKit where no container runtime exists.
|
||||||
|
#
|
||||||
|
# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE:
|
||||||
|
# * uid 65532 ............ exact (USER inherited from the conformance-runner stage).
|
||||||
|
# * no network ........... exact: RUN --network=none, and the first check proves it.
|
||||||
|
# * --read-only rootfs ... NOT reproducible (a RUN step always has a writable layer).
|
||||||
|
# Approximated by permission: everything uid 65532 can write must lie
|
||||||
|
# under /workspace, /tmp or /var/tmp. In the Job only /workspace (and
|
||||||
|
# /results) are writable — the image points HOME and TMPDIR there — and
|
||||||
|
# readOnlyRootFilesystem is enforced by
|
||||||
|
# platform/kyverno/admission/conformance-trigger-job-shape.yaml.
|
||||||
|
# =============================================================================
|
||||||
|
FROM conformance-runner AS verify
|
||||||
|
RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
|
||||||
|
echo "--- no network (the build-time stand-in for the Job's isolation) ---"
|
||||||
|
if getent hosts gitea.home.keithjackson.dev >/dev/null 2>&1 || timeout 3 bash -c '</dev/tcp/1.1.1.1/443' 2>/dev/null; then
|
||||||
|
echo "UNEXPECTED: network reachable in the verify stage"; exit 1
|
||||||
|
fi
|
||||||
|
echo "confirmed: no network"
|
||||||
|
echo "--- id ---"; id
|
||||||
|
echo "--- HOME ---"; echo "HOME=${HOME}"
|
||||||
|
echo "--- bash ---"; bash --version | head -1
|
||||||
|
echo "--- python3 ---"; python3 --version
|
||||||
|
echo "--- git ---"; git --version
|
||||||
|
echo "--- kubectl ---"; kubectl version --client
|
||||||
|
echo "--- chainsaw ---"; chainsaw version
|
||||||
|
echo "--- baked versions ---"; cat /etc/conformance-runner-versions
|
||||||
|
echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
|
||||||
|
stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
|
||||||
|
if [ -w /usr/local/bin/conformance-clone ]; then
|
||||||
|
echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
|
||||||
|
fi
|
||||||
|
echo "confirmed: not writable by uid $(id -u)"
|
||||||
|
# Each of these MUST fail. Assert the non-zero exit as well as the message: a check that only
|
||||||
|
# greps stderr would pass a build where the guard had stopped rejecting.
|
||||||
|
expect_fail() {
|
||||||
|
local label="$1"; shift
|
||||||
|
local out rc=0
|
||||||
|
out="$(env "$@" 2>&1)" || rc=$?
|
||||||
|
echo "--- ${label} ---"
|
||||||
|
printf "%s\n" "${out}" | head -2
|
||||||
|
if [ "${rc}" -eq 0 ]; then
|
||||||
|
echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
|
||||||
|
fi
|
||||||
|
echo "(rejected, exit ${rc})"
|
||||||
|
}
|
||||||
|
expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
|
||||||
|
conformance-clone --layer l2
|
||||||
|
expect_fail "conformance-clone rejects a missing SHA" \
|
||||||
|
conformance-clone
|
||||||
|
expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
|
||||||
|
CONFORMANCE_GIT_SHA=main conformance-clone
|
||||||
|
expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
|
||||||
|
CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
|
||||||
|
expect_fail "conformance-clone rejects a HOME off the workspace volume" \
|
||||||
|
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
|
||||||
|
HOME=/etc conformance-clone
|
||||||
|
expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
|
||||||
|
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
|
||||||
|
CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
|
||||||
|
expect_fail "conformance-clone refuses a clone target off the workspace volume" \
|
||||||
|
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
|
||||||
|
CONFORMANCE_SRC=/etc/src conformance-clone
|
||||||
|
echo "--- the root filesystem is not writable by the runner (expected: a failure) ---"
|
||||||
|
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi
|
||||||
|
echo "confirmed: / is not writable"
|
||||||
|
echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---"
|
||||||
|
stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \
|
||||||
|
| grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)"
|
||||||
|
if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf ' %s\n' ${stray} | head -20; exit 1; fi
|
||||||
|
echo "confirmed: no writable path outside /workspace, /tmp, /var/tmp"
|
||||||
|
echo "--- HOME is writable (the workspace emptyDir) ---"
|
||||||
|
touch "${HOME}/.probe" || { echo "UNEXPECTED: ${HOME} not writable"; exit 1; }
|
||||||
|
echo "confirmed: ${HOME} is writable"
|
||||||
|
echo "--- uid is non-zero ---"
|
||||||
|
[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; }
|
||||||
|
echo "confirmed: running as uid $(id -u), not root"
|
||||||
|
echo "CONFORMANCE-RUNNER VERIFIED"
|
||||||
|
VERIFY
|
||||||
@@ -61,10 +61,12 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
|
|||||||
> digest-pinned to the conformance-runner image, `command` exactly
|
> digest-pinned to the conformance-runner image, `command` exactly
|
||||||
> `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
|
> `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
|
||||||
> `env`/`envFrom` were the same shape of hole and are closed in the same policy.
|
> `env`/`envFrom` were the same shape of hole and are closed in the same policy.
|
||||||
- `build-image.sh` — builds the above and **verifies the toolchain by running it** inside the
|
- `build-image.sh` — builds the above and **verifies the toolchain by running it**: the
|
||||||
image as uid 65532 with `--read-only`. It **never pushes**, and neither does anyone from a
|
Containerfile's `verify` stage executes every tool and every `conformance-clone` refusal as
|
||||||
workstation: CI (`.gitea/workflows/images.yaml`) runs this same script, and on a push to
|
uid 65532 with no network, at build time. It **never pushes**, and neither does anyone from a
|
||||||
`main` publishes the verified image under `sha-<commit>` and `main`. Renovate then moves
|
workstation: CI (`.gitea/workflows/images.yaml`) builds the same Containerfile and `verify`
|
||||||
|
stage on rootless BuildKit (`platform/buildkit/`), and on a push to `main` publishes the
|
||||||
|
verified image under `sha-<commit>` and `main`. Renovate then moves
|
||||||
`CONFORMANCE_RUNNER_IMAGE_DIGEST` in `conformance/versions.env` onto the registry-returned
|
`CONFORMANCE_RUNNER_IMAGE_DIGEST` in `conformance/versions.env` onto the registry-returned
|
||||||
digest in a pull request (that file is a sensitive path, so the bump needs an
|
digest in a pull request (that file is a sensitive path, so the bump needs an
|
||||||
administrator's `/approve-override`).
|
administrator's `/approve-override`).
|
||||||
|
|||||||
@@ -3,7 +3,8 @@
|
|||||||
# conformance/incluster/build-image.sh — build the pinned `conformance-runner`
|
# conformance/incluster/build-image.sh — build the pinned `conformance-runner`
|
||||||
# image and prove it works. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
|
# image and prove it works. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
|
||||||
#
|
#
|
||||||
# ./build-image.sh # build + verify + print the image id
|
# ./build-image.sh # build the verify target, then the image
|
||||||
|
# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes)
|
||||||
# ./build-image.sh --no-verify # build only (skip the in-image tool proof)
|
# ./build-image.sh --no-verify # build only (skip the in-image tool proof)
|
||||||
# ./build-image.sh --verify-clone # ALSO really clone this repo with conformance-clone.
|
# ./build-image.sh --verify-clone # ALSO really clone this repo with conformance-clone.
|
||||||
# # Opt-in because it needs the Gitea host reachable; the
|
# # Opt-in because it needs the Gitea host reachable; the
|
||||||
@@ -13,13 +14,12 @@
|
|||||||
# IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this)
|
# IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this)
|
||||||
#
|
#
|
||||||
# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
|
# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
|
||||||
# Publishing is done by CI: .gitea/workflows/images.yaml calls this script, and on a push to
|
# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on
|
||||||
# `main` it pushes what this script built AND verified to the Gitea registry, records the
|
# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and
|
||||||
# registry-returned digest in the job summary, and Renovate then opens the pull request that
|
# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest
|
||||||
# moves conformance/versions.env onto that digest. So this script's job is to build and to
|
# in the job summary; Renovate then opens the pull request that moves conformance/versions.env
|
||||||
# prove the toolchain works INSIDE the image as the non-root uid the policy demands — the same
|
# onto that digest. The proof that the image works is the Containerfile's `verify` stage, which
|
||||||
# proof locally and in CI. It runs under ENGINE=docker in CI (Docker-in-Docker), so nothing
|
# this script (podman locally) and CI both build.
|
||||||
# below may be podman-only.
|
|
||||||
#
|
#
|
||||||
# VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
|
# VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
|
||||||
# single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
|
# single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
|
||||||
@@ -39,9 +39,11 @@ TAG_OVERRIDE="${IMAGE_TAG:-}"
|
|||||||
|
|
||||||
VERIFY=1
|
VERIFY=1
|
||||||
VERIFY_CLONE=0
|
VERIFY_CLONE=0
|
||||||
|
PRINT_ARGS=0
|
||||||
for arg in "$@"; do
|
for arg in "$@"; do
|
||||||
case "$arg" in
|
case "$arg" in
|
||||||
--no-verify) VERIFY=0 ;;
|
--no-verify) VERIFY=0 ;;
|
||||||
|
--print-build-args) PRINT_ARGS=1 ;;
|
||||||
--verify-clone) VERIFY_CLONE=1 ;;
|
--verify-clone) VERIFY_CLONE=1 ;;
|
||||||
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
|
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||||
*) echo "unknown argument: $arg" >&2; exit 2 ;;
|
*) echo "unknown argument: $arg" >&2; exit 2 ;;
|
||||||
@@ -52,8 +54,6 @@ log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
|||||||
info() { printf ' %s\n' "$*"; }
|
info() { printf ' %s\n' "$*"; }
|
||||||
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
|
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
|
||||||
|
|
||||||
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
|
|
||||||
|
|
||||||
# --- pins: sourced, never invented -------------------------------------------
|
# --- pins: sourced, never invented -------------------------------------------
|
||||||
TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
|
TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
|
||||||
[ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
|
[ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
|
||||||
@@ -63,6 +63,13 @@ source "${TOOLCHAIN_ENV}"
|
|||||||
: "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
|
: "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
|
||||||
: "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
|
: "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
|
||||||
|
|
||||||
|
# THE build args, in ONE place; `--print-build-args` hands them to the CI pipeline, which builds
|
||||||
|
# this Containerfile on rootless BuildKit.
|
||||||
|
BUILD_ARGS=("CHAINSAW_VERSION=${CHAINSAW_VERSION}" "KUBECTL_VERSION=${KUBECTL_VERSION}" "FEDORA_VERSION=${FEDORA_VERSION}")
|
||||||
|
if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
|
||||||
|
|
||||||
|
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
|
||||||
|
|
||||||
# --- image name: fixed by the admission policy -------------------------------
|
# --- image name: fixed by the admission policy -------------------------------
|
||||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
|
# platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
|
||||||
# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
|
# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
|
||||||
@@ -84,86 +91,27 @@ info " FEDORA ${FEDORA_VERSION}"
|
|||||||
LABEL_ARGS=()
|
LABEL_ARGS=()
|
||||||
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
|
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
|
||||||
|
|
||||||
"${ENGINE}" build "${LABEL_ARGS[@]}" \
|
ARG_FLAGS=()
|
||||||
--file "${HERE}/Containerfile" \
|
for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
|
||||||
--tag "${IMAGE_REF}" \
|
|
||||||
--build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
|
|
||||||
--build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \
|
|
||||||
--build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \
|
|
||||||
"${HERE}"
|
|
||||||
|
|
||||||
# --- verify: the point of the whole exercise ---------------------------------
|
# --- verify: the point of the whole exercise ---------------------------------
|
||||||
# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the
|
# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So the proof is that
|
||||||
# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root
|
# the tools EXECUTE, as the NON-ROOT uid, with no network, that conformance-clone is not writable
|
||||||
# filesystem — the shape the admission policy forces. A pass here is the only evidence
|
# by that uid and refuses every hostile input, and that nothing outside /workspace is writable.
|
||||||
# this image is usable; a build that merely succeeded is not.
|
# That proof is the Containerfile's `verify` stage, built here and in CI alike (CI has no
|
||||||
|
# container runtime — rootless BuildKit only). See the stage's header for what a build step can
|
||||||
|
# and cannot reproduce of the Job's shape. A build that merely succeeded is not evidence.
|
||||||
if [ "${VERIFY}" -eq 1 ]; then
|
if [ "${VERIFY}" -eq 1 ]; then
|
||||||
log "Verifying the toolchain inside the image (non-root, read-only rootfs)"
|
log "Verifying the toolchain (Containerfile target: verify)"
|
||||||
"${ENGINE}" run --rm \
|
"${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}"
|
||||||
--user 65532:65532 \
|
|
||||||
--read-only \
|
|
||||||
--tmpfs /workspace:rw,mode=1777 \
|
|
||||||
--entrypoint "" \
|
|
||||||
"${IMAGE_REF}" \
|
|
||||||
/bin/bash -euo pipefail -c '
|
|
||||||
echo "--- id ---"; id
|
|
||||||
echo "--- HOME ---"; echo "HOME=${HOME}"
|
|
||||||
echo "--- bash ---"; bash --version | head -1
|
|
||||||
echo "--- python3 ---"; python3 --version
|
|
||||||
echo "--- git ---"; git --version
|
|
||||||
echo "--- kubectl ---"; kubectl version --client
|
|
||||||
echo "--- chainsaw ---"; chainsaw version
|
|
||||||
echo "--- baked versions ---"; cat /etc/conformance-runner-versions
|
|
||||||
echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
|
|
||||||
stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
|
|
||||||
if [ -w /usr/local/bin/conformance-clone ]; then
|
|
||||||
echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
|
|
||||||
fi
|
|
||||||
echo "confirmed: not writable by uid $(id -u)"
|
|
||||||
# Each of these MUST fail. Assert the non-zero exit as well as the message: a check
|
|
||||||
# that only greps stderr would pass a build where the guard had stopped rejecting.
|
|
||||||
# (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the
|
|
||||||
# expected failure — without it the verify dies at the first negative case.)
|
|
||||||
expect_fail() {
|
|
||||||
local label="$1"; shift
|
|
||||||
local out rc=0
|
|
||||||
out="$(env "$@" 2>&1)" || rc=$?
|
|
||||||
echo "--- ${label} ---"
|
|
||||||
printf "%s\n" "${out}" | head -2
|
|
||||||
if [ "${rc}" -eq 0 ]; then
|
|
||||||
echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
|
|
||||||
fi
|
|
||||||
echo "(rejected, exit ${rc})"
|
|
||||||
}
|
|
||||||
expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
|
|
||||||
conformance-clone --layer l2
|
|
||||||
expect_fail "conformance-clone rejects a missing SHA" \
|
|
||||||
conformance-clone
|
|
||||||
expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
|
|
||||||
CONFORMANCE_GIT_SHA=main conformance-clone
|
|
||||||
expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
|
|
||||||
CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
|
|
||||||
expect_fail "conformance-clone rejects a HOME off the workspace volume" \
|
|
||||||
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
|
|
||||||
HOME=/etc conformance-clone
|
|
||||||
expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
|
|
||||||
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
|
|
||||||
CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
|
|
||||||
expect_fail "conformance-clone refuses a clone target off the workspace volume" \
|
|
||||||
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
|
|
||||||
CONFORMANCE_SRC=/etc/src conformance-clone
|
|
||||||
echo "--- rootfs is read-only (expected: a failure) ---"
|
|
||||||
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
|
|
||||||
echo "confirmed: / is read-only"
|
|
||||||
echo "--- HOME is writable (the workspace emptyDir) ---"
|
|
||||||
touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable"
|
|
||||||
echo "--- uid is non-zero ---"
|
|
||||||
[ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root"
|
|
||||||
'
|
|
||||||
log "Toolchain verified"
|
log "Toolchain verified"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target conformance-runner \
|
||||||
|
--file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}"
|
||||||
|
|
||||||
# --- verify the clone helper against the REAL repository (opt-in: needs network) ----
|
# --- verify the clone helper against the REAL repository (opt-in: needs network) ----
|
||||||
|
# LOCAL ONLY: this one `run`s the image, which needs a container engine, so CI does not do it.
|
||||||
# The negative cases above prove conformance-clone refuses bad input. This proves the
|
# The negative cases above prove conformance-clone refuses bad input. This proves the
|
||||||
# positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
|
# positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
|
||||||
# rev-parse assertion is only an unexercised line of shell.
|
# rev-parse assertion is only an unexercised line of shell.
|
||||||
|
|||||||
@@ -250,8 +250,8 @@ check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text,
|
|||||||
# The build recipe moved from dev.sh into build-image.sh on 2026-10-29, so that `dev.sh build`
|
# The build recipe moved from dev.sh into build-image.sh on 2026-10-29, so that `dev.sh build`
|
||||||
# and the CI image pipeline (.gitea/workflows/images.yaml) share ONE recipe. Check the recipe.
|
# and the CI image pipeline (.gitea/workflows/images.yaml) share ONE recipe. Check the recipe.
|
||||||
BUILDSH = ROOT / "infra/devcontainer/build-image.sh"
|
BUILDSH = ROOT / "infra/devcontainer/build-image.sh"
|
||||||
check(BUILDSH.is_file() and "--build-arg PROMTOOL_VERSION=" in BUILDSH.read_text(),
|
check(BUILDSH.is_file() and '"PROMTOOL_VERSION=$PROMTOOL_VERSION"' in BUILDSH.read_text(),
|
||||||
"infra/devcontainer/build-image.sh does not pass --build-arg PROMTOOL_VERSION — the ARG "
|
"infra/devcontainer/build-image.sh does not pass PROMTOOL_VERSION in its BUILD_ARGS — the ARG "
|
||||||
"would be empty and the build would fetch a bogus URL")
|
"would be empty and the build would fetch a bogus URL")
|
||||||
|
|
||||||
INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh"
|
INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh"
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
---
|
---
|
||||||
date: 2026-10-29
|
date: 2026-10-29
|
||||||
branch: feat/image-pipeline
|
branch: feat/image-pipeline
|
||||||
title: "Images are built, verified and published by CI; Renovate carries the digests into git"
|
title: "Images are built, verified and published by CI on rootless BuildKit; Renovate carries the digests into git"
|
||||||
---
|
---
|
||||||
|
|
||||||
## What changed
|
## What changed
|
||||||
@@ -11,127 +11,177 @@ the codebase, never to a one-off workstation build/push. Before this branch the
|
|||||||
package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and
|
package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and
|
||||||
both `build-image.sh` scripts said "publishing is an operator action".
|
both `build-image.sh` scripts said "publishing is an operator action".
|
||||||
|
|
||||||
- **`.gitea/workflows/images.yaml`** + **`.gitea/workflows/scripts/images.sh`** (the logic).
|
### The builder: rootless BuildKit — DECISION MADE (operator, on PR #81)
|
||||||
- Pull request: build and verify only. No push, and the step references no secret at all.
|
|
||||||
- Push to `main`, or `workflow_dispatch` on `main`: build, verify, then push to
|
|
||||||
`gitea.home.keithjackson.dev/home-network/{toolbox,agent-runner,conformance-runner}`.
|
|
||||||
Tags are `sha-<12 hex>` (immutable: the driver refuses to overwrite one) and `main` (moving).
|
|
||||||
The digest is read back from the registry with a HEAD on the manifest, compared with what the
|
|
||||||
client printed, and written to the job summary.
|
|
||||||
- Only images whose inputs changed are built. A re-run on the same commit skips images that are
|
|
||||||
already published.
|
|
||||||
- agent-runner builds FROM the toolbox built in the same run (on main, FROM its just-pushed
|
|
||||||
digest). Otherwise:
|
|
||||||
- in verify mode the toolbox is built anyway as its base, because a PR holds no credential;
|
|
||||||
- in publish mode it uses `AGENT_BASE_IMAGE` from `infra/agent/versions.env`;
|
|
||||||
- if that is still the zero placeholder, the toolbox is built and published first. This is
|
|
||||||
how the first run on main bootstraps.
|
|
||||||
- An always-run step prunes the DinD image store.
|
|
||||||
- **One build recipe per image, reused by CI.**
|
|
||||||
- New `infra/devcontainer/build-image.sh`. `dev.sh build` now delegates to it. It has an
|
|
||||||
in-image verification step that the toolbox never had: every tool is on PATH, and
|
|
||||||
promtool, kubectl, chainsaw, kyverno and sops report their pins.
|
|
||||||
- `infra/agent/build-image.sh` and `conformance/incluster/build-image.sh` were made
|
|
||||||
engine-neutral, because CI runs `ENGINE=docker` against DinD:
|
|
||||||
- `podman image exists` became `image inspect`, with a pull fallback;
|
|
||||||
- `{{.Digest}}` and `--digestfile` are gone. No local digest is printed, since only the
|
|
||||||
registry's digest is meaningful;
|
|
||||||
- `probe-validation.sh` is fed over **stdin** instead of a bind mount. Under DinD a bind
|
|
||||||
source is resolved on the daemon's filesystem, which would silently mount an empty
|
|
||||||
directory;
|
|
||||||
- `probe-validation.sh` itself gained `${BASH_SOURCE[0]:-$0}` so it survives `bash -s`
|
|
||||||
under `set -u`.
|
|
||||||
- `AGENT_BASE_IMAGE` and `IMAGE_TAG` can be overridden from the environment, and
|
|
||||||
`BUILD_REVISION` adds an `org.opencontainers.image.revision` label.
|
|
||||||
- The "publishing is an operator action" text was replaced throughout.
|
|
||||||
- **Digest references.** Each now carries a `# renovate-digest: image=<repo> tag=main` marker
|
|
||||||
on the line above it:
|
|
||||||
- `platform/pm-agent/deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`), which share
|
|
||||||
one dependency, so they bump in one PR;
|
|
||||||
- `infra/agent/versions.env` `AGENT_BASE_IMAGE`, which was `home-k3s-dev:local` and is now
|
|
||||||
the published toolbox by digest (zero placeholder until the first publish);
|
|
||||||
- `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to the
|
|
||||||
zero placeholder because Renovate can only replace a digest it can match.
|
|
||||||
|
|
||||||
The admission policies are **untouched**; the `<repo>@sha256:<digest>` form is preserved.
|
**The problem it answers, measured on this PR's first CI run (images run 325, failed in 6 s):** a
|
||||||
|
CI job on this runner gets **no Docker daemon at all**, for two reasons, both in the deployed
|
||||||
|
versions' source:
|
||||||
|
- act_runner v0.2.11 `internal/app/cmd/daemon.go` sets `container.docker_host` to `-` ("never
|
||||||
|
mount into jobs") whenever `DOCKER_HOST` is `tcp://`, which it is here (the DinD sidecar);
|
||||||
|
- gitea/act v0.261.2 `pkg/container/docker_run.go` `sanitizeConfig` drops **every** bind,
|
||||||
|
socket included, when `valid_volumes` is empty, which it is (the runner has no config file).
|
||||||
|
|
||||||
|
Giving jobs the DinD daemon would have made every branch push root on the node, because that
|
||||||
|
daemon is `privileged: true`.
|
||||||
|
|
||||||
|
**The decision:** no job may reach a privileged daemon, and images are built by a **rootless
|
||||||
|
BuildKit** daemon over mTLS. The runner (`apps/wave-1/gitea-actions.yaml`, DOCKER_HOST,
|
||||||
|
valid_volumes) is **untouched**.
|
||||||
|
|
||||||
|
- **`platform/buildkit/`** — a new hand-managed component in its own namespace `buildkit`. The
|
||||||
|
runner manifest is renderer-generated (`catalog/render.py` EXTRA_UNITS), and the builder has no
|
||||||
|
business sitting next to a privileged container. Flux reconciles it as the new Kustomization
|
||||||
|
`buildkit` (`clusters/base/buildkit.yaml`, `dependsOn: platform`).
|
||||||
|
- **Deployment:** `moby/buildkit:v0.34.1-rootless@sha256:f68752a5…0c4f89`, pinned by index
|
||||||
|
digest. The securityContext, every line measured (see *What I verified*):
|
||||||
|
- `privileged: false`;
|
||||||
|
- uid/gid 1000, `runAsNonRoot`;
|
||||||
|
- `capabilities: drop ALL, add SETUID, SETGID`;
|
||||||
|
- `allowPrivilegeEscalation: true`;
|
||||||
|
- `seccompProfile: Unconfined` and `appArmorProfile: Unconfined`;
|
||||||
|
- `readOnlyRootFilesystem: true`, with five emptyDirs.
|
||||||
|
|
||||||
|
Other settings:
|
||||||
|
- `--oci-worker-no-process-sandbox`;
|
||||||
|
- no ServiceAccount token;
|
||||||
|
- memory limit 4Gi, which covers build steps too;
|
||||||
|
- cache emptyDir with `sizeLimit: 6Gi`, and GC capped at 4 GB in `buildkitd.toml`
|
||||||
|
(ConfigMap).
|
||||||
|
- **mTLS via cert-manager** (`certificates.yaml`): a self-signed CA used only here, the server
|
||||||
|
cert `buildkitd-tls`, and the client cert `buildkit-client-ci`. buildkitd verifies every
|
||||||
|
client against that CA.
|
||||||
|
- **Service:** ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`.
|
||||||
|
- **NetworkPolicies:**
|
||||||
|
- default-deny;
|
||||||
|
- ingress on 1234 **only from the act-runner pod** (`gitea-actions`,
|
||||||
|
`app.kubernetes.io/name=act-runner`);
|
||||||
|
- egress: DNS, world :443/:80 for downloads, and the `ingress` entity (the Gateway) for the
|
||||||
|
registry push.
|
||||||
|
- `versions.env`: the CI-side client, `buildx` v0.38.1 with the sha256 from its
|
||||||
|
`checksums.txt`, plus the address and certificate name CI dials.
|
||||||
|
- **How DinD job containers appear to Cilium (measured, read-only).**
|
||||||
|
`iptables -t nat -S POSTROUTING` in the dind container shows
|
||||||
|
`-A POSTROUTING -s 172.18.0.0/16 ! -o br-… -j MASQUERADE` (and the same for docker0). So a job's
|
||||||
|
connection leaves with the **act-runner pod's IP, and therefore that pod's Cilium identity**.
|
||||||
|
That makes "only the act-runner pod" expressible and correct, but it cannot tell one job from
|
||||||
|
another, or a job from act_runner itself. Per-job authentication is the client certificate.
|
||||||
|
- **Admission: no Kyverno exception needed.**
|
||||||
|
- Every ClusterPolicy actually applied (`platform-config/kustomization.yaml`; live:
|
||||||
|
`conformance-trigger-job-shape`, `label-cdi-worker-pods`, `pm-agent-job-shape`,
|
||||||
|
`require-auth-mode-label`, `require-default-deny-baseline`, `require-egress-class-label`,
|
||||||
|
`restrict-host-namespaces`, `secret-provenance`, `validate-default-deny-shape`) matches only
|
||||||
|
namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`, or only its own
|
||||||
|
namespace in the job-shape policies' case. `buildkit` carries no such label.
|
||||||
|
- There are no namespaced Kyverno Policies and no cluster-wide PSA default (both measured).
|
||||||
|
- The namespace is deliberately left without a PSA label, because PSS `baseline` forbids
|
||||||
|
`Unconfined` seccomp and AppArmor.
|
||||||
|
- `validate-default-deny-shape` does apply to the namespace's default-deny NetworkPolicy, and
|
||||||
|
its shape (both directions) satisfies it.
|
||||||
|
- **Unprivileged user namespaces on the nodes (measured, read-only, via Ansible `sysctl`).** On
|
||||||
|
all three Debian 13.7 nodes, kernel `7.1.8+deb13-amd64`:
|
||||||
|
- `kernel.unprivileged_userns_clone = 1`;
|
||||||
|
- `user.max_user_namespaces = 127206`;
|
||||||
|
- no `kernel.apparmor_restrict_unprivileged_userns` (an Ubuntu-only key);
|
||||||
|
- AppArmor enabled (`Y`).
|
||||||
|
|
||||||
|
So this is a satisfied prerequisite, not an open one.
|
||||||
|
- **`infra/gitea/buildkit-client-secret.sh`** (operator action): copies `ca.crt`, `tls.crt` and
|
||||||
|
`tls.key` from the Secret `buildkit/buildkit-client-ci` into the Actions secrets `BUILDKIT_CA`,
|
||||||
|
`BUILDKIT_CLIENT_CERT` and `BUILDKIT_CLIENT_KEY`. Before writing anything it checks that:
|
||||||
|
- the cert verifies against the CA;
|
||||||
|
- the cert is a client-auth cert;
|
||||||
|
- the key matches the cert.
|
||||||
|
|
||||||
|
It never prints a value. `--check` reports whether the stored copy is older than the current
|
||||||
|
certificate.
|
||||||
|
|
||||||
|
### Verification moved into the Containerfiles
|
||||||
|
|
||||||
|
`docker run` cannot exist on BuildKit, so each Containerfile now names its shipped stage after the
|
||||||
|
image (`toolbox`, `agent-runner`, `conformance-runner`) and ends in a **`verify` test stage**,
|
||||||
|
`FROM <image> AS verify`. That stage runs every previous assertion at build time, as the image's
|
||||||
|
non-root uid (65532 for agent-runner and conformance-runner), in `RUN --network=none`. It is never
|
||||||
|
pushed.
|
||||||
|
- **Every existing assertion is preserved**, and the toolbox gained one it never had: every tool
|
||||||
|
on PATH, and five tools report their pinned versions.
|
||||||
|
- `probe-validation.sh` is `COPY`ed into the verify stage only, so it never ships.
|
||||||
|
- **Bug fixed on the way:** in the old `docker run` block, `cmd | grep -q … && echo ok` and
|
||||||
|
`[ … ] && echo` could never fail the script under `set -e` + `pipefail`. The "entrypoints refuse
|
||||||
|
arguments", "not root", "no API key baked in" and "HOME writable" checks asserted nothing. They
|
||||||
|
are real assertions now.
|
||||||
|
- **What a build step cannot reproduce, and how it is approximated** (also written in each stage's
|
||||||
|
header):
|
||||||
|
- **uid:** exact.
|
||||||
|
- **`--network none`:** exact, and each stage *proves* it. Its first check fails if DNS or a
|
||||||
|
TCP connection works. A negative control without `--network=none` fails that check.
|
||||||
|
- **`--read-only` rootfs:** not reproducible, because a RUN step always has a writable layer.
|
||||||
|
It is approximated by permission:
|
||||||
|
- `touch /should-fail` must fail;
|
||||||
|
- a `find / -xdev -writable ! -type l` inventory, run as the uid, must lie entirely under
|
||||||
|
`/workspace`, `/tmp` or `/var/tmp`. `! -type l` was needed: Fedora's `/usr/tmp` symlink
|
||||||
|
to `/var/tmp` tripped it.
|
||||||
|
|
||||||
|
The real control remains the pods' `readOnlyRootFilesystem`, which the admission policies
|
||||||
|
enforce.
|
||||||
|
- **`--tmpfs /workspace`:** approximated by the image's own `/workspace`, owned by 65532, which
|
||||||
|
is exactly what the pod's emptyDir replaces.
|
||||||
|
|
||||||
|
The opt-in `--verify-clone` still `run`s the image, so it remains local-only.
|
||||||
|
- **`build-image.sh` (all three):**
|
||||||
|
- verify is `build --target verify` and the image is `build --target <image>`;
|
||||||
|
- a new `--print-build-args` prints the pins as `KEY=VALUE`, which is what CI consumes. The
|
||||||
|
arg list therefore exists once;
|
||||||
|
- the local podman path still works.
|
||||||
|
- The L0 check `alert-rules-unit-tested` now looks for `"PROMTOOL_VERSION=$PROMTOOL_VERSION"` in
|
||||||
|
the toolbox's build-arg list.
|
||||||
|
|
||||||
|
### The pipeline (`.gitea/workflows/images.yaml` + `scripts/images.sh`)
|
||||||
|
|
||||||
|
- **The builder.** The driver downloads the pinned `buildx` and refuses a checksum mismatch. It
|
||||||
|
writes the client cert from the `BUILDKIT_*` secrets to a 0600 temp dir that is removed on exit,
|
||||||
|
creates a `remote` builder, and runs `buildx inspect --bootstrap`.
|
||||||
|
- **It fails loudly, never skips.** It stops with a clear message if:
|
||||||
|
- the secrets are absent;
|
||||||
|
- buildkitd is unreachable. The message lists the causes in order: not deployed yet (Flux
|
||||||
|
applies it only after merge), stale certificate, NetworkPolicy.
|
||||||
|
- **Pull request:** `buildx bake <image>-verify` for changed images. Nothing is pushed and no
|
||||||
|
registry credential is referenced. The `BUILDKIT_*` builder secrets *are* given to PR jobs,
|
||||||
|
because they must build.
|
||||||
|
- agent-runner is built FROM the toolbox built **in the same run** through a bake `target:`
|
||||||
|
context. That toolbox exists in no registry, so the toolbox is built as its base even when
|
||||||
|
its own inputs did not change.
|
||||||
|
- **Push to main / dispatch on main:** for each image, the verify target first, and only then the
|
||||||
|
image itself with `push=true`. Tags are `sha-<12 hex>` (immutable) and `main` (moving).
|
||||||
|
- The digest is read back from the registry and compared with the client's metadata.
|
||||||
|
- agent-runner builds FROM the toolbox digest just pushed, or else from the pinned
|
||||||
|
`AGENT_BASE_IMAGE` (the zero placeholder bootstraps).
|
||||||
|
- `BUILDX_NO_DEFAULT_ATTESTATIONS=1` keeps one manifest per tag, which is what admission,
|
||||||
|
Renovate and the retention script assume.
|
||||||
|
- **Unchanged from the first iteration:** change detection, publish idempotence,
|
||||||
|
`REGISTRY_PUSH_TOKEN` and its preflight, the Renovate markers, and `registry-retention.sh`. The
|
||||||
|
DinD prune step is gone, because nothing is built in DinD now.
|
||||||
|
|
||||||
|
### Unchanged from the first iteration (still true)
|
||||||
|
|
||||||
|
- **Digest references** carry `# renovate-digest: image=<repo> tag=main` markers:
|
||||||
|
- `platform/pm-agent/deployment.yaml` and `configmap.yaml`;
|
||||||
|
- `infra/agent/versions.env` `AGENT_BASE_IMAGE`, now the published toolbox by digest, zero
|
||||||
|
until the first publish;
|
||||||
|
- `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to zero.
|
||||||
|
|
||||||
|
The admission policies are untouched.
|
||||||
- **`renovate.json5`**:
|
- **`renovate.json5`**:
|
||||||
- a new `custom.regex` manager for the markers;
|
- the `custom.regex` marker manager;
|
||||||
- `enabledManagers` `regex` → `custom.regex`;
|
- `enabledManagers` `regex` → `custom.regex`;
|
||||||
- a packageRule that disables the `kubernetes` manager for these three repos. Otherwise it
|
- the `kubernetes` manager disabled for these three repos;
|
||||||
would also see the digest-only lines and look them up as `latest`;
|
- `ignorePaths` narrowed so `conformance/versions.env` is visible.
|
||||||
- an `own-image` label;
|
|
||||||
- `ignorePaths` narrowed from `conformance/**` to its subdirectories. `ignorePaths` applies
|
|
||||||
to custom managers too, so the blanket glob hid `conformance/versions.env`.
|
|
||||||
- **`renovate.yaml`**: `renovate-version: 44.167.0` pinned, plus a `RENOVATE_HOST_RULES` docker
|
|
||||||
rule for the Gitea registry, using the same `RENOVATE_TOKEN`, which now also needs
|
|
||||||
`read:package`. See below for why the version pin is a real fix and not tidiness.
|
|
||||||
- **`infra/gitea/registry-retention.sh`**: dry run by default, `--apply` to delete. This is an
|
|
||||||
operator action. Per image it keeps:
|
|
||||||
- `main` and `latest`;
|
|
||||||
- the 10 newest `sha-` tags;
|
|
||||||
- every tag whose digest is pinned anywhere in `origin/main`;
|
|
||||||
- every digest-named version.
|
|
||||||
- **`infra/gitea/conformance-trigger-kubeconfig.sh`**: the "runner image is digest-pinned"
|
|
||||||
precondition grepped the literal line `CONFORMANCE_RUNNER_IMAGE="${…_REPO}@${…_DIGEST}"`
|
|
||||||
for 64 hex digits, so it **could never pass**. It now sources the file, evaluates the value,
|
|
||||||
and rejects the zero placeholder.
|
|
||||||
- **`conformance/tests/l0-static/alert-rules-unit-tested/check.sh`**: its "the toolbox build
|
|
||||||
passes `--build-arg PROMTOOL_VERSION`" assertion now reads `infra/devcontainer/build-image.sh`
|
|
||||||
instead of `dev.sh`. The gate caught this on the first run, because the recipe had moved.
|
|
||||||
- **Docs**: `platform/pm-agent/README.md`, `runbooks/pm-agent.md` (two rows),
|
|
||||||
`docs/plans/pm-agent-issue-to-pr.md` (follow-up 1), `conformance/incluster/README.md`,
|
|
||||||
`infra/devcontainer/README.md`, and a new "How images are built" section in
|
|
||||||
`.gitea/workflows/README.md`.
|
|
||||||
|
|
||||||
### BLOCKER — the runner cannot build images, and fixing that is a security decision
|
`renovate.yaml` pins `renovate-version: 44.167.0` and adds a docker `RENOVATE_HOST_RULES` entry
|
||||||
|
using `RENOVATE_TOKEN`, which now needs `read:package`.
|
||||||
**Measured on this PR's own CI run (images run 325): the job failed in 6 s** with
|
- **`infra/gitea/registry-retention.sh`** (dry run by default; operator action); the
|
||||||
`failed to connect to the docker API at unix:///var/run/docker.sock ... no such file`. The job
|
`conformance-trigger-kubeconfig.sh` precondition fix; and the docs (pm-agent README, runbook,
|
||||||
container is given **no Docker daemon at all**, for two independent reasons, both read from the
|
plan, conformance README, devcontainer README, workflows README). The docs are updated again
|
||||||
exact versions deployed:
|
for BuildKit, and there is a new `platform/buildkit/README.md`.
|
||||||
|
|
||||||
- act_runner **v0.2.11** `internal/app/cmd/daemon.go`: when `DOCKER_HOST` has a scheme other than
|
|
||||||
`unix`/`npipe` — here `tcp://localhost:2376`, the DinD sidecar — it sets
|
|
||||||
`cfg.Container.DockerHost = "-"`, which means "never mount into job containers".
|
|
||||||
- gitea/act **v0.261.2** (act_runner's `go.mod` replace) `pkg/container/docker_run.go`
|
|
||||||
`sanitizeConfig`: with `valid_volumes` empty (the runner has no config file, so the default
|
|
||||||
`[]`), `hostConfig.Binds = []string{}`. **Every** bind is dropped, the daemon socket included,
|
|
||||||
and so is any `volumes:` a workflow asks for.
|
|
||||||
|
|
||||||
I assumed the opposite while designing (from act's `getDockerDaemonSocketMountPath`, which never
|
|
||||||
gets the chance to run). That was wrong, and the PR run is what showed it.
|
|
||||||
|
|
||||||
**Why I did not fix it in this PR.** Any fix gives workflow code the DinD daemon. That daemon is
|
|
||||||
`privileged: true`, so `docker run --privileged -v /:/host` from a job is **root on the node**.
|
|
||||||
Today a malicious workflow — for example one pushed by a prompt-injected pm-agent worker, which
|
|
||||||
holds a repo write token — runs in an unprivileged container and holds no cluster credential.
|
|
||||||
After the fix it does not. That is a change to the security boundary, and it is the operator's
|
|
||||||
call. It also cannot be tested from a PR anyway: the runner only changes after merge + Flux.
|
|
||||||
|
|
||||||
**Options, for the operator:**
|
|
||||||
|
|
||||||
1. **Socket to every job (smallest change):** in `apps/wave-1/gitea-actions.yaml`,
|
|
||||||
- mount a shared `emptyDir` at `/var/run` in both the `dind` sidecar and the `runner`, so
|
|
||||||
dockerd's own `unix:///var/run/docker.sock` is visible at the same path in both;
|
|
||||||
- set the runner's `DOCKER_HOST=unix:///var/run/docker.sock`, and drop `DOCKER_TLS_VERIFY`
|
|
||||||
and `DOCKER_CERT_PATH`, because TLS over a unix socket breaks the client;
|
|
||||||
- add an act_runner config file (`CONFIG_FILE`) with `container.valid_volumes:
|
|
||||||
["/var/run/docker.sock"]` and `runner.file: /data/.runner`, so the existing registration
|
|
||||||
is kept.
|
|
||||||
|
|
||||||
Consequence: any workflow on any branch can reach node root.
|
|
||||||
2. **Option 1 on a SECOND runner** with its own label (e.g. `image-builder`) and its own DinD,
|
|
||||||
with `images.yaml` `runs-on` that label. The gate runner stays socket-less, and image builds
|
|
||||||
stop competing with the gate for the single slot and its 7 GB node disk. **It is not a
|
|
||||||
security boundary**: Gitea cannot restrict which workflow may target a label, so any branch
|
|
||||||
could still `runs-on: image-builder`.
|
|
||||||
3. **A builder that is not a privileged daemon**, e.g. a rootless BuildKit sidecar exposed to
|
|
||||||
jobs. That bounds a job to BuildKit's rootless sandbox instead of node root. It needs the
|
|
||||||
verification steps rewritten, because they *run* the image (as uid 65532, `--read-only`),
|
|
||||||
which BuildKit alone does not do. This is a larger piece of work.
|
|
||||||
|
|
||||||
Nothing in this PR has to change for options 1 or 2 except `runs-on` for option 2. The driver
|
|
||||||
already produced verified images with a stock docker 27 CLI (see *What I verified*).
|
|
||||||
|
|
||||||
### Decisions and pushback
|
### Decisions and pushback
|
||||||
|
|
||||||
@@ -147,7 +197,8 @@ already produced verified images with a stock docker 27 CLI (see *What I verifie
|
|||||||
- `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3.
|
- `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3.
|
||||||
|
|
||||||
The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight
|
The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight
|
||||||
pattern.
|
pattern. With BuildKit the credential reaches the daemon through the buildx session (a
|
||||||
|
0600 `config.json` in the job's temp dir), never through the cluster.
|
||||||
- **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate:
|
- **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate:
|
||||||
- `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` →
|
- `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` →
|
||||||
`custom.regex`;
|
`custom.regex`;
|
||||||
@@ -195,89 +246,112 @@ already produced verified images with a stock docker 27 CLI (see *What I verifie
|
|||||||
|
|
||||||
## What I verified
|
## What I verified
|
||||||
|
|
||||||
- **Local gate, pinned toolbox, from this worktree, committed HEAD.** Command:
|
**Proved locally, against the exact pinned images:**
|
||||||
`ENGINE=<shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`.
|
|
||||||
The shim adds the `.git` mount. Result: see the PR body for the rc and stage list. They are
|
- **Rootless buildkitd, configured as the manifest configures it.**
|
||||||
recorded there rather than here because this entry is part of the tree being gated.
|
- Setup: the pinned `moby/buildkit:v0.34.1-rootless` digest, under rootless podman, with
|
||||||
- **Local podman builds through the CI driver**, run as
|
`--cap-drop ALL --cap-add SETUID --cap-add SETGID`, seccomp unconfined and `--read-only`;
|
||||||
`MODE=verify ENGINE=podman BASE_SHA=$(git merge-base origin/main HEAD) .gitea/workflows/scripts/images.sh`.
|
tmpfs at the same five paths; the ConfigMap's `buildkitd.toml`; mTLS on :1234.
|
||||||
It returned rc 0 and built and verified all three images:
|
- The image's `/etc/subuid` was overridden with `user:1:999` and `user:1001:64535`, because
|
||||||
- toolbox: every tool on PATH, and the five pins matched;
|
rootless podman maps only 65,537 uids. rootlesskit's real `newuidmap` user-namespace path
|
||||||
- agent-runner: `PM_SELFTEST=1 pm-dispatch` OK, and the stdin-fed `probe-validation.sh`
|
still ran.
|
||||||
gave "20 refusal(s), 0 failure(s)";
|
- It started, `buildctl debug workers` listed the OCI worker, and `debug workers -v` showed the
|
||||||
- conformance-runner: all seven `conformance-clone` refusals.
|
GC config applied (max 4.29 GB, reserved 1.07 GB).
|
||||||
- **The docker-CLI code path**, i.e. what CI runs. I ran the same driver inside
|
- **The securityContext, item by item.** Each requirement was measured by taking it away:
|
||||||
`docker:27-cli` (`ENGINE=docker`, `DOCKER_BUILDKIT=0`) against podman's Docker-compatible API
|
- **default seccomp:** daemon starts, every RUN step fails → `Unconfined` is required;
|
||||||
socket. Exit code 0; all three images built and verified (selftest OK; probe OK, 20 refusals).
|
- **no `SETUID`/`SETGID`:** rootlesskit dies with
|
||||||
This covers the docker-specific calls (`image inspect` formats, `run -i` with stdin,
|
`newuidmap … operation not permitted` → both capabilities are required;
|
||||||
`--tmpfs`, `--read-only`, `--user`). It does **not** cover BuildKit or a real DinD daemon.
|
- **`--read-only` rootfs:** works once the five emptyDirs exist. Its only failure was the
|
||||||
- **This PR's CI, run 325 (`images`): FAILED in 6 s**, because there is no Docker daemon in the
|
probe client's `~/.docker`, which is now an emptyDir;
|
||||||
job container (see BLOCKER). Gate run 324 failed only on `approval-policy` (expected,
|
- **minimal caps + Unconfined seccomp:** works, and a RUN step executes.
|
||||||
sensitive paths). Every other stage passed there, including `gitea-settings`.
|
- **The full CI path, in the actual CI job image.** Setup: `ghcr.io/catthehacker/ubuntu:act-22.04`
|
||||||
- **Other driver paths, run locally:**
|
running `.gitea/workflows/scripts/images.sh` in verify mode. It downloaded and sha256-checked
|
||||||
- agent-runner alone builds the toolbox as its base;
|
buildx v0.38.1, connected over mTLS, and bake-built all three `verify` targets from an empty
|
||||||
- an empty diff builds nothing;
|
cache. rc 0, 1m32s total:
|
||||||
- publish mode without a token dies before doing anything;
|
- `TOOLBOX VERIFIED` (42 s);
|
||||||
- `infra/agent/build-image.sh` with the zero placeholder dies with build-locally
|
- `AGENT-RUNNER VERIFIED` (26 s): selftest OK, probe "20 refusal(s), 0 failure(s)", and built
|
||||||
instructions.
|
FROM the in-run toolbox through the `target:` context, with the log showing
|
||||||
- **Renovate, with the pinned 44.167.0 image, locally:**
|
`[agent-runner-verify toolbox 1/6]` and `base=toolbox-in-this-run`;
|
||||||
- `renovate-config-validator` exits 0;
|
- `CONFORMANCE-RUNNER VERIFIED` (21 s): all seven refusals;
|
||||||
- a `--platform=local` `extract` dry run finds all four marker references (custom regex manager,
|
- every stage printed `confirmed: no network`.
|
||||||
`currentValue: main`, `currentDigest: sha256:000…`);
|
- **mTLS rejects what it should.** Tested with a real BuildKit API call (`buildctl debug workers`):
|
||||||
- a `lookup` dry run shows the `kubernetes`-manager duplicates as `skipReason: disabled` and the
|
- no client cert → `remote error: tls: certificate required`;
|
||||||
marker deps taking the "unsupported/unversioned value main → getDigest(main)" path. That
|
- a client cert from another CA → rejected the same way;
|
||||||
lookup currently 404s because nothing is published yet.
|
- the valid cert → accepted.
|
||||||
- **Renovate 39.264.0 rejects main's current `renovate.json5`.** Validator output reproduced
|
|
||||||
above.
|
Through the driver:
|
||||||
- **`registry-retention.sh` dry run against the live Gitea (read-only):** zero versions per image,
|
- another CA's client cert → rc 1;
|
||||||
nothing to delete. The token exchange and the package listing both answered.
|
- the wrong CA for the server → rc 1;
|
||||||
- **Live, read-only measurements** (kubectl and the Gitea API):
|
- an unreachable host → rc 1 in about 20 s, with the explanatory message;
|
||||||
- the runner pod is on k3s-prod-1, which had **7 GB** nodefs available;
|
- missing secrets → rc 1, naming the three secrets.
|
||||||
- DinD holds only the 1.54 GB act image; buildx is v0.20.1; the daemon is 27.5.1 on overlay2
|
- **The local podman path**, `build-image.sh` for all three (rc 0).
|
||||||
with cgroup v2;
|
- Negative control: a verify stage expecting a different promtool version printed
|
||||||
- Gitea `/data` is 9.7G, 1% used;
|
`DRIFT promtool want 9.9.9` and failed the build.
|
||||||
- the Gateway's Envoy (Cilium) has `streamIdleTimeout: 300s`, `maxStreamDuration: 0s`, and
|
- Negative control: the `--network=none` check fails when the flag is removed.
|
||||||
**no route `timeout`** on the gitea route;
|
- **`buildkit-client-secret.sh`'s sanity checks**, run offline on local certs. They tell apart:
|
||||||
- Gitea's `LIMIT_SIZE_CONTAINER` is unset (default −1 = unlimited, `modules/setting/packages.go`).
|
- a good client cert;
|
||||||
|
- a server cert (not client-auth);
|
||||||
|
- a mismatched key;
|
||||||
|
- a cert from another CA.
|
||||||
|
|
||||||
|
**Measured live, read-only:**
|
||||||
|
- node user namespaces, Debian 13.7 and AppArmor (via an Ansible `sysctl`/`cat`);
|
||||||
|
- DinD's MASQUERADE rules;
|
||||||
|
- the live ClusterPolicies, namespaced Policies, PSA labels, and cert-manager v1.21.3 with its
|
||||||
|
ClusterIssuers;
|
||||||
|
- the act-runner pod labels.
|
||||||
|
|
||||||
|
**Renovate (unchanged since the first iteration):**
|
||||||
|
- the validator with 44.167.0: exit 0;
|
||||||
|
- `extract` finds all four markers;
|
||||||
|
- the `kubernetes` duplicates are `skipReason: disabled`;
|
||||||
|
- 39.264.0 rejects `main`'s config.
|
||||||
|
|
||||||
|
**Local gate, CI results:** see the PR body. They are recorded there because this entry is part of
|
||||||
|
the tree being gated.
|
||||||
|
|
||||||
## What I did NOT verify
|
## What I did NOT verify
|
||||||
|
|
||||||
- **Nothing has been pushed to the registry, by me or by CI.** The publish path has never run:
|
- **platform/buildkit has never run in the cluster.** It can't until it merges: agents may not
|
||||||
- the registry HEAD/digest read-back;
|
`kubectl apply`, and Flux reconciles only `main`. So this PR's own `images` job **cannot build
|
||||||
|
and is expected to fail at the builder preflight**, which is the designed loud failure. The
|
||||||
|
following are therefore all unproven in-cluster:
|
||||||
|
- kubelet admitting the pod;
|
||||||
|
- containerd honouring `appArmorProfile: Unconfined`;
|
||||||
|
- the five emptyDirs;
|
||||||
|
- cert-manager issuing the three certs;
|
||||||
|
- the CiliumNetworkPolicies.
|
||||||
|
- **AppArmor `Unconfined` is reasoned, not measured.** My test host runs SELinux, so the AppArmor
|
||||||
|
requirement comes from upstream's example plus the nodes having AppArmor enabled. If the pod
|
||||||
|
were to work under `RuntimeDefault` AppArmor, that line could be tightened; I did not test it.
|
||||||
|
- **The local userns mapping is narrower than the cluster's.** Locally it was 65,535 ids across
|
||||||
|
two ranges; in the cluster it is the image's own `100000:65536` against real host subuids. That
|
||||||
|
is the normal case, but it is the cluster's first test.
|
||||||
|
- **The NetworkPolicies are untested.** In particular:
|
||||||
|
- that `toEntities: [ingress]` admits the push to the Gateway VIP. The comment explains the
|
||||||
|
reasoning, but it is unmeasured;
|
||||||
|
- that world :443/:80 covers every download the builds make. A dnf mirror on another port
|
||||||
|
would fail a build visibly.
|
||||||
|
- **No push, anywhere.** The whole publish path is unexercised:
|
||||||
|
- `REGISTRY_PUSH_TOKEN` reaching buildkitd through the buildx session;
|
||||||
|
- `push=true`;
|
||||||
|
- the metadata digest compared with the registry read-back;
|
||||||
|
- agent-runner FROM `toolbox@sha256:`;
|
||||||
- the immutable-tag guard;
|
- the immutable-tag guard;
|
||||||
- "already published at this commit" idempotence;
|
- idempotence.
|
||||||
- agent-runner `FROM <toolbox>@sha256:`;
|
|
||||||
- `docker login` with a PAT.
|
|
||||||
|
|
||||||
The first merge to main is the first test, and it needs `REGISTRY_PUSH_TOKEN` to exist first.
|
So are the 15 s Envoy route-timeout risk on blob finalisation, and the Gitea PVC (10Gi, shared
|
||||||
- **Large-layer push through the Gateway is UNVERIFIED.** Envoy has no request-body cap and no
|
with repositories) absorbing ~2 GB per toolbox publish until retention runs.
|
||||||
stream duration cap here. But with no route `timeout` set, Envoy's **default 15 s upstream
|
- **Node disk under real builds.** The cache emptyDir (6Gi limit, 4 GB GC cap) sits on nodes that
|
||||||
timeout** applies from the end of the request to the end of the response. That covers Gitea
|
had 7–9 GB free. It is untested under concurrent pressure.
|
||||||
finalising a blob: copying the upload into package storage on Longhorn. A ~250 MB compressed
|
- **Certificate rotation is designed, not exercised.** The server cert renews 30 days early but
|
||||||
layer should finish well inside 15 s, but I could not measure it without pushing. If the first
|
buildkitd only reads it at start; the client copy needs the script re-run.
|
||||||
publish shows 504s on blob PUTs, set `timeouts.request` on the gitea HTTPRoute.
|
- **`--oci-worker-no-process-sandbox` is an accepted risk, not a mitigated one.** A build step can
|
||||||
- **DinD fit is estimated, not measured** (no build has run there, see the BLOCKER):
|
ptrace the daemon and read its TLS key and cache. Anyone with the client cert, which means every
|
||||||
- memory: 2Gi is shared by the job container and every build container;
|
PR job, can run build steps.
|
||||||
- disk: the toolbox, agent-runner, conformance-runner, Fedora base and build cache should
|
- **Renovate has still never completed a run.** It is blocked on #80, on `RENOVATE_TOKEN` gaining
|
||||||
total roughly 3 GB on top of the 1.5 GB act image, against 7 GB free on the node, with
|
`read:package`, and on this branch. The `[email protected]` + 44.x image pairing is
|
||||||
`dind-storage` having no sizeLimit.
|
untested.
|
||||||
- **No image has been built in this runner's DinD**, because it cannot be done until the BLOCKER
|
- **Pods cannot pull these images until PR #61** (the kubelet pull secret) merges.
|
||||||
is resolved. Two things are therefore unmeasured until then:
|
- **`registry-retention.sh --apply` has never run**; its keep logic has only run against an empty
|
||||||
- DinD memory (2Gi) and disk fit;
|
registry.
|
||||||
- BuildKit (`buildx` v0.20.1 is on the daemon) as opposed to the legacy builder I tested
|
|
||||||
with.
|
|
||||||
- **Renovate has still never completed a run here.** Three things block it:
|
|
||||||
- #80, the `configurationFile` name;
|
|
||||||
- `RENOVATE_TOKEN` gaining `read:package`;
|
|
||||||
- this branch's version pin.
|
|
||||||
|
|
||||||
`renovatebot/github-action@v41.0.9` running a 44.x image is untested. The action only passes
|
|
||||||
the tag to `docker run`.
|
|
||||||
- **Pods cannot pull these images yet regardless.** The kubelet pull credential is PR #61
|
|
||||||
(`pm-agent-registry` dockerconfigjson, `read:package`), still open. Until it merges, a correct
|
|
||||||
digest still ends in ImagePullBackOff (401), as before.
|
|
||||||
- **`registry-retention.sh --apply` has never run**, and its keep logic has only run against an
|
|
||||||
empty registry.
|
|
||||||
- **`git diff` change detection on Gitea `push` events relies on `github.event.before`.** A
|
|
||||||
force-push or a first push falls back to building everything, which is intended but not
|
|
||||||
observed.
|
|
||||||
@@ -245,8 +245,8 @@ about runtime behaviour in this document is a design claim, not a measurement.
|
|||||||
the credential decision now being local Gitea accounts with PATs, that means a
|
the credential decision now being local Gitea accounts with PATs, that means a
|
||||||
`dockerconfigjson` for an account whose token carries `read:package`. Only then does the
|
`dockerconfigjson` for an account whose token carries `read:package`. Only then does the
|
||||||
digest matter. **Superseded 2026-10-29:** the build-and-push is no longer an operator step.
|
digest matter. **Superseded 2026-10-29:** the build-and-push is no longer an operator step.
|
||||||
`.gitea/workflows/images.yaml` builds and verifies the image with
|
`.gitea/workflows/images.yaml` builds and verifies the image on rootless BuildKit
|
||||||
`infra/agent/build-image.sh`, publishes it on a push to `main`, and Renovate opens the one
|
(`platform/buildkit/`), publishes it on a push to `main`, and Renovate opens the one
|
||||||
PR that replaces both placeholder digests (`decisions/2026-10-29-image-pipeline.md`). The
|
PR that replaces both placeholder digests (`decisions/2026-10-29-image-pipeline.md`). The
|
||||||
same two blockers sit in front of `conformance-runner`, which shares the registry path and
|
same two blockers sit in front of `conformance-runner`, which shares the registry path and
|
||||||
has never been pulled.
|
has never been pulled.
|
||||||
|
|||||||
@@ -50,7 +50,10 @@
|
|||||||
# default is deliberately EMPTY so a build that forgets it fails loudly instead of
|
# default is deliberately EMPTY so a build that forgets it fails loudly instead of
|
||||||
# resolving some other image named `agent-base`.
|
# resolving some other image named `agent-base`.
|
||||||
ARG AGENT_BASE_IMAGE=
|
ARG AGENT_BASE_IMAGE=
|
||||||
FROM ${AGENT_BASE_IMAGE}
|
# Stage `agent-runner` is the image that ships; stage `verify` (end of file) is a TEST stage built
|
||||||
|
# FROM it that only runs assertions and is never pushed. build-image.sh and the CI pipeline
|
||||||
|
# build `--target verify` to prove the image and `--target agent-runner` for the image itself.
|
||||||
|
FROM ${AGENT_BASE_IMAGE} AS agent-runner
|
||||||
|
|
||||||
# Re-declared AFTER the FROM: an ARG from before the first FROM is not in scope in the
|
# Re-declared AFTER the FROM: an ARG from before the first FROM is not in scope in the
|
||||||
# build stage, so the provenance layer below would otherwise record an empty base.
|
# build stage, so the provenance layer below would otherwise record an empty base.
|
||||||
@@ -208,3 +211,80 @@ WORKDIR /workspace
|
|||||||
# paths above; an ENTRYPOINT here would either be ignored or prepend something the policy
|
# paths above; an ENTRYPOINT here would either be ignored or prepend something the policy
|
||||||
# cannot see. `bash` is the interactive fallback for a human debugging the image locally.
|
# cannot see. `bash` is the interactive fallback for a human debugging the image locally.
|
||||||
CMD ["/bin/bash"]
|
CMD ["/bin/bash"]
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# verify — a TEST stage, never shipped. Every assertion infra/agent/build-image.sh used to make
|
||||||
|
# with `run --user 65532 --read-only --tmpfs /workspace --tmpfs /tmp --network none` now runs
|
||||||
|
# here, at build time, because CI builds on rootless BuildKit where no container runtime exists.
|
||||||
|
#
|
||||||
|
# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE, stated so nobody over-reads a green verify:
|
||||||
|
# * uid 65532 ............ exact: USER is inherited from the agent-runner stage.
|
||||||
|
# * --network none ....... exact: RUN --network=none, and the first check proves it.
|
||||||
|
# * --read-only rootfs ... NOT reproducible: a RUN step always has a writable layer. It is
|
||||||
|
# APPROXIMATED by permission: as uid 65532, every path the process can
|
||||||
|
# write is inventoried and must lie under /workspace, /tmp or /var/tmp
|
||||||
|
# (the pod mounts emptyDirs on the first two; /var/tmp is read-only
|
||||||
|
# there, and TMPDIR points into /workspace so nothing should use it).
|
||||||
|
# The pod's readOnlyRootFilesystem is enforced by admission
|
||||||
|
# (platform/kyverno/admission/pm-agent-job-shape.yaml), not by this.
|
||||||
|
# * --tmpfs /workspace ... approximated by the image's own /workspace (owned 65532), which is
|
||||||
|
# what the pod's emptyDir replaces.
|
||||||
|
# The probe script is COPYed into THIS stage only, so it never ships in the image.
|
||||||
|
# =============================================================================
|
||||||
|
FROM agent-runner AS verify
|
||||||
|
COPY probe-validation.sh /opt/verify/probe-validation.sh
|
||||||
|
RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
|
||||||
|
echo "--- no network (the build-time stand-in for run --network none) ---"
|
||||||
|
if getent hosts registry.npmjs.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then
|
||||||
|
echo "UNEXPECTED: network reachable in the verify stage"; exit 1
|
||||||
|
fi
|
||||||
|
echo " confirmed: no network"
|
||||||
|
echo "--- id ---"; id
|
||||||
|
echo "--- HOME ---"; echo "HOME=${HOME} CLAUDE_CONFIG_DIR=${CLAUDE_CONFIG_DIR}"
|
||||||
|
echo "--- baked versions ---"; cat /etc/agent-runner-versions
|
||||||
|
echo "--- the gate toolchain the worker needs (CLAUDE.md §3) ---"
|
||||||
|
for t in git python3 bash yamllint check-jsonschema kyverno promtool chainsaw flux sops age kubectl jq diff; do
|
||||||
|
if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t";
|
||||||
|
else printf " MISSING %s\n" "$t"; missing=1; fi
|
||||||
|
done
|
||||||
|
[ -z "${missing:-}" ] || { echo "the gate cannot run in this image"; exit 1; }
|
||||||
|
echo "--- python can import the renderer dependency ---"
|
||||||
|
python3 -c "import yaml; print('PyYAML', yaml.__version__)"
|
||||||
|
echo "--- claude ---"; claude --version
|
||||||
|
echo "--- the two entrypoints refuse arguments ---"
|
||||||
|
# Each must REFUSE. (The old docker-run version wrote `cmd | grep -q ... && echo`, which under
|
||||||
|
# `set -e` + pipefail could never fail the script; these are real assertions.)
|
||||||
|
for e in pm-worker pm-dispatch; do
|
||||||
|
out="$(/usr/local/bin/$e --anything 2>&1 || true)"
|
||||||
|
grep -q "takes no arguments" <<<"$out" || { echo "UNEXPECTED: $e accepted an argument: $out"; exit 1; }
|
||||||
|
echo " $e: rejected"
|
||||||
|
done
|
||||||
|
echo "--- the dispatcher selftest (branch vocabulary, result envelopes, Job template, persona) ---"
|
||||||
|
PM_SELFTEST=1 /usr/local/bin/pm-dispatch
|
||||||
|
echo "--- the entrypoints are root-owned and NOT writable by the runtime uid ---"
|
||||||
|
for f in /usr/local/bin/pm-worker /usr/local/bin/pm-dispatch /etc/pm-agent/worker-job-template.yaml; do
|
||||||
|
stat -c " %n %U:%G %a" "$f"
|
||||||
|
if [ -w "$f" ]; then echo "UNEXPECTED: uid $(id -u) can write $f"; exit 1; fi
|
||||||
|
done
|
||||||
|
echo " confirmed: not writable by uid $(id -u)"
|
||||||
|
echo "--- the root filesystem is not writable by the runtime uid (expected: a failure) ---"
|
||||||
|
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi
|
||||||
|
echo " confirmed: / is not writable"
|
||||||
|
echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---"
|
||||||
|
stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \
|
||||||
|
| grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)"
|
||||||
|
if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf ' %s\n' ${stray} | head -20; exit 1; fi
|
||||||
|
echo " confirmed: no writable path outside /workspace, /tmp, /var/tmp"
|
||||||
|
echo "--- the two writable paths really are writable ---"
|
||||||
|
touch "${HOME}/.probe" /tmp/.probe || { echo "UNEXPECTED: ${HOME} or /tmp not writable"; exit 1; }
|
||||||
|
echo " confirmed: ${HOME} and /tmp are writable"
|
||||||
|
echo "--- uid is non-zero ---"
|
||||||
|
[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; }
|
||||||
|
echo " confirmed: uid $(id -u), not root"
|
||||||
|
echo "--- no Anthropic API key is baked into the image (it would outrank the subscription token) ---"
|
||||||
|
[ -z "${ANTHROPIC_API_KEY:-}${ANTHROPIC_AUTH_TOKEN:-}" ] || { echo "UNEXPECTED: an Anthropic key is baked in"; exit 1; }
|
||||||
|
echo " confirmed: neither is set"
|
||||||
|
echo "--- pm-worker's input validation, with the interpreter and script that run in the cluster ---"
|
||||||
|
PM_WORKER_PATH=/usr/local/bin/pm-worker /bin/bash /opt/verify/probe-validation.sh
|
||||||
|
echo "AGENT-RUNNER VERIFIED"
|
||||||
|
VERIFY
|
||||||
+39
-91
@@ -2,7 +2,8 @@
|
|||||||
###############################################################################
|
###############################################################################
|
||||||
# infra/agent/build-image.sh — build the pinned `agent-runner` image and prove it works.
|
# infra/agent/build-image.sh — build the pinned `agent-runner` image and prove it works.
|
||||||
#
|
#
|
||||||
# ./build-image.sh # build + verify + print the image id
|
# ./build-image.sh # build the verify target, then the image
|
||||||
|
# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes)
|
||||||
# ./build-image.sh --no-verify # build only (skip the in-image proof)
|
# ./build-image.sh --no-verify # build only (skip the in-image proof)
|
||||||
# ENGINE=docker ./build-image.sh # build with docker instead of podman
|
# ENGINE=docker ./build-image.sh # build with docker instead of podman
|
||||||
# AGENT_BASE_IMAGE=home-k3s-dev:local ./build-image.sh
|
# AGENT_BASE_IMAGE=home-k3s-dev:local ./build-image.sh
|
||||||
@@ -11,12 +12,12 @@
|
|||||||
# IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this)
|
# IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this)
|
||||||
#
|
#
|
||||||
# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
|
# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
|
||||||
# Publishing is done by CI: .gitea/workflows/images.yaml calls this script, and on a push to
|
# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on
|
||||||
# `main` it pushes what this script built AND verified to the Gitea registry, records the
|
# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and
|
||||||
# registry-returned digest in the job summary, and Renovate then opens the pull request that
|
# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest
|
||||||
# moves platform/pm-agent/ onto that digest. So this script's job is to build, and to prove
|
# in the job summary. Renovate then opens the pull request that moves platform/pm-agent/ onto
|
||||||
# the toolchain and both entrypoints work INSIDE the image as the non-root uid with the
|
# that digest. This script is the LOCAL path (podman by default); its proof — the
|
||||||
# read-only root filesystem the admission policy forces — the same proof locally and in CI.
|
# Containerfile's `verify` stage — is the same one CI runs.
|
||||||
#
|
#
|
||||||
# VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the pins this image adds
|
# VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the pins this image adds
|
||||||
# (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds the
|
# (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds the
|
||||||
@@ -26,11 +27,10 @@
|
|||||||
# kept current by Renovate. An AGENT_BASE_IMAGE in the environment overrides it — CI passes
|
# kept current by Renovate. An AGENT_BASE_IMAGE in the environment overrides it — CI passes
|
||||||
# the toolbox it built in the same run, and a developer can pass `home-k3s-dev:local`.
|
# the toolbox it built in the same run, and a developer can pass `home-k3s-dev:local`.
|
||||||
#
|
#
|
||||||
# ENGINE PORTABILITY. CI runs this with ENGINE=docker against a Docker-in-Docker daemon, so
|
# NO `run` STEP. Verification used to `run` the image as uid 65532 with a read-only rootfs.
|
||||||
# nothing below may be podman-only (`image exists`, `--digestfile`, `{{.Digest}}`), and
|
# CI has no container runtime (rootless BuildKit only), so the assertions moved into the
|
||||||
# nothing may BIND-MOUNT a file from this checkout into a container: under DinD the daemon
|
# Containerfile's `verify` stage and this script builds that target. `--print-build-args`
|
||||||
# resolves bind sources on ITS filesystem, where this checkout does not exist, and silently
|
# prints the pins as KEY=VALUE for the CI pipeline; AGENT_BASE_IMAGE is the caller's choice.
|
||||||
# mounts an empty directory instead. Files go in over stdin.
|
|
||||||
###############################################################################
|
###############################################################################
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
@@ -39,9 +39,11 @@ REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
|
|||||||
ENGINE="${ENGINE:-podman}"
|
ENGINE="${ENGINE:-podman}"
|
||||||
|
|
||||||
VERIFY=1
|
VERIFY=1
|
||||||
|
PRINT_ARGS=0
|
||||||
for arg in "$@"; do
|
for arg in "$@"; do
|
||||||
case "$arg" in
|
case "$arg" in
|
||||||
--no-verify) VERIFY=0 ;;
|
--no-verify) VERIFY=0 ;;
|
||||||
|
--print-build-args) PRINT_ARGS=1 ;;
|
||||||
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
|
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||||
*) echo "unknown argument: $arg" >&2; exit 2 ;;
|
*) echo "unknown argument: $arg" >&2; exit 2 ;;
|
||||||
esac
|
esac
|
||||||
@@ -51,8 +53,6 @@ log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
|||||||
info() { printf ' %s\n' "$*"; }
|
info() { printf ' %s\n' "$*"; }
|
||||||
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
|
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
|
||||||
|
|
||||||
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
|
|
||||||
|
|
||||||
# --- pins: sourced, never invented -------------------------------------------
|
# --- pins: sourced, never invented -------------------------------------------
|
||||||
AGENT_ENV="${HERE}/versions.env"
|
AGENT_ENV="${HERE}/versions.env"
|
||||||
[ -f "${AGENT_ENV}" ] || die "missing ${AGENT_ENV} — the pins live there, not here"
|
[ -f "${AGENT_ENV}" ] || die "missing ${AGENT_ENV} — the pins live there, not here"
|
||||||
@@ -68,6 +68,17 @@ source "${AGENT_ENV}"
|
|||||||
: "${AGENT_MODEL:?AGENT_MODEL not set in ${AGENT_ENV}}"
|
: "${AGENT_MODEL:?AGENT_MODEL not set in ${AGENT_ENV}}"
|
||||||
: "${AGENT_IMAGE_REPO:?AGENT_IMAGE_REPO not set in ${AGENT_ENV}}"
|
: "${AGENT_IMAGE_REPO:?AGENT_IMAGE_REPO not set in ${AGENT_ENV}}"
|
||||||
|
|
||||||
|
# THE build args, in ONE place; `--print-build-args` hands them to CI (which builds on rootless
|
||||||
|
# BuildKit and supplies its own base reference). AGENT_BASE_IMAGE is deliberately NOT in this
|
||||||
|
# list: the caller decides the base.
|
||||||
|
BUILD_ARGS=(
|
||||||
|
"NODEJS_MAJOR=${NODEJS_MAJOR}" "CLAUDE_CODE_VERSION=${CLAUDE_CODE_VERSION}"
|
||||||
|
"CLAUDE_CODE_MIN_VERSION=${CLAUDE_CODE_MIN_VERSION}" "AGENT_MODEL=${AGENT_MODEL}"
|
||||||
|
)
|
||||||
|
if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
|
||||||
|
|
||||||
|
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
|
||||||
|
|
||||||
# --- the base must exist, locally or in the registry ---------------------------
|
# --- the base must exist, locally or in the registry ---------------------------
|
||||||
# Deliberately NOT built on demand. The toolbox is its own reviewed artefact with its own
|
# Deliberately NOT built on demand. The toolbox is its own reviewed artefact with its own
|
||||||
# pins, and silently building it here would mean this script could change the toolchain a
|
# pins, and silently building it here would mean this script could change the toolchain a
|
||||||
@@ -103,89 +114,26 @@ info " model ${AGENT_MODEL}"
|
|||||||
# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
|
# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
|
||||||
LABEL_ARGS=()
|
LABEL_ARGS=()
|
||||||
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
|
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
|
||||||
|
ARG_FLAGS=(--build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}")
|
||||||
"${ENGINE}" build "${LABEL_ARGS[@]}" \
|
for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
|
||||||
--file "${HERE}/Containerfile" \
|
|
||||||
--tag "${IMAGE_REF}" \
|
|
||||||
--build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}" \
|
|
||||||
--build-arg "NODEJS_MAJOR=${NODEJS_MAJOR}" \
|
|
||||||
--build-arg "CLAUDE_CODE_VERSION=${CLAUDE_CODE_VERSION}" \
|
|
||||||
--build-arg "CLAUDE_CODE_MIN_VERSION=${CLAUDE_CODE_MIN_VERSION}" \
|
|
||||||
--build-arg "AGENT_MODEL=${AGENT_MODEL}" \
|
|
||||||
"${HERE}"
|
|
||||||
|
|
||||||
# --- verify: the point of the whole exercise ---------------------------------
|
# --- verify: the point of the whole exercise ---------------------------------
|
||||||
# A build that merely succeeded is not evidence this image is usable. Prove the tools
|
# A build that merely succeeded is not evidence this image is usable. The proof is the
|
||||||
# EXECUTE, as the NON-ROOT uid, with a READ-ONLY root filesystem and only the two volumes
|
# Containerfile's `verify` stage: as the NON-ROOT uid, with no network, the toolchain executes,
|
||||||
# the admission policy admits — the exact shape a worker Job runs in. Anything that works
|
# both entrypoints refuse arguments, the dispatcher selftest passes, the entrypoints are not
|
||||||
# here but not there is a 3am discovery.
|
# writable, nothing outside /workspace and /tmp is writable, and pm-worker's input guards
|
||||||
|
# (probe-validation.sh) refuse every hostile input. It is a build target rather than a
|
||||||
|
# `run` so the SAME proof runs here and in CI, where no container runtime exists — see the
|
||||||
|
# stage's header for what a build step can and cannot reproduce of the pod's shape.
|
||||||
if [ "${VERIFY}" -eq 1 ]; then
|
if [ "${VERIFY}" -eq 1 ]; then
|
||||||
log "Verifying inside the image (uid 65532, read-only rootfs, workspace+tmp only)"
|
log "Verifying (Containerfile target: verify)"
|
||||||
"${ENGINE}" run --rm \
|
"${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}"
|
||||||
--user 65532:65532 \
|
|
||||||
--read-only \
|
|
||||||
--tmpfs /workspace:rw,mode=1777 \
|
|
||||||
--tmpfs /tmp:rw,mode=1777 \
|
|
||||||
--network none \
|
|
||||||
--entrypoint "" \
|
|
||||||
"${IMAGE_REF}" \
|
|
||||||
/bin/bash -euo pipefail -c '
|
|
||||||
echo "--- id ---"; id
|
|
||||||
echo "--- HOME ---"; echo "HOME=${HOME} CLAUDE_CONFIG_DIR=${CLAUDE_CONFIG_DIR}"
|
|
||||||
echo "--- baked versions ---"; cat /etc/agent-runner-versions
|
|
||||||
echo "--- the gate toolchain the worker needs (CLAUDE.md §3) ---"
|
|
||||||
for t in git python3 bash yamllint check-jsonschema kyverno promtool chainsaw flux sops age kubectl jq diff; do
|
|
||||||
if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t";
|
|
||||||
else printf " MISSING %s\n" "$t"; missing=1; fi
|
|
||||||
done
|
|
||||||
[ -z "${missing:-}" ] || { echo "the gate cannot run in this image"; exit 1; }
|
|
||||||
echo "--- python can import the renderer dependency ---"
|
|
||||||
python3 -c "import yaml; print(\"PyYAML\", yaml.__version__)"
|
|
||||||
echo "--- claude ---"; claude --version
|
|
||||||
echo "--- the two entrypoints refuse arguments ---"
|
|
||||||
/usr/local/bin/pm-worker --anything 2>&1 | grep -q "takes no arguments" && echo " pm-worker: rejected"
|
|
||||||
/usr/local/bin/pm-dispatch --anything 2>&1 | grep -q "takes no arguments" && echo " pm-dispatch: rejected"
|
|
||||||
echo "--- the dispatcher selftest (branch vocabulary, result envelopes, Job template, persona) ---"
|
|
||||||
PM_SELFTEST=1 /usr/local/bin/pm-dispatch
|
|
||||||
echo "--- the entrypoints are root-owned and NOT writable by the runtime uid ---"
|
|
||||||
for f in /usr/local/bin/pm-worker /usr/local/bin/pm-dispatch /etc/pm-agent/worker-job-template.yaml; do
|
|
||||||
stat -c " %n %U:%G %a" "$f"
|
|
||||||
if [ -w "$f" ]; then echo "UNEXPECTED: uid $(id -u) can write $f"; exit 1; fi
|
|
||||||
done
|
|
||||||
echo " confirmed: not writable by uid $(id -u)"
|
|
||||||
echo "--- rootfs is read-only (expected: a failure) ---"
|
|
||||||
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
|
|
||||||
echo " confirmed: / is read-only"
|
|
||||||
echo "--- the two writable paths really are writable ---"
|
|
||||||
touch "${HOME}/.probe" /tmp/.probe && echo " confirmed: ${HOME} and /tmp are writable"
|
|
||||||
echo "--- uid is non-zero ---"
|
|
||||||
[ "$(id -u)" -ne 0 ] && echo " confirmed: uid $(id -u), not root"
|
|
||||||
echo "--- no Anthropic API key is baked into the image (it would outrank the subscription token) ---"
|
|
||||||
[ -z "${ANTHROPIC_API_KEY:-}${ANTHROPIC_AUTH_TOKEN:-}" ] && echo " confirmed: neither is set"
|
|
||||||
'
|
|
||||||
# The worker's input guards, exercised in the image, with the interpreter and the script
|
|
||||||
# that will actually run in the cluster. The probe is mounted read-only rather than baked
|
|
||||||
# in: it is a build-time check, not something a worker Job has any business carrying.
|
|
||||||
log "Probing pm-worker's input validation inside the image"
|
|
||||||
#
|
|
||||||
# The probe goes in over STDIN, not as a bind mount: under Docker-in-Docker (CI) a bind
|
|
||||||
# source is resolved on the daemon's filesystem, where this file does not exist, and the
|
|
||||||
# container would get an empty directory. PM_WORKER_PATH points it at the baked worker,
|
|
||||||
# since the probe's own default (a sibling of the probe file) has no meaning on stdin.
|
|
||||||
"${ENGINE}" run --rm -i \
|
|
||||||
--user 65532:65532 \
|
|
||||||
--read-only \
|
|
||||||
--tmpfs /workspace:rw,mode=1777 \
|
|
||||||
--tmpfs /tmp:rw,mode=1777 \
|
|
||||||
--network none \
|
|
||||||
-e PM_WORKER_PATH=/usr/local/bin/pm-worker \
|
|
||||||
--entrypoint "" \
|
|
||||||
"${IMAGE_REF}" \
|
|
||||||
/bin/bash -s < "${HERE}/probe-validation.sh"
|
|
||||||
|
|
||||||
log "Image verified"
|
log "Image verified"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target agent-runner \
|
||||||
|
--file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}"
|
||||||
|
|
||||||
# --- result ------------------------------------------------------------------
|
# --- result ------------------------------------------------------------------
|
||||||
# No digest is printed: a manifest digest does not exist until a registry has the manifest,
|
# No digest is printed: a manifest digest does not exist until a registry has the manifest,
|
||||||
# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at
|
# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at
|
||||||
|
|||||||
@@ -16,14 +16,14 @@
|
|||||||
# scrubbed environment and fake credentials, and if a guard ever stopped firing, the case
|
# scrubbed environment and fake credentials, and if a guard ever stopped firing, the case
|
||||||
# would fail here rather than by really cloning something.
|
# would fail here rather than by really cloning something.
|
||||||
#
|
#
|
||||||
# Run offline, no cluster, no Gitea, no model. infra/agent/build-image.sh runs it inside the
|
# Run offline, no cluster, no Gitea, no model. The Containerfile's `verify` stage runs it inside
|
||||||
# built image as well, where the interpreter and the script are the ones that will actually
|
# the built image as well (as uid 65532, no network), where the interpreter and the script are
|
||||||
# run in the cluster.
|
# the ones that will actually run in the cluster — locally via build-image.sh and in CI.
|
||||||
###############################################################################
|
###############################################################################
|
||||||
set -uo pipefail
|
set -uo pipefail
|
||||||
|
|
||||||
# `:-$0` because build-image.sh feeds this script over STDIN (`bash -s`), where BASH_SOURCE is
|
# `:-$0` keeps this safe if it is ever fed over stdin (`bash -s`), where BASH_SOURCE is empty
|
||||||
# empty and `set -u` would abort on it; PM_WORKER_PATH is always set in that case.
|
# and `set -u` would abort on it.
|
||||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)"
|
HERE="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)"
|
||||||
WORKER="${PM_WORKER_PATH:-${HERE}/pm-worker}"
|
WORKER="${PM_WORKER_PATH:-${HERE}/pm-worker}"
|
||||||
[ -r "$WORKER" ] || { echo "cannot read ${WORKER}" >&2; exit 2; }
|
[ -r "$WORKER" ] || { echo "cannot read ${WORKER}" >&2; exit 2; }
|
||||||
|
|||||||
@@ -8,7 +8,10 @@
|
|||||||
# packages are retained as harmless disk-image utilities. Fedora base is pinned for
|
# packages are retained as harmless disk-image utilities. Fedora base is pinned for
|
||||||
# reproducibility.
|
# reproducibility.
|
||||||
ARG FEDORA_VERSION=44
|
ARG FEDORA_VERSION=44
|
||||||
FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION}
|
# Stage `toolbox` is the image that ships. Stage `verify` (end of file) is a TEST stage: it is
|
||||||
|
# built FROM the finished image and only runs assertions, and it is never pushed. Build with
|
||||||
|
# `--target toolbox` for the image and `--target verify` to prove it (build-image.sh does both).
|
||||||
|
FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} AS toolbox
|
||||||
|
|
||||||
ARG OPENTOFU_VERSION
|
ARG OPENTOFU_VERSION
|
||||||
ARG AGE_VERSION
|
ARG AGE_VERSION
|
||||||
@@ -95,3 +98,48 @@ RUN { echo "opentofu=${OPENTOFU_VERSION}"; echo "age=${AGE_VERSION}"; echo "sops
|
|||||||
|
|
||||||
WORKDIR /work
|
WORKDIR /work
|
||||||
CMD ["/bin/bash"]
|
CMD ["/bin/bash"]
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# verify — a TEST stage, never shipped. It runs every assertion against the finished image AT
|
||||||
|
# BUILD TIME, so CI needs no container runtime to prove the image: the image pipeline builds
|
||||||
|
# this target on rootless BuildKit (platform/buildkit/), where `docker run` does not exist.
|
||||||
|
# `RUN --network=none` reproduces the old `run --network none` exactly, and the first check
|
||||||
|
# below proves it (it fails if any network is reachable).
|
||||||
|
# =============================================================================
|
||||||
|
FROM toolbox AS verify
|
||||||
|
ARG PROMTOOL_VERSION
|
||||||
|
ARG KUBECTL_VERSION
|
||||||
|
ARG CHAINSAW_VERSION
|
||||||
|
ARG KYVERNO_VERSION
|
||||||
|
ARG SOPS_VERSION
|
||||||
|
RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
|
||||||
|
echo "--- no network (the build-time stand-in for run --network none) ---"
|
||||||
|
if getent hosts registry.fedoraproject.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then
|
||||||
|
echo "UNEXPECTED: network reachable in the verify stage"; exit 1
|
||||||
|
fi
|
||||||
|
echo " confirmed: no network"
|
||||||
|
echo "--- baked versions ---"; cat /etc/dev-toolchain-versions
|
||||||
|
echo "--- every tool is on PATH ---"
|
||||||
|
for t in tofu age age-keygen sops flux kubectl helm kyverno kyverno-json chainsaw yq \
|
||||||
|
yamllint check-jsonschema promtool ansible-playbook git make jq diff python3 \
|
||||||
|
wg qrencode; do
|
||||||
|
if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t";
|
||||||
|
else printf " MISSING %s\n" "$t"; missing=1; fi
|
||||||
|
done
|
||||||
|
[ -z "${missing:-}" ] || { echo "toolbox is missing tools"; exit 1; }
|
||||||
|
echo "--- version-sensitive tools report their pins ---"
|
||||||
|
: "${PROMTOOL_VERSION:?}" "${KUBECTL_VERSION:?}" "${CHAINSAW_VERSION:?}" "${KYVERNO_VERSION:?}" "${SOPS_VERSION:?}"
|
||||||
|
check() { # name pin actual-output
|
||||||
|
if printf "%s" "$3" | grep -qF "$2"; then printf " ok %-10s %s\n" "$1" "$2";
|
||||||
|
else printf " DRIFT %-10s want %s, got: %s\n" "$1" "$2" "$(printf "%s" "$3" | head -1)"; drift=1; fi
|
||||||
|
}
|
||||||
|
check promtool "${PROMTOOL_VERSION}" "$(promtool --version 2>&1)"
|
||||||
|
check kubectl "${KUBECTL_VERSION}" "$(kubectl version --client 2>&1)"
|
||||||
|
check chainsaw "${CHAINSAW_VERSION}" "$(chainsaw version 2>&1)"
|
||||||
|
check kyverno "${KYVERNO_VERSION}" "$(kyverno version 2>&1)"
|
||||||
|
check sops "${SOPS_VERSION}" "$(sops --version --disable-version-check 2>&1 || sops --version 2>&1)"
|
||||||
|
[ -z "${drift:-}" ] || { echo "a tool does not report its pinned version"; exit 1; }
|
||||||
|
echo "--- python can import what the renderer and the gate need ---"
|
||||||
|
python3 -c "import yaml; print(' PyYAML', yaml.__version__)"
|
||||||
|
echo "TOOLBOX VERIFIED"
|
||||||
|
VERIFY
|
||||||
@@ -36,9 +36,11 @@ cd infra/devcontainer
|
|||||||
|
|
||||||
### The published toolbox
|
### The published toolbox
|
||||||
|
|
||||||
`./dev.sh build` delegates to [`build-image.sh`](build-image.sh), which is the **one** build
|
`./dev.sh build` delegates to [`build-image.sh`](build-image.sh). The recipe is the
|
||||||
recipe: CI runs the same script (with its in-image verification — every tool executes, and the
|
`Containerfile` plus that script's build-arg list, and the proof is the Containerfile's `verify`
|
||||||
version-sensitive ones report their `versions.env` pins) and, on a push to `main` that changes
|
stage (every tool on `PATH`, the version-sensitive ones report their `versions.env` pins, no
|
||||||
|
network). CI builds the same Containerfile on rootless BuildKit with the args from
|
||||||
|
`build-image.sh --print-build-args`, and, on a push to `main` that changes
|
||||||
`Containerfile`, `versions.env` or `build-image.sh`, **publishes** the result as
|
`Containerfile`, `versions.env` or `build-image.sh`, **publishes** the result as
|
||||||
`gitea.home.keithjackson.dev/home-network/toolbox` (tags `sha-<commit>` and `main`) —
|
`gitea.home.keithjackson.dev/home-network/toolbox` (tags `sha-<commit>` and `main`) —
|
||||||
see [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). `agent-runner` is
|
see [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). `agent-runner` is
|
||||||
|
|||||||
@@ -6,12 +6,14 @@
|
|||||||
# ./build-image.sh --no-verify # build only
|
# ./build-image.sh --no-verify # build only
|
||||||
# IMAGE=foo:bar ./build-image.sh # build under another local name
|
# IMAGE=foo:bar ./build-image.sh # build under another local name
|
||||||
# ENGINE=docker ./build-image.sh # build with docker instead of podman
|
# ENGINE=docker ./build-image.sh # build with docker instead of podman
|
||||||
|
# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes)
|
||||||
#
|
#
|
||||||
# ONE BUILD DEFINITION, TWO CALLERS. `dev.sh build` calls this for local development, and the
|
# ONE BUILD DEFINITION, TWO CALLERS. `dev.sh build` calls this for local development. The CI
|
||||||
# CI image pipeline (.gitea/workflows/images.yaml -> .gitea/workflows/scripts/images.sh) calls
|
# image pipeline (.gitea/workflows/images.yaml -> .gitea/workflows/scripts/images.sh) builds the
|
||||||
# it to build the image it then PUBLISHES as gitea.home.keithjackson.dev/home-network/toolbox.
|
# SAME Containerfile on rootless BuildKit (platform/buildkit/) — no container engine exists in a
|
||||||
# Keeping the build args in exactly one place is what makes "the toolbox CI publishes" and "the
|
# CI job — taking its build args from `--print-build-args`, and PUBLISHES the result as
|
||||||
# toolbox you gate with locally" the same recipe.
|
# gitea.home.keithjackson.dev/home-network/toolbox. The Containerfile (including its `verify`
|
||||||
|
# test stage) and the arg list here are the one recipe both use.
|
||||||
#
|
#
|
||||||
# THIS SCRIPT NEVER PUSHES. Publishing is CI's job (images.yaml, on push to main), not a
|
# THIS SCRIPT NEVER PUSHES. Publishing is CI's job (images.yaml, on push to main), not a
|
||||||
# workstation's: an image that exists only because someone ran `podman push` from a laptop has
|
# workstation's: an image that exists only because someone ran `podman push` from a laptop has
|
||||||
@@ -26,9 +28,11 @@ ENGINE="${ENGINE:-podman}"
|
|||||||
IMAGE="${IMAGE:-home-k3s-dev:local}"
|
IMAGE="${IMAGE:-home-k3s-dev:local}"
|
||||||
|
|
||||||
VERIFY=1
|
VERIFY=1
|
||||||
|
PRINT_ARGS=0
|
||||||
for arg in "$@"; do
|
for arg in "$@"; do
|
||||||
case "$arg" in
|
case "$arg" in
|
||||||
--no-verify) VERIFY=0 ;;
|
--no-verify) VERIFY=0 ;;
|
||||||
|
--print-build-args) PRINT_ARGS=1 ;;
|
||||||
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
|
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||||
*) echo "unknown argument: $arg" >&2; exit 2 ;;
|
*) echo "unknown argument: $arg" >&2; exit 2 ;;
|
||||||
esac
|
esac
|
||||||
@@ -37,72 +41,44 @@ done
|
|||||||
log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
||||||
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
|
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
|
||||||
|
|
||||||
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
|
|
||||||
|
|
||||||
# shellcheck disable=SC1091
|
# shellcheck disable=SC1091
|
||||||
source "${HERE}/versions.env"
|
source "${HERE}/versions.env"
|
||||||
|
|
||||||
|
# THE build args, in ONE place. `--print-build-args` hands exactly this list to the CI pipeline
|
||||||
|
# (.gitea/workflows/scripts/images.sh), which builds on rootless BuildKit rather than calling
|
||||||
|
# "$ENGINE build" — so CI and a local build cannot drift apart on a pin.
|
||||||
|
BUILD_ARGS=(
|
||||||
|
"FEDORA_VERSION=$FEDORA_VERSION" "OPENTOFU_VERSION=$OPENTOFU_VERSION" "AGE_VERSION=$AGE_VERSION"
|
||||||
|
"SOPS_VERSION=$SOPS_VERSION" "FLUX_VERSION=$FLUX_VERSION" "KUBECTL_VERSION=$KUBECTL_VERSION"
|
||||||
|
"HELM_VERSION=$HELM_VERSION" "KYVERNO_VERSION=$KYVERNO_VERSION"
|
||||||
|
"KYVERNO_JSON_VERSION=$KYVERNO_JSON_VERSION" "CHAINSAW_VERSION=$CHAINSAW_VERSION"
|
||||||
|
"YAMLLINT_VERSION=$YAMLLINT_VERSION" "YQ_VERSION=$YQ_VERSION"
|
||||||
|
"CHECK_JSONSCHEMA_VERSION=$CHECK_JSONSCHEMA_VERSION" "PROMTOOL_VERSION=$PROMTOOL_VERSION"
|
||||||
|
)
|
||||||
|
if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
|
||||||
|
|
||||||
|
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
|
||||||
|
|
||||||
|
ARG_FLAGS=()
|
||||||
|
for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
|
||||||
# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
|
# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
|
||||||
LABEL_ARGS=()
|
LABEL_ARGS=()
|
||||||
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
|
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
|
||||||
|
|
||||||
log "Building ${IMAGE}"
|
# --- verify --------------------------------------------------------------------
|
||||||
"$ENGINE" build "${LABEL_ARGS[@]}" \
|
# The assertions live in the Containerfile's `verify` stage (every tool on PATH, the
|
||||||
--build-arg FEDORA_VERSION="$FEDORA_VERSION" \
|
# version-sensitive ones report their versions.env pins — promtool above all, because
|
||||||
--build-arg OPENTOFU_VERSION="$OPENTOFU_VERSION" \
|
# ci/versions.env asserts a promtool/Prometheus lockstep — and no network), so the SAME proof
|
||||||
--build-arg AGE_VERSION="$AGE_VERSION" \
|
# runs here under podman and in CI under rootless BuildKit. A failing assertion fails the build.
|
||||||
--build-arg SOPS_VERSION="$SOPS_VERSION" \
|
|
||||||
--build-arg FLUX_VERSION="$FLUX_VERSION" \
|
|
||||||
--build-arg KUBECTL_VERSION="$KUBECTL_VERSION" \
|
|
||||||
--build-arg HELM_VERSION="$HELM_VERSION" \
|
|
||||||
--build-arg KYVERNO_VERSION="$KYVERNO_VERSION" \
|
|
||||||
--build-arg KYVERNO_JSON_VERSION="$KYVERNO_JSON_VERSION" \
|
|
||||||
--build-arg CHAINSAW_VERSION="$CHAINSAW_VERSION" \
|
|
||||||
--build-arg YAMLLINT_VERSION="$YAMLLINT_VERSION" \
|
|
||||||
--build-arg YQ_VERSION="$YQ_VERSION" \
|
|
||||||
--build-arg CHECK_JSONSCHEMA_VERSION="$CHECK_JSONSCHEMA_VERSION" \
|
|
||||||
--build-arg PROMTOOL_VERSION="$PROMTOOL_VERSION" \
|
|
||||||
-t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}"
|
|
||||||
|
|
||||||
# --- verify ------------------------------------------------------------------
|
|
||||||
# A build that succeeded is not evidence the toolbox can run the gate. Every tool the gate and
|
|
||||||
# the runbooks call must EXECUTE, and the version-sensitive ones must report the version
|
|
||||||
# versions.env pins — promtool above all, because ci/versions.env asserts a promtool/Prometheus
|
|
||||||
# lockstep and a drifted engine evaluates alert-rule tests differently (see versions.env).
|
|
||||||
if [ "${VERIFY}" -eq 1 ]; then
|
if [ "${VERIFY}" -eq 1 ]; then
|
||||||
log "Verifying the toolchain inside ${IMAGE}"
|
log "Verifying (Containerfile target: verify)"
|
||||||
"$ENGINE" run --rm --network none --entrypoint "" \
|
"$ENGINE" build "${ARG_FLAGS[@]}" --target verify -f "${HERE}/Containerfile" "${HERE}"
|
||||||
-e PROMTOOL_VERSION="${PROMTOOL_VERSION}" \
|
|
||||||
-e KUBECTL_VERSION="${KUBECTL_VERSION}" \
|
|
||||||
-e CHAINSAW_VERSION="${CHAINSAW_VERSION}" \
|
|
||||||
-e KYVERNO_VERSION="${KYVERNO_VERSION}" \
|
|
||||||
-e SOPS_VERSION="${SOPS_VERSION}" \
|
|
||||||
"$IMAGE" /bin/bash -euo pipefail -c '
|
|
||||||
echo "--- baked versions ---"; cat /etc/dev-toolchain-versions
|
|
||||||
echo "--- every tool is on PATH ---"
|
|
||||||
for t in tofu age age-keygen sops flux kubectl helm kyverno kyverno-json chainsaw yq \
|
|
||||||
yamllint check-jsonschema promtool ansible-playbook git make jq diff python3 \
|
|
||||||
wg qrencode; do
|
|
||||||
if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t";
|
|
||||||
else printf " MISSING %s\n" "$t"; missing=1; fi
|
|
||||||
done
|
|
||||||
[ -z "${missing:-}" ] || { echo "toolbox is missing tools"; exit 1; }
|
|
||||||
echo "--- version-sensitive tools report their pins ---"
|
|
||||||
check() { # name pin actual-output
|
|
||||||
if printf "%s" "$3" | grep -qF "$2"; then printf " ok %-10s %s\n" "$1" "$2";
|
|
||||||
else printf " DRIFT %-10s want %s, got: %s\n" "$1" "$2" "$(printf "%s" "$3" | head -1)"; drift=1; fi
|
|
||||||
}
|
|
||||||
check promtool "${PROMTOOL_VERSION}" "$(promtool --version 2>&1)"
|
|
||||||
check kubectl "${KUBECTL_VERSION}" "$(kubectl version --client 2>&1)"
|
|
||||||
check chainsaw "${CHAINSAW_VERSION}" "$(chainsaw version 2>&1)"
|
|
||||||
check kyverno "${KYVERNO_VERSION}" "$(kyverno version 2>&1)"
|
|
||||||
check sops "${SOPS_VERSION}" "$(sops --version --disable-version-check 2>&1 || sops --version 2>&1)"
|
|
||||||
[ -z "${drift:-}" ] || { echo "a tool does not report its pinned version"; exit 1; }
|
|
||||||
echo "--- python can import what the renderer and the gate need ---"
|
|
||||||
python3 -c "import yaml; print(\" PyYAML\", yaml.__version__)"
|
|
||||||
'
|
|
||||||
log "Toolbox verified"
|
log "Toolbox verified"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
log "Building ${IMAGE}"
|
||||||
|
"$ENGINE" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target toolbox \
|
||||||
|
-t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}"
|
||||||
|
|
||||||
log "Built ${IMAGE}"
|
log "Built ${IMAGE}"
|
||||||
printf ' image id %s\n' "$("$ENGINE" image inspect --format '{{.Id}}' "$IMAGE")"
|
printf ' image id %s\n' "$("$ENGINE" image inspect --format '{{.Id}}' "$IMAGE")"
|
||||||
Executable
+104
@@ -0,0 +1,104 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Copy the image builder's CLIENT certificate from the cluster into the three repository Actions
|
||||||
|
# secrets the image pipeline reads: BUILDKIT_CA, BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY.
|
||||||
|
#
|
||||||
|
# ./infra/gitea/buildkit-client-secret.sh # copy and verify
|
||||||
|
# ./infra/gitea/buildkit-client-secret.sh --check # compare only; changes nothing
|
||||||
|
#
|
||||||
|
# RUNNING IT IS AN OPERATOR ACTION. It writes repository secrets.
|
||||||
|
#
|
||||||
|
# WHERE THE CERTIFICATE COMES FROM. platform/buildkit/certificates.yaml has cert-manager issue
|
||||||
|
# `buildkit-client-ci` from a CA that exists only for the builder; buildkitd admits a connection
|
||||||
|
# only with a certificate signed by that CA. CI jobs hold NO cluster credential (R-SEC-02), so they
|
||||||
|
# cannot read the Secret themselves: this script is the one bridge, run by a human who has both
|
||||||
|
# a kube context and the Gitea credential.
|
||||||
|
#
|
||||||
|
# WHEN TO RUN IT
|
||||||
|
# 1. once, after Flux has applied platform/buildkit (Kustomization `buildkit` Ready, and
|
||||||
|
# `kubectl -n buildkit get certificate buildkit-client-ci` READY=True);
|
||||||
|
# 2. after every renewal of that certificate (duration 1y, renewed 30 days early). Until then
|
||||||
|
# the image pipeline fails LOUDLY at its builder preflight — it never skips.
|
||||||
|
# `--check` tells you whether the copy is stale.
|
||||||
|
#
|
||||||
|
# NOTHING SECRET IS PRINTED. The key goes from `kubectl get secret` into the Gitea secrets API
|
||||||
|
# through shell variables and nowhere else. The script prints only names, sizes and the
|
||||||
|
# certificate's subject and expiry (public data).
|
||||||
|
#
|
||||||
|
# NEEDS: kubectl with a context for the cluster (default k3s-prod), jq, curl, openssl, base64,
|
||||||
|
# and the Gitea credential `git` already has stored for the host — the same requirements as
|
||||||
|
# infra/gitea/conformance-trigger-kubeconfig.sh, run from the same place.
|
||||||
|
set -euo pipefail
|
||||||
|
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||||
|
NS=buildkit; SECRET=buildkit-client-ci; KCTX="${KCTX:-k3s-prod}"
|
||||||
|
CHECK=0
|
||||||
|
for a in "$@"; do
|
||||||
|
case "$a" in
|
||||||
|
--check) CHECK=1 ;;
|
||||||
|
-h|--help) sed -n '2,/^set -euo/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||||
|
*) echo "unknown argument: $a" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
die() { echo "ERROR: $*" >&2; exit 1; }
|
||||||
|
for t in kubectl jq curl openssl base64 git; do command -v "$t" >/dev/null || die "missing required tool: $t"; done
|
||||||
|
|
||||||
|
echo "==> reading Secret ${NS}/${SECRET} (context ${KCTX}; values not shown)"
|
||||||
|
js="$(kubectl --context "$KCTX" -n "$NS" get secret "$SECRET" -o json)" \
|
||||||
|
|| die "cannot read ${NS}/${SECRET} — is platform/buildkit deployed and the Certificate Ready?"
|
||||||
|
CA="$(jq -r '.data["ca.crt"] // empty' <<<"$js" | base64 -d)"
|
||||||
|
CRT="$(jq -r '.data["tls.crt"] // empty' <<<"$js" | base64 -d)"
|
||||||
|
KEY="$(jq -r '.data["tls.key"] // empty' <<<"$js" | base64 -d)"
|
||||||
|
unset js
|
||||||
|
[ -n "$CA" ] && [ -n "$CRT" ] && [ -n "$KEY" ] || die "${NS}/${SECRET} lacks ca.crt/tls.crt/tls.key"
|
||||||
|
|
||||||
|
# Sanity before anything is written: the leaf is a CLIENT cert issued by this CA, and the key
|
||||||
|
# belongs to it. A wrong pairing would otherwise surface only as a TLS failure in CI.
|
||||||
|
openssl verify -CAfile <(printf '%s\n' "$CA") <(printf '%s\n' "$CRT") >/dev/null 2>&1 \
|
||||||
|
|| die "tls.crt does not verify against ca.crt"
|
||||||
|
openssl x509 -noout -ext extendedKeyUsage <<<"$CRT" 2>/dev/null | grep -q "TLS Web Client Authentication" \
|
||||||
|
|| die "tls.crt is not a client-auth certificate"
|
||||||
|
[ "$(openssl x509 -noout -pubkey <<<"$CRT" | openssl sha256)" = "$(openssl pkey -pubout <<<"$KEY" 2>/dev/null | openssl sha256)" ] \
|
||||||
|
|| die "tls.key does not match tls.crt"
|
||||||
|
echo " $(openssl x509 -noout -subject <<<"$CRT") / $(openssl x509 -noout -enddate <<<"$CRT")"
|
||||||
|
|
||||||
|
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||||
|
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"; unset creds
|
||||||
|
[ -n "$U" ] && [ -n "$P" ] || die "no stored credential for $HOST"
|
||||||
|
echo "==> authenticating to Gitea as '${U}' (secret not shown)"
|
||||||
|
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||||
|
API="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets"
|
||||||
|
|
||||||
|
if [ "$CHECK" -eq 1 ]; then
|
||||||
|
# Gitea never returns a secret's value, only its name and creation time. "Stale" therefore
|
||||||
|
# means: the secret was created before the certificate's current notBefore.
|
||||||
|
names="$(api "$API" | jq -c '[.[]? | {name, created_at}]')"
|
||||||
|
nb="$(date -u -d "$(openssl x509 -noout -startdate <<<"$CRT" | cut -d= -f2)" +%s)"
|
||||||
|
rc=0
|
||||||
|
for n in BUILDKIT_CA BUILDKIT_CLIENT_CERT BUILDKIT_CLIENT_KEY; do
|
||||||
|
c="$(jq -r --arg n "$n" '.[] | select(.name==$n) | .created_at' <<<"$names")"
|
||||||
|
if [ -z "$c" ]; then echo " MISSING $n"; rc=1
|
||||||
|
elif [ "$(date -u -d "$c" +%s)" -lt "$nb" ]; then echo " STALE $n (stored $c, certificate re-issued since)"; rc=1
|
||||||
|
else echo " current $n (stored $c)"; fi
|
||||||
|
done
|
||||||
|
exit "$rc"
|
||||||
|
fi
|
||||||
|
|
||||||
|
put() { # name value
|
||||||
|
local code
|
||||||
|
code="$(api -o /dev/null -w '%{http_code}' -X PUT -H 'Content-Type: application/json' \
|
||||||
|
-d "$(jq -n --arg d "$2" '{data:$d}')" "${API}/$1")"
|
||||||
|
[ "$code" = 201 ] || [ "$code" = 204 ] || die "PUT $1 failed: HTTP ${code}"
|
||||||
|
echo " stored $1 (${#2} bytes, value not shown)"
|
||||||
|
}
|
||||||
|
echo "==> storing repository Actions secrets"
|
||||||
|
put BUILDKIT_CA "$CA"
|
||||||
|
put BUILDKIT_CLIENT_CERT "$CRT"
|
||||||
|
put BUILDKIT_CLIENT_KEY "$KEY"
|
||||||
|
unset CA CRT KEY
|
||||||
|
|
||||||
|
# VERIFY BY READING BACK (names only — Gitea never returns values). curl -sS exits 0 on an HTTP
|
||||||
|
# error, which is how set-ci-token.sh once claimed a store that never happened.
|
||||||
|
names="$(api "$API" | jq -r '[.[]?.name] | join(" ")')"
|
||||||
|
for n in BUILDKIT_CA BUILDKIT_CLIENT_CERT BUILDKIT_CLIENT_KEY; do
|
||||||
|
grep -qw "$n" <<<"$names" || die "stored $n but it is NOT listed (${names}) — refusing to claim success"
|
||||||
|
done
|
||||||
|
echo "==> stored and VERIFIED present. Re-run the images workflow (workflow_dispatch) to use them."
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
# platform/buildkit — rootless BuildKit, the CI image builder
|
||||||
|
|
||||||
|
The builder behind [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). It
|
||||||
|
builds `toolbox`, `agent-runner` and `conformance-runner`, and on `main` pushes them to the
|
||||||
|
Gitea registry. Flux reconciles it as the Kustomization `buildkit`
|
||||||
|
([`clusters/base/buildkit.yaml`](../../clusters/base/buildkit.yaml), `dependsOn: platform`).
|
||||||
|
|
||||||
|
## Why it exists (operator decision on PR #81, 2026-10-29)
|
||||||
|
|
||||||
|
CI jobs run as containers on the act runner's privileged DinD sidecar, and **cannot reach any
|
||||||
|
Docker daemon**. act_runner 0.2.11 forces `docker_host: "-"` for a `tcp://` DinD, and gitea/act
|
||||||
|
strips every bind mount when `valid_volumes` is empty. Measured on PR #81's first run. Giving jobs
|
||||||
|
that daemon would make every branch push root on the node. So the rule is: **no job may reach a
|
||||||
|
privileged daemon.** Builds go to this rootless daemon over mTLS instead.
|
||||||
|
|
||||||
|
## Shape
|
||||||
|
|
||||||
|
| Piece | What it is |
|
||||||
|
|---|---|
|
||||||
|
| `deployment.yaml` | `moby/buildkit:v0.34.1-rootless`, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The `securityContext` is commented line by line with how each item was measured |
|
||||||
|
| `certificates.yaml` | cert-manager: a self-signed CA used only here, plus the server cert `buildkitd-tls` and the CI client cert `buildkit-client-ci` |
|
||||||
|
| `service.yaml` | ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`, mTLS only. Never on the Gateway |
|
||||||
|
| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and the Gateway (`ingress` entity) for the registry push |
|
||||||
|
| `configmap.yaml` | `buildkitd.toml`: cache GC capped at 4 GB, because the cache is an `emptyDir` on node disk |
|
||||||
|
| `versions.env` | The CI side: pinned `buildx` (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource |
|
||||||
|
|
||||||
|
## Admission
|
||||||
|
|
||||||
|
**No Kyverno exception is needed.**
|
||||||
|
- Every ClusterPolicy that is actually applied (the list in
|
||||||
|
[`platform-config/kustomization.yaml`](../../platform-config/kustomization.yaml)) matches only
|
||||||
|
namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`. This namespace, like
|
||||||
|
`gitea-actions`, does not carry that label.
|
||||||
|
- The job-shape policies (`pm-agent-job-shape`, `conformance-trigger-job-shape`) match only their
|
||||||
|
own namespaces.
|
||||||
|
- There is no cluster-wide Pod Security default. This namespace is deliberately left without a
|
||||||
|
`pod-security` label, because PSS `baseline` forbids the `Unconfined` seccomp and AppArmor
|
||||||
|
profiles that rootless BuildKit needs.
|
||||||
|
|
||||||
|
## Prerequisites (operator, in order)
|
||||||
|
|
||||||
|
1. **Unprivileged user namespaces on the nodes.** Already true. Measured 2026-10-29 on all three
|
||||||
|
Debian 13.7 nodes (kernel 7.1.8): `kernel.unprivileged_userns_clone = 1`,
|
||||||
|
`user.max_user_namespaces = 127206`, and no Ubuntu-style AppArmor userns restriction.
|
||||||
|
2. **Merge.** Flux applies this directory. Check that the Kustomization `buildkit` is Ready and
|
||||||
|
that all three Certificates in namespace `buildkit` are READY.
|
||||||
|
3. **Copy the client certificate to CI.** Run
|
||||||
|
[`infra/gitea/buildkit-client-secret.sh`](../../infra/gitea/buildkit-client-secret.sh).
|
||||||
|
Re-run it yearly, after renewal; `--check` reports whether the copy is stale.
|
||||||
|
4. **Restart buildkitd after the server certificate renews.** It reads the certificate at start,
|
||||||
|
and cert-manager renews it 30 days before expiry, so a restart inside that window is needed.
|
||||||
|
|
||||||
|
Until step 3 is done, the `images` workflow fails at its builder preflight with a message naming
|
||||||
|
the cause. It never skips.
|
||||||
|
|
||||||
|
## What a build step can do
|
||||||
|
|
||||||
|
The daemon runs with `--oci-worker-no-process-sandbox`, so build steps share its PID namespace.
|
||||||
|
buildkitd warns about this itself at startup.
|
||||||
|
|
||||||
|
- **A build step CAN** signal or ptrace the daemon and read what it can read, including this
|
||||||
|
pod's TLS key and the build cache.
|
||||||
|
- **A build step CANNOT** reach the node. Everything runs as uid 1000 in a user namespace,
|
||||||
|
unprivileged, with no host mounts.
|
||||||
|
- **Who can run one:** anyone holding the client certificate, i.e. every CI job, including pull
|
||||||
|
requests.
|
||||||
|
|
||||||
|
## Local equivalent
|
||||||
|
|
||||||
|
The pinned image runs under rootless podman with the same restrictions:
|
||||||
|
`--cap-drop ALL --cap-add SETUID --cap-add SETGID --security-opt seccomp=unconfined --read-only`,
|
||||||
|
plus tmpfs at the same five paths. That is how the `securityContext` was measured. Rootless podman
|
||||||
|
maps only 65,537 uids, so the image's `/etc/subuid` (`user:100000:65536`) must be overridden with
|
||||||
|
two ranges that fit: `user:1:999` and `user:1001:64535`.
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
---
|
||||||
|
# mTLS for buildkitd, issued by cert-manager from a CA that exists ONLY for this purpose.
|
||||||
|
#
|
||||||
|
# buildkit-selfsigned (Issuer) --signs--> buildkit-ca (CA Certificate, 10y)
|
||||||
|
# buildkit-ca (Issuer) --signs--> buildkitd-tls (server, the daemon presents it)
|
||||||
|
# --signs--> buildkit-client-ci (client, CI presents it)
|
||||||
|
#
|
||||||
|
# buildkitd verifies EVERY client certificate against this CA (--tlscacert), so holding a client
|
||||||
|
# certificate signed by it is what "may build" means. Nothing else in the cluster trusts this CA,
|
||||||
|
# and it trusts nothing else.
|
||||||
|
#
|
||||||
|
# THE CLIENT CERTIFICATE REACHES CI AS ACTIONS SECRETS, not through the cluster: CI jobs hold no
|
||||||
|
# cluster credential (R-SEC-02). infra/gitea/buildkit-client-secret.sh copies ca.crt/tls.crt/
|
||||||
|
# tls.key from the Secret below into the repository secrets BUILDKIT_CA, BUILDKIT_CLIENT_CERT and
|
||||||
|
# BUILDKIT_CLIENT_KEY without printing them. Re-run it after every renewal (see renewBefore).
|
||||||
|
#
|
||||||
|
# ROTATION. cert-manager renews both leaves `renewBefore` their expiry and rewrites the Secrets.
|
||||||
|
# * server: buildkitd reads its certificate at START, so the pod must restart inside the 30-day
|
||||||
|
# renewal window (a node reboot by kured, a Deployment rollout, or a manual restart).
|
||||||
|
# * client: the Actions secrets are a COPY; re-run the script. Until then CI fails loudly at
|
||||||
|
# its TLS preflight (.gitea/workflows/scripts/images.sh), never silently.
|
||||||
|
apiVersion: cert-manager.io/v1
|
||||||
|
kind: Issuer
|
||||||
|
metadata:
|
||||||
|
name: buildkit-selfsigned
|
||||||
|
namespace: buildkit
|
||||||
|
spec:
|
||||||
|
selfSigned: {}
|
||||||
|
---
|
||||||
|
apiVersion: cert-manager.io/v1
|
||||||
|
kind: Certificate
|
||||||
|
metadata:
|
||||||
|
name: buildkit-ca
|
||||||
|
namespace: buildkit
|
||||||
|
spec:
|
||||||
|
isCA: true
|
||||||
|
commonName: buildkit-ca
|
||||||
|
secretName: buildkit-ca
|
||||||
|
duration: 87600h # 10 years: rotating the CA means re-issuing everything and re-copying
|
||||||
|
privateKey:
|
||||||
|
algorithm: ECDSA
|
||||||
|
size: 256
|
||||||
|
issuerRef:
|
||||||
|
name: buildkit-selfsigned
|
||||||
|
kind: Issuer
|
||||||
|
---
|
||||||
|
apiVersion: cert-manager.io/v1
|
||||||
|
kind: Issuer
|
||||||
|
metadata:
|
||||||
|
name: buildkit-ca
|
||||||
|
namespace: buildkit
|
||||||
|
spec:
|
||||||
|
ca:
|
||||||
|
secretName: buildkit-ca
|
||||||
|
---
|
||||||
|
apiVersion: cert-manager.io/v1
|
||||||
|
kind: Certificate
|
||||||
|
metadata:
|
||||||
|
name: buildkitd-tls
|
||||||
|
namespace: buildkit
|
||||||
|
spec:
|
||||||
|
secretName: buildkitd-tls
|
||||||
|
commonName: buildkitd.buildkit.svc.cluster.local
|
||||||
|
dnsNames:
|
||||||
|
- buildkitd.buildkit.svc.cluster.local # what CI dials (platform/buildkit/versions.env)
|
||||||
|
- buildkitd.buildkit.svc
|
||||||
|
- buildkitd
|
||||||
|
duration: 8760h
|
||||||
|
renewBefore: 720h
|
||||||
|
usages:
|
||||||
|
- server auth
|
||||||
|
- digital signature
|
||||||
|
privateKey:
|
||||||
|
algorithm: ECDSA
|
||||||
|
size: 256
|
||||||
|
rotationPolicy: Always
|
||||||
|
issuerRef:
|
||||||
|
name: buildkit-ca
|
||||||
|
kind: Issuer
|
||||||
|
---
|
||||||
|
apiVersion: cert-manager.io/v1
|
||||||
|
kind: Certificate
|
||||||
|
metadata:
|
||||||
|
name: buildkit-client-ci
|
||||||
|
namespace: buildkit
|
||||||
|
spec:
|
||||||
|
secretName: buildkit-client-ci
|
||||||
|
commonName: gitea-actions-image-pipeline
|
||||||
|
duration: 8760h
|
||||||
|
renewBefore: 720h
|
||||||
|
usages:
|
||||||
|
- client auth
|
||||||
|
- digital signature
|
||||||
|
privateKey:
|
||||||
|
algorithm: ECDSA
|
||||||
|
size: 256
|
||||||
|
rotationPolicy: Always
|
||||||
|
issuerRef:
|
||||||
|
name: buildkit-ca
|
||||||
|
kind: Issuer
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
---
|
||||||
|
# buildkitd.toml. Validated against the pinned image (v0.34.1-rootless): `buildctl debug workers
|
||||||
|
# -v` reports "Maximum used space: 4.29GB / Reserved space: 1.07GB" for the default policy.
|
||||||
|
#
|
||||||
|
# WHY 4GB. The cache lives in an emptyDir on the NODE's disk (deployment.yaml), and the nodes had
|
||||||
|
# 7-9 GB free when this was written. The largest image (toolbox) is ~1.4 GB uncompressed; the
|
||||||
|
# three images share their Fedora base. The emptyDir's sizeLimit sits above this cap, so GC
|
||||||
|
# trims the cache well before the kubelet would evict the pod for exceeding it.
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: buildkitd-config
|
||||||
|
namespace: buildkit
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: buildkitd
|
||||||
|
data:
|
||||||
|
buildkitd.toml: |
|
||||||
|
[worker.oci]
|
||||||
|
enabled = true
|
||||||
|
gc = true
|
||||||
|
maxUsedSpace = "4GB"
|
||||||
|
reservedSpace = "1GB"
|
||||||
|
[worker.containerd]
|
||||||
|
enabled = false
|
||||||
@@ -0,0 +1,142 @@
|
|||||||
|
---
|
||||||
|
# buildkitd, ROOTLESS. The builder behind .gitea/workflows/images.yaml; see README.md.
|
||||||
|
#
|
||||||
|
# THE SECURITY CONTEXT IS THE BOUNDARY, AND EVERY LINE OF IT WAS MEASURED (2026-10-29, the pinned
|
||||||
|
# image under podman with the same restrictions; decisions/2026-10-29-image-pipeline.md):
|
||||||
|
# * privileged: false — never. This is the whole point of the operator's decision on PR #81:
|
||||||
|
# the alternative was handing CI the privileged DinD daemon, i.e. root on the node.
|
||||||
|
# * runAsUser/Group 1000 — the image's `user`; rootlesskit maps it to uid 0 INSIDE a user
|
||||||
|
# namespace only. The nodes allow that (measured: kernel.unprivileged_userns_clone = 1,
|
||||||
|
# user.max_user_namespaces = 127206 on all three Debian 13 nodes).
|
||||||
|
# * capabilities: drop ALL, add SETUID + SETGID — the image's newuidmap/newgidmap are NOT
|
||||||
|
# setuid; they carry FILE capabilities, which need SETUID/SETGID in the bounding set.
|
||||||
|
# Measured: with drop ALL and nothing added, rootlesskit fails "newuidmap ... operation not
|
||||||
|
# permitted" and the daemon never starts.
|
||||||
|
# * allowPrivilegeEscalation: true — required for the same reason: no_new_privs (what `false`
|
||||||
|
# sets) stops a file capability from ever taking effect. It does NOT grant privileges the
|
||||||
|
# bounding set above does not contain.
|
||||||
|
# * seccompProfile: Unconfined — measured: with the default seccomp profile the daemon starts
|
||||||
|
# but every RUN step fails (the runtime cannot create the build step's namespaces/mounts).
|
||||||
|
# * appArmorProfile: Unconfined — upstream's documented requirement (buildkit
|
||||||
|
# examples/kubernetes, *.rootless.yaml); the nodes run AppArmor (enabled=Y) and containerd's
|
||||||
|
# default profile denies mount. NOT measured here: the test host runs SELinux, not AppArmor.
|
||||||
|
# * readOnlyRootFilesystem: true — measured to work with the five emptyDirs below.
|
||||||
|
#
|
||||||
|
# --oci-worker-no-process-sandbox: build steps share the daemon's PID namespace instead of getting
|
||||||
|
# their own (creating one needs an unmasked /proc, i.e. more privilege). Consequence, stated by
|
||||||
|
# buildkitd itself at startup: a build step can signal or ptrace the daemon and read what the
|
||||||
|
# daemon can read — including this pod's TLS key and the build cache. So a build step is trusted
|
||||||
|
# at the level of "may impersonate this builder", which is the level anyone holding the client
|
||||||
|
# certificate already has. It can NOT reach the node: everything still runs as uid 1000 in a user
|
||||||
|
# namespace, unprivileged, with no host mounts.
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: buildkitd
|
||||||
|
namespace: buildkit
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: buildkitd
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
strategy:
|
||||||
|
type: Recreate # one builder, one cache; never two daemons fighting over a node's disk
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app.kubernetes.io/name: buildkitd
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: buildkitd
|
||||||
|
spec:
|
||||||
|
automountServiceAccountToken: false # the builder needs no Kubernetes API at all
|
||||||
|
enableServiceLinks: false
|
||||||
|
securityContext:
|
||||||
|
runAsNonRoot: true
|
||||||
|
runAsUser: 1000
|
||||||
|
runAsGroup: 1000
|
||||||
|
fsGroup: 1000 # makes the 0440 TLS Secret below readable by uid 1000, and nobody else
|
||||||
|
containers:
|
||||||
|
- name: buildkitd
|
||||||
|
# Pinned by digest (the multi-arch index). The kubernetes manager in renovate.json5
|
||||||
|
# tracks platform/**, so Renovate proposes bumps; the CI client (buildx, pinned in
|
||||||
|
# versions.env) talks to any recent daemon.
|
||||||
|
image: docker.io/moby/buildkit:v0.34.1-rootless@sha256:f68752a5a94acfcdad2d07867b7075601a421638e17fc5f28efc1c814d0c4f89
|
||||||
|
args:
|
||||||
|
- --addr
|
||||||
|
- unix:///run/user/1000/buildkit/buildkitd.sock # local only: the probes below
|
||||||
|
- --addr
|
||||||
|
- tcp://0.0.0.0:1234 # CI, mTLS only
|
||||||
|
- --tlscacert
|
||||||
|
- /certs/ca.crt # client certs are VERIFIED against this CA: no cert, no build
|
||||||
|
- --tlscert
|
||||||
|
- /certs/tls.crt
|
||||||
|
- --tlskey
|
||||||
|
- /certs/tls.key
|
||||||
|
- --config
|
||||||
|
- /etc/buildkit/buildkitd.toml
|
||||||
|
- --oci-worker-no-process-sandbox
|
||||||
|
ports:
|
||||||
|
- name: buildkit
|
||||||
|
containerPort: 1234
|
||||||
|
protocol: TCP
|
||||||
|
readinessProbe:
|
||||||
|
exec:
|
||||||
|
command: ["buildctl", "debug", "workers"]
|
||||||
|
initialDelaySeconds: 5
|
||||||
|
periodSeconds: 30
|
||||||
|
livenessProbe:
|
||||||
|
exec:
|
||||||
|
command: ["buildctl", "debug", "workers"]
|
||||||
|
initialDelaySeconds: 30
|
||||||
|
periodSeconds: 60
|
||||||
|
timeoutSeconds: 10
|
||||||
|
securityContext:
|
||||||
|
privileged: false
|
||||||
|
runAsNonRoot: true
|
||||||
|
runAsUser: 1000
|
||||||
|
runAsGroup: 1000
|
||||||
|
allowPrivilegeEscalation: true
|
||||||
|
readOnlyRootFilesystem: true
|
||||||
|
capabilities:
|
||||||
|
drop: ["ALL"]
|
||||||
|
add: ["SETUID", "SETGID"]
|
||||||
|
seccompProfile:
|
||||||
|
type: Unconfined
|
||||||
|
appArmorProfile:
|
||||||
|
type: Unconfined
|
||||||
|
resources:
|
||||||
|
# Build steps (dnf, npm, pip) run INSIDE this container's cgroup, so the limit covers
|
||||||
|
# the daemon and the build together.
|
||||||
|
requests:
|
||||||
|
cpu: 250m
|
||||||
|
memory: 512Mi
|
||||||
|
ephemeral-storage: 2Gi
|
||||||
|
limits:
|
||||||
|
memory: 4Gi
|
||||||
|
volumeMounts:
|
||||||
|
- { name: certs, mountPath: /certs, readOnly: true }
|
||||||
|
- { name: config, mountPath: /etc/buildkit, readOnly: true }
|
||||||
|
- { name: cache, mountPath: /home/user/.local/share/buildkit }
|
||||||
|
- { name: run, mountPath: /run/user }
|
||||||
|
- { name: tmpdir, mountPath: /home/user/.local/tmp } # the image's TMPDIR
|
||||||
|
- { name: tmp, mountPath: /tmp }
|
||||||
|
- { name: clicfg, mountPath: /home/user/.docker } # buildctl (the probes) writes here
|
||||||
|
volumes:
|
||||||
|
- name: certs
|
||||||
|
secret:
|
||||||
|
secretName: buildkitd-tls # cert-manager: ca.crt + tls.crt + tls.key
|
||||||
|
defaultMode: 0440
|
||||||
|
- name: config
|
||||||
|
configMap:
|
||||||
|
name: buildkitd-config
|
||||||
|
- name: cache
|
||||||
|
emptyDir:
|
||||||
|
sizeLimit: 6Gi # GC caps use at 4GB (configmap.yaml); this is the backstop
|
||||||
|
- name: run
|
||||||
|
emptyDir: {}
|
||||||
|
- name: tmpdir
|
||||||
|
emptyDir: {}
|
||||||
|
- name: tmp
|
||||||
|
emptyDir: {}
|
||||||
|
- name: clicfg
|
||||||
|
emptyDir: {}
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
---
|
||||||
|
# platform/buildkit — rootless BuildKit, the CI image builder. Reconciled by the Flux
|
||||||
|
# Kustomization `buildkit` (clusters/base/buildkit.yaml). See README.md.
|
||||||
|
# versions.env is NOT a resource: it pins the CI-side client (buildx) and the address CI dials.
|
||||||
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||||
|
kind: Kustomization
|
||||||
|
resources:
|
||||||
|
- namespace.yaml
|
||||||
|
- certificates.yaml
|
||||||
|
- configmap.yaml
|
||||||
|
- deployment.yaml
|
||||||
|
- service.yaml
|
||||||
|
- networkpolicy.yaml
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
---
|
||||||
|
# buildkit — the image builder for CI (.gitea/workflows/images.yaml). Operator decision on PR #81
|
||||||
|
# (2026-10-29): images are built by a ROOTLESS BuildKit daemon reached over mTLS, and NO CI job
|
||||||
|
# may reach a privileged daemon. See README.md for the whole design.
|
||||||
|
#
|
||||||
|
# A NAMESPACE OF ITS OWN, deliberately NOT `gitea-actions`:
|
||||||
|
# * The act-runner namespace exists to hold a PRIVILEGED DinD sidecar. Putting the builder
|
||||||
|
# beside it would put a TLS server key next to a privileged container for no benefit.
|
||||||
|
# * Its NetworkPolicies, Secrets and quota then belong to one component, and the ingress rule
|
||||||
|
# ("only the act-runner pod may connect") can be written from the outside.
|
||||||
|
#
|
||||||
|
# NO `app-namespace` LABEL, like gitea-actions: this is CI plumbing, not a user-facing app. That
|
||||||
|
# is also why NO Kyverno exception is needed — every applied ClusterPolicy
|
||||||
|
# (platform-config/kustomization.yaml) matches only namespaces labelled
|
||||||
|
# homelab.keithjackson.dev/app-namespace: "true", and none of the job-shape policies match here.
|
||||||
|
#
|
||||||
|
# NO pod-security.kubernetes.io LABEL, on purpose. Rootless BuildKit needs seccomp AND AppArmor
|
||||||
|
# `Unconfined` (measured — see deployment.yaml), which PSS `baseline` forbids; there is no
|
||||||
|
# cluster-wide PSA default here, so an unlabelled namespace admits it. It is NOT privileged and
|
||||||
|
# the deployment's securityContext is the boundary — read it before relaxing anything.
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Namespace
|
||||||
|
metadata:
|
||||||
|
name: buildkit
|
||||||
|
labels:
|
||||||
|
homelab.keithjackson.dev/ci: buildkit
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
---
|
||||||
|
# Default-deny for the namespace, then two narrow allows.
|
||||||
|
#
|
||||||
|
# WHO MAY CONNECT — AND WHY THE ACT-RUNNER POD IS THE RIGHT SOURCE (measured 2026-10-29).
|
||||||
|
# CI jobs do not run as pods. act_runner starts each job as a container on its DinD sidecar, on a
|
||||||
|
# docker bridge network INSIDE the act-runner pod's network namespace, and dockerd masquerades it:
|
||||||
|
# kubectl -n gitea-actions exec deploy/act-runner -c dind -- iptables -t nat -S POSTROUTING
|
||||||
|
# -A POSTROUTING -s 172.18.0.0/16 ! -o br-c0f13d041ad6 -j MASQUERADE
|
||||||
|
# -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
|
||||||
|
# So a job's connection leaves with the act-runner POD's IP and therefore Cilium's identity for
|
||||||
|
# that pod (namespace gitea-actions, app.kubernetes.io/name=act-runner). That is exactly what the
|
||||||
|
# ingress rule names. It cannot distinguish one job from another, or a job from act_runner
|
||||||
|
# itself — per-job authentication is the mTLS client certificate, not this policy. What the
|
||||||
|
# policy adds is that NOTHING ELSE in the cluster can even open the port.
|
||||||
|
apiVersion: networking.k8s.io/v1
|
||||||
|
kind: NetworkPolicy
|
||||||
|
metadata:
|
||||||
|
name: default-deny
|
||||||
|
namespace: buildkit
|
||||||
|
labels:
|
||||||
|
homelab.keithjackson.dev/baseline: default-deny
|
||||||
|
spec:
|
||||||
|
podSelector: {}
|
||||||
|
policyTypes:
|
||||||
|
- Ingress
|
||||||
|
- Egress
|
||||||
|
ingress: []
|
||||||
|
egress:
|
||||||
|
- to:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
kubernetes.io/metadata.name: kube-system
|
||||||
|
ports:
|
||||||
|
- protocol: UDP
|
||||||
|
port: 53
|
||||||
|
- protocol: TCP
|
||||||
|
port: 53
|
||||||
|
---
|
||||||
|
apiVersion: cilium.io/v2
|
||||||
|
kind: CiliumNetworkPolicy
|
||||||
|
metadata:
|
||||||
|
name: buildkitd-allow-ci-ingress
|
||||||
|
namespace: buildkit
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: buildkitd
|
||||||
|
spec:
|
||||||
|
endpointSelector:
|
||||||
|
matchLabels:
|
||||||
|
app.kubernetes.io/name: buildkitd
|
||||||
|
ingress:
|
||||||
|
- fromEndpoints:
|
||||||
|
- matchLabels:
|
||||||
|
io.kubernetes.pod.namespace: gitea-actions
|
||||||
|
app.kubernetes.io/name: act-runner
|
||||||
|
toPorts:
|
||||||
|
- ports:
|
||||||
|
- { port: "1234", protocol: TCP }
|
||||||
|
---
|
||||||
|
# WHAT A BUILD NEEDS TO REACH. Build steps download (dnf mirrors, GitHub releases, dl.k8s.io,
|
||||||
|
# npm, PyPI, Fedora/Docker registries) and the result is pushed to the Gitea registry.
|
||||||
|
# * world 443/80: the downloads. 80 because Fedora's mirror list includes plain-http mirrors
|
||||||
|
# (dnf verifies package GPG signatures, not transport).
|
||||||
|
# * ingress (the Gateway's Envoy, reserved:ingress): the push. gitea.home.keithjackson.dev
|
||||||
|
# resolves in-cluster to the Gateway VIP (split-horizon CoreDNS), and Cilium hands that
|
||||||
|
# traffic to the Gateway, whose identity is `ingress` — the same identity gitea's own
|
||||||
|
# ingress allow admits.
|
||||||
|
# Nothing else in the cluster: not the API server, not other pods. A build step is code from a
|
||||||
|
# pull request.
|
||||||
|
apiVersion: cilium.io/v2
|
||||||
|
kind: CiliumNetworkPolicy
|
||||||
|
metadata:
|
||||||
|
name: buildkitd-allow-egress
|
||||||
|
namespace: buildkit
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: buildkitd
|
||||||
|
spec:
|
||||||
|
endpointSelector:
|
||||||
|
matchLabels:
|
||||||
|
app.kubernetes.io/name: buildkitd
|
||||||
|
egress:
|
||||||
|
- toEntities:
|
||||||
|
- world
|
||||||
|
toPorts:
|
||||||
|
- ports:
|
||||||
|
- { port: "443", protocol: TCP }
|
||||||
|
- { port: "80", protocol: TCP }
|
||||||
|
- toEntities:
|
||||||
|
- ingress
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
---
|
||||||
|
# The address CI dials: tcp://buildkitd.buildkit.svc.cluster.local:1234 (platform/buildkit/
|
||||||
|
# versions.env), mTLS only. ClusterIP — never exposed on the Gateway.
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: buildkitd
|
||||||
|
namespace: buildkit
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: buildkitd
|
||||||
|
spec:
|
||||||
|
type: ClusterIP
|
||||||
|
selector:
|
||||||
|
app.kubernetes.io/name: buildkitd
|
||||||
|
ports:
|
||||||
|
- name: buildkit
|
||||||
|
port: 1234
|
||||||
|
targetPort: buildkit
|
||||||
|
protocol: TCP
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
# platform/buildkit/versions.env — the CI side of the builder. Sourced by
|
||||||
|
# .gitea/workflows/scripts/images.sh. NOT a Kubernetes resource (kustomization.yaml omits it).
|
||||||
|
#
|
||||||
|
# The CLIENT. CI drives the daemon with `buildx` (remote driver) rather than bare `buildctl`
|
||||||
|
# because the image pipeline needs `bake`'s `target:` contexts: on a pull request agent-runner is
|
||||||
|
# built FROM the toolbox built in the same run, which never exists in any registry.
|
||||||
|
# Pinned with the sha256 from the release's own checksums.txt
|
||||||
|
# (https://github.com/docker/buildx/releases/download/v0.38.1/checksums.txt); the pipeline
|
||||||
|
# refuses a download that does not match. Bump BOTH lines together.
|
||||||
|
BUILDX_VERSION=0.38.1
|
||||||
|
BUILDX_SHA256=e25d5cd675092bb5d1d03632fdc46cdf3fede7b514dc5f023a105e00d3ce5ad4
|
||||||
|
|
||||||
|
# The DAEMON's address and the name its certificate carries (platform/buildkit/certificates.yaml).
|
||||||
|
BUILDKIT_HOST=tcp://buildkitd.buildkit.svc.cluster.local:1234
|
||||||
|
BUILDKIT_SERVERNAME=buildkitd.buildkit.svc.cluster.local
|
||||||
@@ -47,9 +47,9 @@ Both are deliberate, and both fail **visibly** rather than silently:
|
|||||||
|
|
||||||
1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
|
1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
|
||||||
`PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder until the first
|
`PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder until the first
|
||||||
publish. `.gitea/workflows/images.yaml` builds `agent-runner` with
|
publish. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
|
||||||
`infra/agent/build-image.sh` (the same script, and the same in-image verification, you can
|
(`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that
|
||||||
run locally), and on a push to `main` publishes it to
|
`infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to
|
||||||
`gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
|
`gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
|
||||||
Renovate then follows the `main` tag through the `# renovate-digest:` marker above **both**
|
Renovate then follows the `main` tag through the `# renovate-digest:` marker above **both**
|
||||||
lines and moves them together in one pull request. Nobody builds-and-pushes this image from
|
lines and moves them together in one pull request. Nobody builds-and-pushes this image from
|
||||||
|
|||||||
Reference in new issue
Block a user