Compare commits

..
Author SHA1 Message Date
jakman85andClaude Opus 5.5 d2c4bd5194 fix(buildkit): allow buildkitd egress to Gitea's backend so the Gateway stops 403ing the push
gate / gate (pull_request) Successful in 1m56s
gate / conformance (pull_request) Has been skipped
Images run 337's push failed with "403 Forbidden: Access denied". Measured
read-only: Gitea never logged the request; an unauthenticated GET /v2/ via the
Gateway gets 401 from Gitea when sent from act-runner (egress unrestricted) but
403 from buildkitd; Hubble logs it as an L7 http-request DROPPED with
traffic_direction EGRESS. Cilium >= 1.16.8 (CVE-2025-30162) re-checks the
source pod's egress against the proxied backend identity, and buildkitd's
egress only allowed world and the ingress entity. Same fix as 68690e5 for
Keycloak. Gitea's ingress already admits reserved:ingress and is unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 00:38:54 -05:00
jakman85 0dd57fbaf0 Merge pull request 'feat(images): build, verify and publish the three images in CI on rootless BuildKit; Renovate carries the digests' (#81) from feat/image-pipeline into main
gate / gate (push) Successful in 1m57s
images / images (push) Failing after 5s
gate / conformance (push) Has been skipped
Reviewed-on: #81
Reviewed-by: Keith Jackson <[email protected]>
2026-10-30 05:15:16 +00:00
jakman85andClaude Opus 5.5 04eb38584a fix(conformance): workload-mTLS certificates may use a private CA, if they prove they are workload-internal
gate / gate (pull_request) Successful in 1m58s
images / images (pull_request) Failing after 7s
gate / conformance (pull_request) Has been skipped
R-TLS-01/docs/08 govern externally reachable TLS and ADR-0006 puts workload mTLS out of scope, but
tls-lifecycle-automated required ACME for every Certificate, which platform/buildkit's internal CA
cannot satisfy. Clause (5b) accepts the tls-scope=workload-mtls annotation only when the
certificate proves it: no Gateway serves it, internal names only, namespaced selfSigned/ca issuer
backed by an exempt CA certificate. Also fixes yamllint comma spacing in the buildkitd mounts.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5 f65c12b395 feat(images): build on rootless BuildKit over mTLS; verification moves into Containerfile verify stages
Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched.

- platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by
  digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined,
  read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP
  :1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it).
  No Kyverno exception: every applied policy matches app-namespace=true only.
- Each Containerfile names its shipped stage and ends in a `verify` stage holding every former
  `docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for
  the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now.
- images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify
  targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes.
  Fails loudly when the builder secrets are absent or buildkitd is unreachable.
- infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5 8f35234fcc fix(images): name the measured runner blocker; the job container gets no Docker daemon
PR #81's first images run failed in 6s: act_runner 0.2.11 forces docker_host '-' for a tcp://
DOCKER_HOST, and gitea/act drops every bind when valid_volumes is empty. Giving jobs the
privileged DinD daemon is node-root for any workflow author, so it is left to the operator;
options are in the handoff entry. The driver's failure now names the cause.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5 e539109830 feat(images): build, verify and publish the three images in CI; Renovate carries the digests
Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a
workstation build/push. Nothing had ever been pushed to the Gitea registry.

- .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main
  builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as
  sha-<commit> + main, reading the digest back from the registry. Only changed images build.
- Each image's own build-image.sh does the build and in-image verification, now engine-neutral
  (DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh.
- Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex
  manager for them. The worker Job template is deliberately untracked (it would loop).
- renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the
  current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION.
- Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in
  Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85 27da5555a4 Merge pull request 'fix(renovate): the workflow still named renovate.json, so every run since #79 died with ENOENT' (#80) from fix/renovate-config-filename into main
gate / gate (push) Successful in 1m54s
gate / conformance (push) Has been skipped
Reviewed-on: #80
Reviewed-by: Keith Jackson <[email protected]>
2026-10-30 04:48:42 +00:00
jakman85andClaude Opus 5.5 da1b36eb64 fix(renovate): pin renovate-version — the action's default (39) rejects renovate.json5
gate / gate (pull_request) Successful in 1m55s
gate / conformance (pull_request) Has been skipped
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 21:49:38 -05:00
jakman85andClaude Opus 5.5 aacf361bb8 fix(renovate): the workflow still named renovate.json, so every run since #79 died with ENOENT
gate / gate (pull_request) Failing after 1m58s
gate / conformance (pull_request) Has been skipped
#79 renamed the config to renovate.json5 and left configurationFile pointing at the old
name. Runs 315-322 (2026-10-22..29) all failed in ~1s before Renovate started.

Also add a ci/lint.sh assertion that every workflow configurationFile: resolves to a real
path — valid YAML, so yamllint could not catch it, and the gate stayed green throughout.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 20:58:15 -05:00
33 changed files with 1775 additions and 614 deletions

No files matched your search

+18 -6
View File
@@ -120,10 +120,20 @@ CI. That is a real coverage gap, not a phase.
This repo's three images — `toolbox` (`infra/devcontainer/`), `agent-runner` (`infra/agent/`,
built `FROM` the toolbox) and `conformance-runner` (`conformance/incluster/`) — are built,
verified and published by [`images.yaml`](images.yaml), never from a workstation
(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh); each
image is built by **its own** `build-image.sh`, the same script a developer runs locally, so the
in-image verification (non-root uid, read-only rootfs, tool presence, `PM_SELFTEST=1
pm-dispatch`, the `conformance-clone` refusals) exists once.
(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh).
**The builder is rootless BuildKit** ([`platform/buildkit/`](../../platform/buildkit/README.md)),
reached over mTLS with a pinned `buildx` — no job here touches a Docker daemon, privileged or
otherwise (operator decision on PR #81). The client certificate reaches jobs as the Actions
secrets `BUILDKIT_CA`, `BUILDKIT_CLIENT_CERT`, `BUILDKIT_CLIENT_KEY`.
**Verification is a build target.** Each Containerfile ends in a `verify` stage that runs every
in-image assertion at build time as the non-root uid with `RUN --network=none` (tool presence and
pinned versions, `PM_SELFTEST=1 pm-dispatch`, the pm-worker probe, the `conformance-clone`
refusals, no writable path outside the workspace). The pipeline builds `<image>-verify` before it
builds, let alone pushes, the image. Build args come from each image's own
`build-image.sh --print-build-args`, which is also the local podman path — so the recipe and the
proof exist once.
| Trigger | What happens |
|---|---|
@@ -143,8 +153,10 @@ account with package write on the org). The automatic Actions token cannot push
[`infra/gitea/registry-retention.sh`](../../infra/gitea/registry-retention.sh), an operator
action, keeps every digest git pins plus the newest tags.
Cost on this runner: the job shares the single act_runner slot with the gate, and the DinD store
is a node `emptyDir`; the job prunes it on exit.
Cost: the job shares the single act_runner slot with the gate, but the build itself runs in the
`buildkitd` pod, not in DinD, so the runner's disk and memory are untouched. Until
`platform/buildkit` is deployed and its client certificate copied
(`infra/gitea/buildkit-client-secret.sh`), the job fails loudly at its builder preflight.
## Files
+28 -26
View File
@@ -41,21 +41,22 @@
# not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER`
# is optional and only labels the login.
#
# ┌─ BLOCKER, MEASURED ON PR #81 (run 325): THIS RUNNER CANNOT BUILD IMAGES YET ───────────────┐
# │ The job container gets no Docker daemon. act_runner 0.2.11 forces container.docker_host to │
# │ "-" whenever DOCKER_HOST is tcp:// (internal/app/cmd/daemon.go), and with valid_volumes: [] │
# │ gitea/act strips every bind including the socket (pkg/container/docker_run.go │
# │ sanitizeConfig). So this job fails at its first check, loudly, until the operator chooses │
# │ how workflow code may reach a daemon — see decisions/2026-10-29-image-pipeline.md. It is │
# │ not a required check, so it does not block merges; it is red, which is the honest state. │
# └──────────────────────────────────────────────────────────────────────────────────────────────┘
# THE BUILDER — rootless BuildKit (operator decision on PR #81, 2026-10-29).
# The act runner's job containers get NO Docker daemon (act_runner 0.2.11 forces docker_host "-"
# for a tcp:// DinD, and gitea/act strips every bind when valid_volumes is empty — measured on
# this PR's first run), and handing them the privileged DinD daemon would be root on the node for
# anyone who can push a branch. So images are built by platform/buildkit/: a ROOTLESS buildkitd
# (not privileged; uid 1000 in a user namespace), reached over mTLS at
# buildkitd.buildkit.svc.cluster.local:1234 with a pinned `buildx` (remote driver). Its client
# certificate reaches jobs as three Actions secrets — BUILDKIT_CA, BUILDKIT_CLIENT_CERT,
# BUILDKIT_CLIENT_KEY — copied from cert-manager by infra/gitea/buildkit-client-secret.sh.
# Pull requests get the client certificate (they must build) but never a registry credential.
#
# UNTIL platform/buildkit IS DEPLOYED (Flux applies it only after merge) THIS JOB FAILS, LOUDLY,
# at its builder preflight. A skipped Gitea job reports `success`, so it is red on purpose.
#
# VERIFICATION is each Containerfile's `verify` stage, built before anything is pushed.
#
# THE RUNNER (apps/wave-1/gitea-actions.yaml): act_runner 0.2.11, one job at a time, Docker
# executor on a privileged `docker:27-dind` sidecar whose image store is an emptyDir on the node
# (no sizeLimit) and whose memory limit is 2Gi — shared by the job container AND every build
# container. The toolbox is ~1.4 GB uncompressed and agent-runner adds ~0.3 GB on top of it, so the
# cleanup step below is not optional: without it each run would leave ~3 GB on a node that had
# 7 GB free when this was written, and the next eviction would take the runner with it.
name: images
on:
@@ -71,6 +72,7 @@ on:
- conformance/incluster/build-image.sh
- .gitea/workflows/images.yaml
- .gitea/workflows/scripts/images.sh
- platform/buildkit/versions.env
push:
branches: [main]
paths:
@@ -83,6 +85,7 @@ on:
- conformance/incluster/build-image.sh
- .gitea/workflows/images.yaml
- .gitea/workflows/scripts/images.sh
- platform/buildkit/versions.env
workflow_dispatch:
inputs:
images:
@@ -115,6 +118,11 @@ jobs:
MODE: verify
IMAGES: auto
PR_BASE_REF: ${{ github.base_ref }}
# The BUILDER's client certificate (platform/buildkit). Not a registry credential:
# a pull request can build, never push.
BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
run: |
set -euo pipefail
BASE_SHA="$(git merge-base "origin/${PR_BASE_REF}" HEAD)"
@@ -127,6 +135,9 @@ jobs:
env:
MODE: verify
IMAGES: ${{ inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images }}
BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
run: ./.gitea/workflows/scripts/images.sh
# ── main: publish ───────────────────────────────────────────────────────────────────────
@@ -168,16 +179,7 @@ jobs:
BASE_SHA: ${{ github.event.before }}
REGISTRY_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
REGISTRY_USER: ${{ vars.REGISTRY_PUSH_USER }}
BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
run: ./.gitea/workflows/scripts/images.sh
# ── always: give the DinD store back ────────────────────────────────────────────────────
# The runner's image store is a node emptyDir with no sizeLimit (see header). Everything this
# job built or pulled is removed; `prune -a` cannot remove the job's own image, because the
# job container is still running on it.
- name: Clean the DinD image store
if: always()
run: |
docker logout gitea.home.keithjackson.dev >/dev/null 2>&1 || true
docker image prune -af >/dev/null || true
docker builder prune -af >/dev/null || true
docker system df || true
+9 -6
View File
@@ -154,13 +154,16 @@ jobs:
# Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers
# includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`.
# This is the one dependency in the repo whose updater is its own updater.
configurationFile: renovate.json
# MUST match the file's real name. #79 renamed it to renovate.json5 and left this line
# saying `renovate.json`; every scheduled run from 2026-10-18 then died in ~1s with
# ENOENT before Renovate started, and nothing in the gate noticed. ci/lint.sh now
# asserts that every `configurationFile:` here names a file that exists.
configurationFile: renovate.json5
token: ${{ secrets.RENOVATE_TOKEN }}
# PINNED 2026-10-29. Unset, this action version runs `ghcr.io/renovatebot/renovate:39`
# (its action.yml default), i.e. 39.264.0 — and renovate.json5 is written for, and
# validated against, a release that knows `managerFilePatterns` (>= 40.2.0). Under 39.x
# the config fails validation with five errors and the run stops at CONFIG_VALIDATION.
# Validate renovate.json5 with THIS tag; see its header for the command.
# PINNED 2026-10-29. Unset, this action version runs Renovate `39` (action.yml
# `renovate-version` default at v41.0.9), i.e. 39.264.0, and renovate.json5 uses
# `managerFilePatterns` (>= 40.2.0): under 39.x validation fails with five errors and
# the run stops at CONFIG_VALIDATION. #79 validated against `latest`, not this.
renovate-version: 44.167.0
env:
# Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
+196 -119
View File
@@ -4,35 +4,41 @@
#
# Builds, verifies and (on main) PUBLISHES the three images this repo ships:
#
# toolbox infra/devcontainer/ the pinned dev/gate toolchain
# agent-runner infra/agent/ FROM toolbox; the PM dispatcher + worker
# toolbox infra/devcontainer/ the pinned dev/gate toolchain
# agent-runner infra/agent/ FROM toolbox; the PM dispatcher + worker
# conformance-runner conformance/incluster/ the in-cluster L2/L3 Chainsaw runner
#
# Each image is built by ITS OWN build-image.sh — the same script a developer runs locally —
# so the in-image verification (non-root uid, read-only rootfs, tool presence, pm-dispatch
# selftest, conformance-clone refusals, ...) is written once and runs in both places. This
# file only decides WHAT to build, in which order, FROM which base, and where it goes.
# THE BUILDER (operator decision on PR #81): a ROOTLESS BuildKit daemon, platform/buildkit/,
# reached over mTLS. No CI job may reach a privileged daemon, so there is no `docker build` and no
# `docker run` here: this script drives buildkitd with a pinned `buildx` (remote driver) and
# nothing else. The client certificate arrives as Actions secrets (BUILDKIT_CA,
# BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY; infra/gitea/buildkit-client-secret.sh puts them
# there). If the daemon is unreachable this FAILS LOUDLY — it never skips.
#
# VERIFICATION IS A BUILD TARGET. Each Containerfile ends in a `verify` stage that runs every
# in-image assertion at build time (as the non-root uid, with RUN --network=none); this script
# builds `<image>-verify` and only then the image. Build args come from each image's own
# `build-image.sh --print-build-args`, the same list the local podman path uses.
#
# MODES (env MODE):
# verify build + verify, never push, holds NO credential. Pull requests.
# publish build + verify + push to ${REGISTRY}/${NAMESPACE}/<image> with tags
# verify build the verify targets, never push, hold no registry credential. Pull requests.
# publish verify, then build + push to ${REGISTRY}/${NAMESPACE}/<image> with tags
# `sha-<12 hex>` (immutable: never re-pushed) and `main` (moving; Renovate follows
# it). The digest recorded is the one the REGISTRY returns, read back after the push.
#
# WHAT GETS BUILT (env IMAGES): `auto` (default) diffs BASE_SHA..HEAD_SHA against each image's
# inputs; otherwise a space-separated list. A change to the pipeline itself (images.yaml, this
# file) builds everything in verify mode, and nothing extra in publish mode: re-publishing
# unchanged inputs would only mint new digests and churn Renovate PRs.
# file, platform/buildkit/versions.env) builds everything in verify mode, and nothing extra in
# publish mode: re-publishing unchanged inputs would only mint new digests and churn Renovate PRs.
#
# THE agent-runner BASE — the one ordering rule, and the reason this is not three jobs:
# * toolbox built in this run -> agent-runner builds FROM it (in publish mode, FROM the
# digest the registry just returned for it).
# * verify mode, toolbox not changed -> the toolbox is built anyway as a base. A pull request
# holds no registry credential, so it cannot pull the published one.
# * publish mode, toolbox not changed -> AGENT_BASE_IMAGE from infra/agent/versions.env,
# i.e. the published toolbox digest that Renovate keeps current. If that is still the
# all-zero placeholder the toolbox has never been published, so it is built and published
# first (this is how the very first run bootstraps).
# THE agent-runner BASE:
# * verify mode -> FROM the toolbox built in this run, linked with a bake `target:` context —
# a pull request holds no registry credential and that toolbox exists in no registry. If the
# toolbox's inputs did not change it is built anyway, as the base.
# * publish mode, toolbox published in this run -> FROM the digest the registry just returned.
# * publish mode otherwise -> AGENT_BASE_IMAGE from infra/agent/versions.env (the published
# toolbox digest Renovate keeps current). If that is still the all-zero placeholder the
# toolbox has never been published, so it is built and published first (the bootstrap).
#
# LOOP-FREEDOM. Renovate turns each publish into a digest-bump PR. Those bumps land in
# platform/pm-agent/ (no image input), conformance/versions.env (no image input) and
@@ -46,9 +52,14 @@ set -euo pipefail
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
cd "${REPO_ROOT}"
# shellcheck disable=SC1091
BK_HOST_OVERRIDE="${BUILDKIT_HOST:-}"; BK_NAME_OVERRIDE="${BUILDKIT_SERVERNAME:-}"
. ./platform/buildkit/versions.env
BUILDKIT_HOST="${BK_HOST_OVERRIDE:-${BUILDKIT_HOST}}"
BUILDKIT_SERVERNAME="${BK_NAME_OVERRIDE:-${BUILDKIT_SERVERNAME}}"
MODE="${MODE:-verify}"
IMAGES="${IMAGES:-auto}"
ENGINE="${ENGINE:-docker}"
REGISTRY="${REGISTRY:-gitea.home.keithjackson.dev}"
NAMESPACE="${NAMESPACE:-home-network}"
HEAD_SHA="${HEAD_SHA:-$(git rev-parse HEAD)}"
@@ -63,13 +74,15 @@ MOVING_TAG="main"
log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
info() { printf ' %s\n' "$*"; }
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; printf -- '- **ERROR**: %s\n' "$*" >>"${SUMMARY}"; exit 1; }
yn() { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; }
case "${MODE}" in verify|publish) ;; *) die "MODE must be verify or publish, got '${MODE}'" ;; esac
for t in git jq curl sha256sum; do command -v "$t" >/dev/null || die "the job image lacks '$t'"; done
# --- inputs per image: a change under any of these rebuilds that image ---------------------
# conformance-runner takes its chainsaw/kubectl/fedora pins from infra/devcontainer/versions.env.
# conformance/versions.env is NOT an input (it only RECORDS the published digest), and must not
# become one: Renovate writes it after every publish, and an input there would loop.
# --- per-image facts ------------------------------------------------------------------------
# Inputs: a change under any of these rebuilds that image. conformance-runner takes its pins from
# infra/devcontainer/versions.env. conformance/versions.env is NOT an input (it only RECORDS the
# published digest), and must not become one: Renovate writes it after every publish -> a loop.
inputs() {
case "$1" in
toolbox) echo "infra/devcontainer/Containerfile infra/devcontainer/versions.env infra/devcontainer/build-image.sh" ;;
@@ -77,16 +90,17 @@ inputs() {
conformance-runner) echo "conformance/incluster/Containerfile conformance/incluster/conformance-clone conformance/incluster/build-image.sh infra/devcontainer/versions.env" ;;
esac
}
PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh"
context_of() { case "$1" in toolbox) echo infra/devcontainer ;; agent-runner) echo infra/agent ;; conformance-runner) echo conformance/incluster ;; esac; }
PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh platform/buildkit/versions.env"
declare -A WANT=() BUILT=() DIGEST=() SECS=() NOTE=()
changed_any() { # <paths...> -> 0 if BASE_SHA..HEAD_SHA touches any
# shellcheck disable=SC2086
[ -n "$(git diff --name-only "${BASE_SHA}" "${HEAD_SHA}" -- "$@")" ]
}
select_images() {
local i
if [ "${IMAGES}" != "auto" ]; then
for i in ${IMAGES}; do
case " ${ALL_IMAGES[*]} " in *" ${i} "*) WANT[$i]=1; NOTE[$i]="requested" ;; *) die "unknown image '${i}'" ;; esac
@@ -99,7 +113,6 @@ select_images() {
return
fi
info "diff base ${BASE_SHA} .. ${HEAD_SHA}"
local i
for i in "${ALL_IMAGES[@]}"; do
# shellcheck disable=SC2046
if changed_any $(inputs "$i"); then WANT[$i]=1; NOTE[$i]="inputs changed"; fi
@@ -110,6 +123,114 @@ select_images() {
fi
}
# --- the builder: pinned buildx + mTLS to rootless buildkitd ---------------------------------
WORK="$(mktemp -d)"
trap 'rm -rf "${WORK}"' EXIT # client key + registry auth live only here, only this long
export BUILDX_CONFIG="${WORK}/buildx" DOCKER_CONFIG="${WORK}/docker"
# Single-manifest images: the admission policies, Renovate and registry-retention.sh all reason
# about one manifest per tag. (Provenance attestations would turn every push into an index.)
export BUILDX_NO_DEFAULT_ATTESTATIONS=1
mkdir -p "${BUILDX_CONFIG}" "${DOCKER_CONFIG}" "${WORK}/certs"
echo '{}' > "${DOCKER_CONFIG}/config.json"
setup_builder() {
log "Builder: rootless buildkitd at ${BUILDKIT_HOST} (mTLS)"
BUILDX="${BUILDX_BIN:-}"
if [ -z "${BUILDX}" ]; then
BUILDX="${WORK}/buildx-bin"
curl -fsSL --retry 3 -o "${BUILDX}" \
"https://github.com/docker/buildx/releases/download/v${BUILDX_VERSION}/buildx-v${BUILDX_VERSION}.linux-amd64" \
|| die "could not download buildx v${BUILDX_VERSION}"
echo "${BUILDX_SHA256} ${BUILDX}" | sha256sum -c --quiet - \
|| die "buildx v${BUILDX_VERSION} does not match the sha256 pinned in platform/buildkit/versions.env — refusing to run it"
chmod +x "${BUILDX}"
fi
info "client: $("${BUILDX}" version)"
local missing=()
[ -n "${BUILDKIT_CA:-}" ] || missing+=(BUILDKIT_CA)
[ -n "${BUILDKIT_CLIENT_CERT:-}" ] || missing+=(BUILDKIT_CLIENT_CERT)
[ -n "${BUILDKIT_CLIENT_KEY:-}" ] || missing+=(BUILDKIT_CLIENT_KEY)
[ "${#missing[@]}" -eq 0 ] || die "the builder's client certificate is missing: ${missing[*]} not set.
These Actions secrets are copied from the cert-manager Secret buildkit/buildkit-client-ci by
infra/gitea/buildkit-client-secret.sh (an operator action, after platform/buildkit is
deployed). Without them nothing can be built — this fails rather than skipping ON PURPOSE."
umask 077
printf '%s\n' "${BUILDKIT_CA}" > "${WORK}/certs/ca.pem"
printf '%s\n' "${BUILDKIT_CLIENT_CERT}" > "${WORK}/certs/cert.pem"
printf '%s\n' "${BUILDKIT_CLIENT_KEY}" > "${WORK}/certs/key.pem"
umask 022
"${BUILDX}" create --name ci-remote --driver remote \
--driver-opt "cacert=${WORK}/certs/ca.pem,cert=${WORK}/certs/cert.pem,key=${WORK}/certs/key.pem,servername=${BUILDKIT_SERVERNAME}" \
"${BUILDKIT_HOST}" >/dev/null
if ! out="$(timeout 90 "${BUILDX}" inspect --bootstrap --builder ci-remote 2>&1)"; then
printf '%s\n' "${out}" >&2
die "buildkitd is UNREACHABLE at ${BUILDKIT_HOST} — nothing can be built, and this job fails
rather than skipping. In order of likelihood:
1. platform/buildkit is not deployed yet. Flux applies it only after it merges to main
(Kustomization 'buildkit'); a pull request that ADDS it cannot reach it.
2. the BUILDKIT_* secrets are stale (the client cert was renewed) or from another CA:
re-run infra/gitea/buildkit-client-secret.sh.
3. the NetworkPolicy (platform/buildkit/networkpolicy.yaml) does not admit this runner."
fi
printf '%s\n' "${out}" | grep -E '^(Name|Driver|Status|BuildKit version|Platforms):' | sed 's/^/ /' || true
}
# KEY=VALUE lines from an image's build-image.sh --print-build-args -> a JSON object.
build_args_json() {
local ctx; ctx="$(context_of "$1")"
"./${ctx}/build-image.sh" --print-build-args | jq -Rn '[inputs | select(length>0) | capture("^(?<k>[^=]+)=(?<v>.*)$")] | map({(.k): .v}) | add'
}
# bake_run <bakefile> <targets...>
bake_run() {
local f="$1"; shift
"${BUILDX}" bake --builder ci-remote --progress plain -f "$f" \
--metadata-file "${WORK}/meta.json" "$@"
}
# Write a bake file with a verify target and a ship target for one image.
# $1 image, $2 agent base mode: "" | "link" | <image ref>
write_bake() {
local img="$1" base="${2:-}" ctx args f
ctx="$(context_of "$img")"
args="$(build_args_json "$img")"
f="${WORK}/bake-${img}.json"
local contexts='{}'
if [ "${img}" = agent-runner ]; then
if [ "${base}" = link ]; then
# FROM ${AGENT_BASE_IMAGE} resolves to the named context below, i.e. the toolbox target
# of this same bake file — built here, never pushed anywhere.
args="$(jq -c '. + {AGENT_BASE_IMAGE: "toolbox-in-this-run"}' <<<"${args}")"
contexts='{"toolbox-in-this-run": "target:toolbox"}'
else
args="$(jq -c --arg b "${base}" '. + {AGENT_BASE_IMAGE: $b}' <<<"${args}")"
fi
fi
jq -n --arg img "$img" --arg ctx "$ctx" --argjson args "$args" --argjson contexts "$contexts" \
--arg tbargs "$( [ "${base}" = link ] && build_args_json toolbox || echo '{}')" \
--arg rev "${HEAD_SHA}" --arg repo "${REGISTRY}/${NAMESPACE}/${img}" \
--arg sha "${SHA_TAG}" --arg moving "${MOVING_TAG}" --arg mode "${MODE}" '
{target: (
{
($img + "-verify"): {context: $ctx, dockerfile: "Containerfile", target: "verify",
args: $args, contexts: $contexts, output: ["type=cacheonly"]},
($img): ({context: $ctx, dockerfile: "Containerfile", target: $img, args: $args,
contexts: $contexts,
labels: {"org.opencontainers.image.revision": $rev}}
+ (if $mode == "publish"
then {tags: [($repo + ":" + $sha), ($repo + ":" + $moving)], output: ["type=image,push=true"]}
else {output: ["type=cacheonly"]} end))
}
+ (if ($contexts | has("toolbox-in-this-run"))
then {toolbox: {context: "infra/devcontainer", dockerfile: "Containerfile", target: "toolbox",
args: ($tbargs | fromjson), output: ["type=cacheonly"]}}
else {} end)
)}' > "$f"
echo "$f"
}
# --- registry helpers (publish mode only) ----------------------------------------------------
REG_TOKEN=""
registry_token() {
@@ -120,8 +241,8 @@ registry_token() {
[ -n "${REG_TOKEN}" ] || die "registry returned no bearer token"
}
# Prints the manifest digest the REGISTRY serves for <image>:<tag>; nothing if it is absent (404).
# Any other answer (401, 5xx, no digest header) is fatal: "cannot tell" must never be read as
# "not published", or the immutable-tag guard would be bypassed by an auth hiccup.
# Any other answer is fatal: "cannot tell" must never be read as "not published", or the
# immutable-tag guard would be bypassed by an auth hiccup.
registry_digest() {
registry_token
local hdrs code digest
@@ -142,87 +263,50 @@ registry_digest() {
esac
}
local_ref() { echo "${REGISTRY}/${NAMESPACE}/$1:ci-${HEAD_SHA:0:12}"; }
build_one() { # <image> [agent base]
local img="$1" base="${2:-}" t0 tag
tag="ci-${HEAD_SHA:0:12}"
# build_one <image> [agent base] — verify, then (publish) build + push + read back.
build_one() {
local img="$1" base="${2:-}" t0 f pushed got existing
t0=$(date +%s)
log "BUILD + VERIFY ${img}"
case "${img}" in
toolbox)
ENGINE="${ENGINE}" IMAGE="$(local_ref toolbox)" BUILD_REVISION="${HEAD_SHA}" \
./infra/devcontainer/build-image.sh ;;
agent-runner)
ENGINE="${ENGINE}" IMAGE_TAG="${tag}" AGENT_BASE_IMAGE="${base}" BUILD_REVISION="${HEAD_SHA}" \
./infra/agent/build-image.sh ;;
conformance-runner)
ENGINE="${ENGINE}" IMAGE_TAG="${tag}" BUILD_REVISION="${HEAD_SHA}" \
./conformance/incluster/build-image.sh ;;
esac
f="$(write_bake "$img" "$base")"
log "VERIFY ${img} (Containerfile target: verify, on rootless BuildKit)"
bake_run "$f" "${img}-verify"
BUILT[$img]=1
SECS[$img]=$(( $(date +%s) - t0 ))
info "${img}: built and verified in ${SECS[$img]}s"
}
push_one() { # <image> -> sets DIGEST[image] to the registry-returned digest
local img="$1" repo src existing pushed got t0
repo="${REGISTRY}/${NAMESPACE}/${img}"
src="$(local_ref "${img}")"
t0=$(date +%s)
log "PUSH ${img} -> ${repo}:{${SHA_TAG},${MOVING_TAG}}"
existing="$(registry_digest "${img}" "${SHA_TAG}")"
if [ -n "${existing}" ]; then
if [ "${MODE}" = publish ]; then
existing="$(registry_digest "${img}" "${SHA_TAG}")"
# Immutable tag: never overwrite. A rebuild is not byte-reproducible, so re-pushing would
# silently change what `sha-<commit>` means after something may already have pinned it.
die "${repo}:${SHA_TAG} already exists (${existing}). Tags here are immutable; this run must not
replace it. (publish skips images that already exist at this commit BEFORE building them,
so reaching this line means the tag appeared mid-run.)"
[ -z "${existing}" ] || die "${REGISTRY}/${NAMESPACE}/${img}:${SHA_TAG} already exists (${existing}); tags are immutable"
log "BUILD + PUSH ${img} -> ${REGISTRY}/${NAMESPACE}/${img}:{${SHA_TAG},${MOVING_TAG}}"
bake_run "$f" "${img}"
pushed="$(jq -r --arg t "$img" '.[$t]["containerimage.digest"] // empty' "${WORK}/meta.json")"
# The AUTHORITATIVE digest is what the registry serves, read back — not what the client
# reported. They should agree; if they do not, nothing downstream may trust either.
got="$(registry_digest "${img}" "${SHA_TAG}")"
[ -n "${got}" ] || die "pushed ${img}:${SHA_TAG} but the registry does not serve it"
[ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] || die "${img}: client reported ${pushed} but the registry serves ${got}"
[ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \
|| die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})"
DIGEST[$img]="${got}"
info "${img}: registry digest ${got}"
fi
"${ENGINE}" tag "${src}" "${repo}:${SHA_TAG}"
"${ENGINE}" tag "${src}" "${repo}:${MOVING_TAG}"
pushed="$("${ENGINE}" push "${repo}:${SHA_TAG}" | tee /dev/stderr | grep -oE 'digest: sha256:[0-9a-f]{64}' | tail -1 | cut -d' ' -f2)"
"${ENGINE}" push "${repo}:${MOVING_TAG}" >/dev/null
# The AUTHORITATIVE digest is what the registry serves, read back — not what the client
# printed. They should agree; if they do not, nothing downstream may trust either.
got="$(registry_digest "${img}" "${SHA_TAG}")"
[ -n "${got}" ] || die "pushed ${repo}:${SHA_TAG} but the registry does not serve it"
[ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] \
|| die "${img}: client reported ${pushed} but the registry serves ${got}"
[ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \
|| die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})"
DIGEST[$img]="${got}"
SECS[$img]=$(( ${SECS[$img]:-0} + $(date +%s) - t0 ))
info "${img}: registry digest ${got}"
SECS[$img]=$(( $(date +%s) - t0 ))
info "${img}: done in ${SECS[$img]}s"
}
# --- main ---------------------------------------------------------------------------------
log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}"
command -v "${ENGINE}" >/dev/null || die "${ENGINE} is not on PATH in the job container"
"${ENGINE}" info >/dev/null 2>&1 \
|| die "the ${ENGINE} daemon is not reachable from the job container, so nothing can be built.
MEASURED ON THIS RUNNER (PR #81, 2026-10-29): act_runner 0.2.11 never gives a job a
daemon. Its DOCKER_HOST is tcp:// (the DinD sidecar), and internal/app/cmd/daemon.go then
forces container.docker_host to '-' (\"can't be mounted\"); separately, with
valid_volumes: [] gitea/act's sanitizeConfig drops EVERY bind, the socket included.
Fixing it means handing workflow code the privileged DinD daemon — an operator security
decision, set out in decisions/2026-10-29-image-pipeline.md. Not a fault in this script."
"${ENGINE}" version 2>/dev/null | sed -n '1,12p' || true
"${ENGINE}" system df || true
select_images
if [ "${MODE}" = publish ]; then
: "${REGISTRY_TOKEN:?REGISTRY_TOKEN is required in publish mode}"
REGISTRY_USER="${REGISTRY_USER:-ci-image-publisher}"
printf '%s' "${REGISTRY_TOKEN}" | "${ENGINE}" login "${REGISTRY}" -u "${REGISTRY_USER}" --password-stdin >/dev/null \
|| die "docker login ${REGISTRY} failed — check REGISTRY_PUSH_TOKEN (scope write:package, owner has package write on the org)"
fi
select_images
# Publish idempotence: an image already published at THIS commit is not rebuilt. This is what
# makes a re-run after a half-finished publish safe — and it is the only way a rerun can
# finish, because the immutable sha tag must never be pushed twice.
if [ "${MODE}" = publish ]; then
# buildx (and through it buildkitd, which does the push) reads registry auth from this file.
jq -n --arg r "${REGISTRY}" --arg a "$(printf '%s:%s' "${REGISTRY_USER}" "${REGISTRY_TOKEN}" | base64 -w0)" \
'{auths: {($r): {auth: $a}}}' > "${DOCKER_CONFIG}/config.json"
chmod 600 "${DOCKER_CONFIG}/config.json"
registry_token
# Publish idempotence: an image already published at THIS commit is not rebuilt. That is what
# makes a re-run after a half-finished publish safe — the immutable tag is never pushed twice.
for i in "${ALL_IMAGES[@]}"; do
[ -n "${WANT[$i]:-}" ] || continue
d="$(registry_digest "${i}" "${SHA_TAG}")"
@@ -236,11 +320,11 @@ fi
# Decide the agent-runner base (see header).
AGENT_BASE=""
if [ -n "${WANT[agent-runner]:-}" ]; then
# shellcheck disable=SC1091
PINNED_BASE="$(. ./infra/agent/versions.env; echo "${AGENT_BASE_IMAGE}")"
if [ "${MODE}" = verify ] && [ -z "${WANT[toolbox]:-}" ]; then
WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)"
elif [ "${MODE}" = publish ] && [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then
if [ "${MODE}" = verify ]; then
AGENT_BASE=link
[ -n "${WANT[toolbox]:-}" ] || { WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)"; }
elif [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then
case "${PINNED_BASE}" in
*@"${ZERO_DIGEST}") WANT[toolbox]=1; NOTE[toolbox]="never published; required as agent-runner's base" ;;
*@sha256:*) AGENT_BASE="${PINNED_BASE}" ;;
@@ -249,37 +333,30 @@ if [ -n "${WANT[agent-runner]:-}" ]; then
fi
fi
yn() { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; }
for i in "${ALL_IMAGES[@]}"; do info "plan: ${i}: $(yn "${WANT[$i]:-}" BUILD skip) (${NOTE[$i]:-inputs unchanged})"; done
if [ -n "${WANT[toolbox]:-}" ]; then
build_one toolbox
if [ "${MODE}" = publish ]; then push_one toolbox; fi
if [ -z "${WANT[toolbox]:-}${WANT[agent-runner]:-}${WANT[conformance-runner]:-}" ]; then
info "nothing to build"
else
setup_builder
fi
if [ -n "${WANT[toolbox]:-}" ]; then build_one toolbox; fi
if [ -n "${WANT[agent-runner]:-}" ]; then
if [ -z "${AGENT_BASE}" ]; then
if [ "${MODE}" = publish ]; then
AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}"
else
AGENT_BASE="$(local_ref toolbox)"
fi
AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}"
fi
info "agent-runner base: ${AGENT_BASE}"
info "agent-runner base: $( [ "${AGENT_BASE}" = link ] && echo "the toolbox built in this run (bake target: context)" || echo "${AGENT_BASE}")"
build_one agent-runner "${AGENT_BASE}"
if [ "${MODE}" = publish ]; then push_one agent-runner; fi
fi
if [ -n "${WANT[conformance-runner]:-}" ]; then
build_one conformance-runner
if [ "${MODE}" = publish ]; then push_one conformance-runner; fi
fi
if [ -n "${WANT[conformance-runner]:-}" ]; then build_one conformance-runner; fi
# --- summary ------------------------------------------------------------------------------
{
echo "## Images (${MODE})"
echo
echo "commit \`${HEAD_SHA}\`"
echo "commit \`${HEAD_SHA}\` — builder: rootless BuildKit \`${BUILDKIT_HOST}\` (mTLS)"
echo
echo "| image | built + verified | seconds | published digest (registry-returned) | note |"
echo "| image | verify target built | seconds | published digest (registry-returned) | note |"
echo "|---|---|---|---|---|"
for i in "${ALL_IMAGES[@]}"; do
d="${DIGEST[$i]:-}"
+15
View File
@@ -27,3 +27,18 @@ fi
yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${files[@]}"
ok "yaml/lint passed (${#files[@]} files)"
# A workflow's `configurationFile:` must name a file that exists. Renovate's did not after #79
# renamed renovate.json -> renovate.json5, and every scheduled run failed with ENOENT for 11 days
# while the gate stayed green. Valid YAML, so yamllint cannot see it.
missing=0
while IFS=: read -r wf _ value; do
value="${value#*configurationFile:}"
value="$(echo "$value" | sed -E 's/#.*//; s/^[[:space:]"'\'']+//; s/[[:space:]"'\'']+$//')"
if [ -n "$value" ] && [ ! -e "$value" ]; then
err "${wf}: configurationFile '${value}' does not exist in the repo"
missing=1
fi
done < <(grep -nE '^[[:space:]]*configurationFile:' .gitea/workflows/*.y*ml 2>/dev/null || true)
[ "$missing" -eq 0 ] || exit 1
ok "workflow configurationFile references resolve"
+21
View File
@@ -0,0 +1,21 @@
---
# Flux Kustomization -> ./platform/buildkit — rootless BuildKit, the image builder CI uses
# (.gitea/workflows/images.yaml). Its own Kustomization so a stuck Certificate or a failing
# builder can never stall any other tier (Flux health-checks Certificates under wait:true).
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: buildkit
namespace: flux-system
spec:
interval: 10m
retryInterval: 2m
timeout: 5m
prune: true
wait: true
dependsOn:
- name: platform # cert-manager CRDs (Issuer/Certificate) come from its HelmRelease
sourceRef:
kind: GitRepository
name: flux-system
path: ./platform/buildkit
+1
View File
@@ -98,4 +98,5 @@ resources:
# deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon.
# See clusters/base/kured.yaml and platform/kured/helmrelease.yaml.
- kured.yaml # Flux Kustomization -> ./platform/kured (dependsOn platform)
- buildkit.yaml # Flux Kustomization -> ./platform/buildkit (dependsOn platform)
- cluster-vars.yaml # overlay knob surface (base defaults)
+87 -1
View File
@@ -98,7 +98,10 @@ RUN set -eux; \
# --- runtime -----------------------------------------------------------------
ARG FEDORA_VERSION=44
FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION}
# Stage `conformance-runner` is the image that ships; stage `verify` (end of file) is a TEST stage
# built FROM it that only runs assertions and is never pushed. build-image.sh and the CI
# pipeline build `--target verify` to prove it and `--target conformance-runner` for the image.
FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION} AS conformance-runner
ARG CHAINSAW_VERSION=
ARG KUBECTL_VERSION=
@@ -198,3 +201,86 @@ WORKDIR /workspace
# prepend something the policy cannot see. `bash` is the interactive fallback for a human
# debugging the image locally.
CMD ["/bin/bash"]
# =============================================================================
# verify — a TEST stage, never shipped. Every assertion conformance/incluster/build-image.sh used
# to make with `run --user 65532 --read-only --tmpfs /workspace` now runs here, at build time,
# because CI builds on rootless BuildKit where no container runtime exists.
#
# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE:
# * uid 65532 ............ exact (USER inherited from the conformance-runner stage).
# * no network ........... exact: RUN --network=none, and the first check proves it.
# * --read-only rootfs ... NOT reproducible (a RUN step always has a writable layer).
# Approximated by permission: everything uid 65532 can write must lie
# under /workspace, /tmp or /var/tmp. In the Job only /workspace (and
# /results) are writable — the image points HOME and TMPDIR there — and
# readOnlyRootFilesystem is enforced by
# platform/kyverno/admission/conformance-trigger-job-shape.yaml.
# =============================================================================
FROM conformance-runner AS verify
RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
echo "--- no network (the build-time stand-in for the Job's isolation) ---"
if getent hosts gitea.home.keithjackson.dev >/dev/null 2>&1 || timeout 3 bash -c '</dev/tcp/1.1.1.1/443' 2>/dev/null; then
echo "UNEXPECTED: network reachable in the verify stage"; exit 1
fi
echo "confirmed: no network"
echo "--- id ---"; id
echo "--- HOME ---"; echo "HOME=${HOME}"
echo "--- bash ---"; bash --version | head -1
echo "--- python3 ---"; python3 --version
echo "--- git ---"; git --version
echo "--- kubectl ---"; kubectl version --client
echo "--- chainsaw ---"; chainsaw version
echo "--- baked versions ---"; cat /etc/conformance-runner-versions
echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
if [ -w /usr/local/bin/conformance-clone ]; then
echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
fi
echo "confirmed: not writable by uid $(id -u)"
# Each of these MUST fail. Assert the non-zero exit as well as the message: a check that only
# greps stderr would pass a build where the guard had stopped rejecting.
expect_fail() {
local label="$1"; shift
local out rc=0
out="$(env "$@" 2>&1)" || rc=$?
echo "--- ${label} ---"
printf "%s\n" "${out}" | head -2
if [ "${rc}" -eq 0 ]; then
echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
fi
echo "(rejected, exit ${rc})"
}
expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
conformance-clone --layer l2
expect_fail "conformance-clone rejects a missing SHA" \
conformance-clone
expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
CONFORMANCE_GIT_SHA=main conformance-clone
expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
expect_fail "conformance-clone rejects a HOME off the workspace volume" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
HOME=/etc conformance-clone
expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
expect_fail "conformance-clone refuses a clone target off the workspace volume" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
CONFORMANCE_SRC=/etc/src conformance-clone
echo "--- the root filesystem is not writable by the runner (expected: a failure) ---"
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi
echo "confirmed: / is not writable"
echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---"
stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \
| grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)"
if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf ' %s\n' ${stray} | head -20; exit 1; fi
echo "confirmed: no writable path outside /workspace, /tmp, /var/tmp"
echo "--- HOME is writable (the workspace emptyDir) ---"
touch "${HOME}/.probe" || { echo "UNEXPECTED: ${HOME} not writable"; exit 1; }
echo "confirmed: ${HOME} is writable"
echo "--- uid is non-zero ---"
[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; }
echo "confirmed: running as uid $(id -u), not root"
echo "CONFORMANCE-RUNNER VERIFIED"
VERIFY
+6 -4
View File
@@ -61,10 +61,12 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
> digest-pinned to the conformance-runner image, `command` exactly
> `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
> `env`/`envFrom` were the same shape of hole and are closed in the same policy.
- `build-image.sh` — builds the above and **verifies the toolchain by running it** inside the
image as uid 65532 with `--read-only`. It **never pushes**, and neither does anyone from a
workstation: CI (`.gitea/workflows/images.yaml`) runs this same script, and on a push to
`main` publishes the verified image under `sha-<commit>` and `main`. Renovate then moves
- `build-image.sh` — builds the above and **verifies the toolchain by running it**: the
Containerfile's `verify` stage executes every tool and every `conformance-clone` refusal as
uid 65532 with no network, at build time. It **never pushes**, and neither does anyone from a
workstation: CI (`.gitea/workflows/images.yaml`) builds the same Containerfile and `verify`
stage on rootless BuildKit (`platform/buildkit/`), and on a push to `main` publishes the
verified image under `sha-<commit>` and `main`. Renovate then moves
`CONFORMANCE_RUNNER_IMAGE_DIGEST` in `conformance/versions.env` onto the registry-returned
digest in a pull request (that file is a sensitive path, so the bump needs an
administrator's `/approve-override`).
+31 -83
View File
@@ -3,7 +3,8 @@
# conformance/incluster/build-image.sh — build the pinned `conformance-runner`
# image and prove it works. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
#
# ./build-image.sh # build + verify + print the image id
# ./build-image.sh # build the verify target, then the image
# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes)
# ./build-image.sh --no-verify # build only (skip the in-image tool proof)
# ./build-image.sh --verify-clone # ALSO really clone this repo with conformance-clone.
# # Opt-in because it needs the Gitea host reachable; the
@@ -13,13 +14,12 @@
# IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this)
#
# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
# Publishing is done by CI: .gitea/workflows/images.yaml calls this script, and on a push to
# `main` it pushes what this script built AND verified to the Gitea registry, records the
# registry-returned digest in the job summary, and Renovate then opens the pull request that
# moves conformance/versions.env onto that digest. So this script's job is to build and to
# prove the toolchain works INSIDE the image as the non-root uid the policy demands — the same
# proof locally and in CI. It runs under ENGINE=docker in CI (Docker-in-Docker), so nothing
# below may be podman-only.
# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on
# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and
# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest
# in the job summary; Renovate then opens the pull request that moves conformance/versions.env
# onto that digest. The proof that the image works is the Containerfile's `verify` stage, which
# this script (podman locally) and CI both build.
#
# VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
# single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
@@ -39,9 +39,11 @@ TAG_OVERRIDE="${IMAGE_TAG:-}"
VERIFY=1
VERIFY_CLONE=0
PRINT_ARGS=0
for arg in "$@"; do
case "$arg" in
--no-verify) VERIFY=0 ;;
--print-build-args) PRINT_ARGS=1 ;;
--verify-clone) VERIFY_CLONE=1 ;;
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown argument: $arg" >&2; exit 2 ;;
@@ -52,8 +54,6 @@ log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
info() { printf ' %s\n' "$*"; }
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
# --- pins: sourced, never invented -------------------------------------------
TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
[ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
@@ -63,6 +63,13 @@ source "${TOOLCHAIN_ENV}"
: "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
: "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
# THE build args, in ONE place; `--print-build-args` hands them to the CI pipeline, which builds
# this Containerfile on rootless BuildKit.
BUILD_ARGS=("CHAINSAW_VERSION=${CHAINSAW_VERSION}" "KUBECTL_VERSION=${KUBECTL_VERSION}" "FEDORA_VERSION=${FEDORA_VERSION}")
if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
# --- image name: fixed by the admission policy -------------------------------
# platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
@@ -84,86 +91,27 @@ info " FEDORA ${FEDORA_VERSION}"
LABEL_ARGS=()
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
"${ENGINE}" build "${LABEL_ARGS[@]}" \
--file "${HERE}/Containerfile" \
--tag "${IMAGE_REF}" \
--build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
--build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \
--build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \
"${HERE}"
ARG_FLAGS=()
for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
# --- verify: the point of the whole exercise ---------------------------------
# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the
# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root
# filesystem — the shape the admission policy forces. A pass here is the only evidence
# this image is usable; a build that merely succeeded is not.
# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So the proof is that
# the tools EXECUTE, as the NON-ROOT uid, with no network, that conformance-clone is not writable
# by that uid and refuses every hostile input, and that nothing outside /workspace is writable.
# That proof is the Containerfile's `verify` stage, built here and in CI alike (CI has no
# container runtime — rootless BuildKit only). See the stage's header for what a build step can
# and cannot reproduce of the Job's shape. A build that merely succeeded is not evidence.
if [ "${VERIFY}" -eq 1 ]; then
log "Verifying the toolchain inside the image (non-root, read-only rootfs)"
"${ENGINE}" run --rm \
--user 65532:65532 \
--read-only \
--tmpfs /workspace:rw,mode=1777 \
--entrypoint "" \
"${IMAGE_REF}" \
/bin/bash -euo pipefail -c '
echo "--- id ---"; id
echo "--- HOME ---"; echo "HOME=${HOME}"
echo "--- bash ---"; bash --version | head -1
echo "--- python3 ---"; python3 --version
echo "--- git ---"; git --version
echo "--- kubectl ---"; kubectl version --client
echo "--- chainsaw ---"; chainsaw version
echo "--- baked versions ---"; cat /etc/conformance-runner-versions
echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
if [ -w /usr/local/bin/conformance-clone ]; then
echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
fi
echo "confirmed: not writable by uid $(id -u)"
# Each of these MUST fail. Assert the non-zero exit as well as the message: a check
# that only greps stderr would pass a build where the guard had stopped rejecting.
# (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the
# expected failure — without it the verify dies at the first negative case.)
expect_fail() {
local label="$1"; shift
local out rc=0
out="$(env "$@" 2>&1)" || rc=$?
echo "--- ${label} ---"
printf "%s\n" "${out}" | head -2
if [ "${rc}" -eq 0 ]; then
echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
fi
echo "(rejected, exit ${rc})"
}
expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
conformance-clone --layer l2
expect_fail "conformance-clone rejects a missing SHA" \
conformance-clone
expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
CONFORMANCE_GIT_SHA=main conformance-clone
expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
expect_fail "conformance-clone rejects a HOME off the workspace volume" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
HOME=/etc conformance-clone
expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
expect_fail "conformance-clone refuses a clone target off the workspace volume" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
CONFORMANCE_SRC=/etc/src conformance-clone
echo "--- rootfs is read-only (expected: a failure) ---"
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
echo "confirmed: / is read-only"
echo "--- HOME is writable (the workspace emptyDir) ---"
touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable"
echo "--- uid is non-zero ---"
[ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root"
'
log "Verifying the toolchain (Containerfile target: verify)"
"${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}"
log "Toolchain verified"
fi
"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target conformance-runner \
--file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}"
# --- verify the clone helper against the REAL repository (opt-in: needs network) ----
# LOCAL ONLY: this one `run`s the image, which needs a container engine, so CI does not do it.
# The negative cases above prove conformance-clone refuses bad input. This proves the
# positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
# rev-parse assertion is only an unexercised line of shell.
@@ -250,8 +250,8 @@ check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text,
# The build recipe moved from dev.sh into build-image.sh on 2026-10-29, so that `dev.sh build`
# and the CI image pipeline (.gitea/workflows/images.yaml) share ONE recipe. Check the recipe.
BUILDSH = ROOT / "infra/devcontainer/build-image.sh"
check(BUILDSH.is_file() and "--build-arg PROMTOOL_VERSION=" in BUILDSH.read_text(),
"infra/devcontainer/build-image.sh does not pass --build-arg PROMTOOL_VERSION — the ARG "
check(BUILDSH.is_file() and '"PROMTOOL_VERSION=$PROMTOOL_VERSION"' in BUILDSH.read_text(),
"infra/devcontainer/build-image.sh does not pass PROMTOOL_VERSION in its BUILD_ARGS — the ARG "
"would be empty and the build would fetch a bogus URL")
INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh"
@@ -42,6 +42,19 @@
# whose every solver is an automated challenge type (dns01 / http01) — the
# same code path serves issuance AND renewal, so a solver a human must
# complete would defeat both;
# (5b) the ONE exception to (5) — WORKLOAD mTLS, added 2026-10-29 — is a Certificate
# annotated `homelab.keithjackson.dev/tls-scope: workload-mtls`. The spec's TLS
# requirements are about EXTERNALLY REACHABLE services (R-TLS-01; docs/08), and
# ADR-0006 says outright that "mTLS between workloads is a separate concern … not
# covered here". A public ACME CA cannot issue for a `*.svc.cluster.local` name or a
# client-auth certificate at all. The annotation is NOT trusted: an exempt
# Certificate must PROVE it is workload-internal — its Secret is served by NO Gateway
# listener, every DNS name / commonName is cluster-internal (a bare label or
# `*.svc[.cluster.local]`), it carries no IP/URI/email SANs, and its issuer is a
# NAMESPACED Issuer in its own namespace whose spec is `selfSigned` or a `ca` backed
# by a Secret that a tracked, likewise-exempt Certificate produces. Issuance and
# renewal stay unattended (cert-manager renews CA-issued certs itself), and (9)
# still applies. Every exemption is printed. First user: platform/buildkit.
# (6) each dns01 solver's provider credential is referenced BY NAME and that
# Secret is delivered SOPS-encrypted from git via a kustomization — the
# operator's one-time token hand-off (docs/08 OQ-4), never a per-cert step;
@@ -265,6 +278,7 @@ CERT_SECRETS: dict[tuple[str, str], str] = { # (namespace, secretName)
for rel, d in CERTS
}
listeners_checked = 0
GATEWAY_SECRETS: set[tuple[str, str]] = set() # every (namespace, Secret) a listener serves
for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
gw_ns = ns_of(gw)
for lis in ((gw.get("spec") or {}).get("listeners") or []):
@@ -280,6 +294,7 @@ for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
for r in refs:
kind = str(r.get("kind", "Secret"))
key = (str(r.get("namespace") or gw_ns), str(r.get("name")))
GATEWAY_SECRETS.add(key)
check(kind == "Secret",
f"{rel}: listener {lis.get('name')!r} certificateRef kind is {kind!r}; only "
"Secret refs can be traced to a cert-manager Certificate")
@@ -324,8 +339,61 @@ CM_NS = ns_of(cm_releases[0][1], "cert-manager") if cm_releases else "cert-manag
ISSUER_INDEX = {(d.get("kind"), name_of(d)): (rel, d) for rel, d in ISSUERS}
solver_creds: list[tuple[str, str, str, str]] = [] # (issuer, ns, secretName, ref field)
issuers_verified: set[str] = set()
# (5b) workload-mTLS exemption — see the header. Proven, never taken on trust.
SCOPE_ANN = "homelab.keithjackson.dev/tls-scope"
INTERNAL_NAME = re.compile(r"^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*$")
NS_ISSUERS = {(ns_of(d), name_of(d)): (rel, d) for rel, d in of_kind("Issuer", "cert-manager.io/")}
EXEMPT_SECRETS = {(ns_of(d), str((d.get("spec") or {}).get("secretName", "")))
for _, d in CERTS
if ((d.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN) == "workload-mtls"}
exempted: list[str] = []
def workload_mtls_ok(rel: str, cert: dict) -> bool:
"""True if `cert` claims the workload-mTLS scope AND proves it; records failures."""
ann = ((cert.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN)
if ann is None:
return False
cns, cname = ns_of(cert), name_of(cert)
spec = cert.get("spec") or {}
ok = True
def need(cond, msg):
nonlocal ok
check(cond, f"{rel}: Certificate {cns}/{cname} claims {SCOPE_ANN}={ann!r} but {msg}")
ok = ok and bool(cond)
need(ann == "workload-mtls", "the only accepted value is 'workload-mtls'")
need((cns, str(spec.get("secretName", ""))) not in GATEWAY_SECRETS,
"its Secret is served by a Gateway listener — that is external TLS, which must be ACME (R-TLS-01)")
names = [str(n) for n in (spec.get("dnsNames") or [])]
if spec.get("commonName"):
names.append(str(spec["commonName"]))
for n in names:
internal = INTERNAL_NAME.match(n) and ("." not in n or n.endswith(".svc") or n.endswith(".svc.cluster.local"))
need(bool(internal), f"name {n!r} is not cluster-internal (bare label or *.svc[.cluster.local])")
for f in ("ipAddresses", "uris", "emailAddresses"):
need(not spec.get(f), f"it carries {f}, which a workload-internal certificate has no use for")
ref = spec.get("issuerRef") or {}
need(str(ref.get("kind", "Issuer")) == "Issuer",
"its issuer is not a NAMESPACED Issuer — a ClusterIssuer could sign for any namespace")
hit = NS_ISSUERS.get((cns, str(ref.get("name", ""))))
need(hit is not None, f"its Issuer {ref.get('name')!r} is not tracked in namespace {cns!r}")
if hit is not None:
ispec = hit[1].get("spec") or {}
kinds = sorted(ispec)
need(kinds in (["selfSigned"], ["ca"]),
f"its Issuer is {kinds} — only selfSigned or ca are accepted for workload mTLS")
if kinds == ["ca"]:
need((cns, str((ispec.get("ca") or {}).get("secretName", ""))) in EXEMPT_SECRETS,
"its CA Issuer's Secret is not produced by a tracked workload-mtls Certificate "
"in the same namespace — the CA itself would have to be minted by hand")
if ok:
exempted.append(f"{cns}/{cname}")
return ok
for rel, cert in CERTS:
spec = cert.get("spec") or {}
if workload_mtls_ok(rel, cert):
continue # (5b): proven workload-internal; (9) below still applies
ref = spec.get("issuerRef") or {}
kind = str(ref.get("kind", "Issuer"))
per_env = resolve_per_env(str(ref.get("name", "")))
@@ -379,6 +447,8 @@ for rel, cert in CERTS:
solver_creds.append((iname, want_ns, n, f"{provider}.{field}"))
check(len(issuers_verified) > 0, "no issuer was reachable from any Certificate — (5) proved nothing")
if exempted:
print(f" (5b) workload-mTLS certificates exempted from ACME after proving it: {', '.join(exempted)}")
# (6) the solver credential is delivered, SOPS-encrypted, from git — never per-cert by hand.
check(len(solver_creds) > 0,
+291 -200
View File
@@ -1,7 +1,7 @@
---
date: 2026-10-29
branch: feat/image-pipeline
title: "Images are built, verified and published by CI; Renovate carries the digests into git"
title: "Images are built, verified and published by CI on rootless BuildKit; Renovate carries the digests into git"
---
## What changed
@@ -11,127 +11,194 @@ the codebase, never to a one-off workstation build/push. Before this branch the
package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and
both `build-image.sh` scripts said "publishing is an operator action".
- **`.gitea/workflows/images.yaml`** + **`.gitea/workflows/scripts/images.sh`** (the logic).
- Pull request: build and verify only. No push, and the step references no secret at all.
- Push to `main`, or `workflow_dispatch` on `main`: build, verify, then push to
`gitea.home.keithjackson.dev/home-network/{toolbox,agent-runner,conformance-runner}`.
Tags are `sha-<12 hex>` (immutable: the driver refuses to overwrite one) and `main` (moving).
The digest is read back from the registry with a HEAD on the manifest, compared with what the
client printed, and written to the job summary.
- Only images whose inputs changed are built. A re-run on the same commit skips images that are
already published.
- agent-runner builds FROM the toolbox built in the same run (on main, FROM its just-pushed
digest). Otherwise:
- in verify mode the toolbox is built anyway as its base, because a PR holds no credential;
- in publish mode it uses `AGENT_BASE_IMAGE` from `infra/agent/versions.env`;
- if that is still the zero placeholder, the toolbox is built and published first. This is
how the first run on main bootstraps.
- An always-run step prunes the DinD image store.
- **One build recipe per image, reused by CI.**
- New `infra/devcontainer/build-image.sh`. `dev.sh build` now delegates to it. It has an
in-image verification step that the toolbox never had: every tool is on PATH, and
promtool, kubectl, chainsaw, kyverno and sops report their pins.
- `infra/agent/build-image.sh` and `conformance/incluster/build-image.sh` were made
engine-neutral, because CI runs `ENGINE=docker` against DinD:
- `podman image exists` became `image inspect`, with a pull fallback;
- `{{.Digest}}` and `--digestfile` are gone. No local digest is printed, since only the
registry's digest is meaningful;
- `probe-validation.sh` is fed over **stdin** instead of a bind mount. Under DinD a bind
source is resolved on the daemon's filesystem, which would silently mount an empty
directory;
- `probe-validation.sh` itself gained `${BASH_SOURCE[0]:-$0}` so it survives `bash -s`
under `set -u`.
- `AGENT_BASE_IMAGE` and `IMAGE_TAG` can be overridden from the environment, and
`BUILD_REVISION` adds an `org.opencontainers.image.revision` label.
- The "publishing is an operator action" text was replaced throughout.
- **Digest references.** Each now carries a `# renovate-digest: image=<repo> tag=main` marker
on the line above it:
- `platform/pm-agent/deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`), which share
one dependency, so they bump in one PR;
- `infra/agent/versions.env` `AGENT_BASE_IMAGE`, which was `home-k3s-dev:local` and is now
the published toolbox by digest (zero placeholder until the first publish);
- `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to the
zero placeholder because Renovate can only replace a digest it can match.
### The builder: rootless BuildKit — DECISION MADE (operator, on PR #81)
The admission policies are **untouched**; the `<repo>@sha256:<digest>` form is preserved.
**The problem it answers, measured on this PR's first CI run (images run 325, failed in 6 s):** a
CI job on this runner gets **no Docker daemon at all**, for two reasons, both in the deployed
versions' source:
- act_runner v0.2.11 `internal/app/cmd/daemon.go` sets `container.docker_host` to `-` ("never
mount into jobs") whenever `DOCKER_HOST` is `tcp://`, which it is here (the DinD sidecar);
- gitea/act v0.261.2 `pkg/container/docker_run.go` `sanitizeConfig` drops **every** bind,
socket included, when `valid_volumes` is empty, which it is (the runner has no config file).
Giving jobs the DinD daemon would have made every branch push root on the node, because that
daemon is `privileged: true`.
**The decision:** no job may reach a privileged daemon, and images are built by a **rootless
BuildKit** daemon over mTLS. The runner (`apps/wave-1/gitea-actions.yaml`, DOCKER_HOST,
valid_volumes) is **untouched**.
- **`platform/buildkit/`** — a new hand-managed component in its own namespace `buildkit`. The
runner manifest is renderer-generated (`catalog/render.py` EXTRA_UNITS), and the builder has no
business sitting next to a privileged container. Flux reconciles it as the new Kustomization
`buildkit` (`clusters/base/buildkit.yaml`, `dependsOn: platform`).
- **Deployment:** `moby/buildkit:v0.34.1-rootless@sha256:f68752a5…0c4f89`, pinned by index
digest. The securityContext, every line measured (see *What I verified*):
- `privileged: false`;
- uid/gid 1000, `runAsNonRoot`;
- `capabilities: drop ALL, add SETUID, SETGID`;
- `allowPrivilegeEscalation: true`;
- `seccompProfile: Unconfined` and `appArmorProfile: Unconfined`;
- `readOnlyRootFilesystem: true`, with five emptyDirs.
Other settings:
- `--oci-worker-no-process-sandbox`;
- no ServiceAccount token;
- memory limit 4Gi, which covers build steps too;
- cache emptyDir with `sizeLimit: 6Gi`, and GC capped at 4 GB in `buildkitd.toml`
(ConfigMap).
- **mTLS via cert-manager** (`certificates.yaml`): a self-signed CA used only here, the server
cert `buildkitd-tls`, and the client cert `buildkit-client-ci`. buildkitd verifies every
client against that CA.
- **Service:** ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`.
- **NetworkPolicies:**
- default-deny;
- ingress on 1234 **only from the act-runner pod** (`gitea-actions`,
`app.kubernetes.io/name=act-runner`);
- egress: DNS, world :443/:80 for downloads, and the `ingress` entity (the Gateway) for the
registry push.
- `versions.env`: the CI-side client, `buildx` v0.38.1 with the sha256 from its
`checksums.txt`, plus the address and certificate name CI dials.
- **How DinD job containers appear to Cilium (measured, read-only).**
`iptables -t nat -S POSTROUTING` in the dind container shows
`-A POSTROUTING -s 172.18.0.0/16 ! -o br-… -j MASQUERADE` (and the same for docker0). So a job's
connection leaves with the **act-runner pod's IP, and therefore that pod's Cilium identity**.
That makes "only the act-runner pod" expressible and correct, but it cannot tell one job from
another, or a job from act_runner itself. Per-job authentication is the client certificate.
- **Admission: no Kyverno exception needed.**
- Every ClusterPolicy actually applied (`platform-config/kustomization.yaml`; live:
`conformance-trigger-job-shape`, `label-cdi-worker-pods`, `pm-agent-job-shape`,
`require-auth-mode-label`, `require-default-deny-baseline`, `require-egress-class-label`,
`restrict-host-namespaces`, `secret-provenance`, `validate-default-deny-shape`) matches only
namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`, or only its own
namespace in the job-shape policies' case. `buildkit` carries no such label.
- There are no namespaced Kyverno Policies and no cluster-wide PSA default (both measured).
- The namespace is deliberately left without a PSA label, because PSS `baseline` forbids
`Unconfined` seccomp and AppArmor.
- `validate-default-deny-shape` does apply to the namespace's default-deny NetworkPolicy, and
its shape (both directions) satisfies it.
- **Unprivileged user namespaces on the nodes (measured, read-only, via Ansible `sysctl`).** On
all three Debian 13.7 nodes, kernel `7.1.8+deb13-amd64`:
- `kernel.unprivileged_userns_clone = 1`;
- `user.max_user_namespaces = 127206`;
- no `kernel.apparmor_restrict_unprivileged_userns` (an Ubuntu-only key);
- AppArmor enabled (`Y`).
So this is a satisfied prerequisite, not an open one.
- **The TLS conformance check needed a narrow, self-proving exception** (`conformance/**`, so this
changes what green means; flagged for the reviewer).
- The first gate run failed L0 `tls-lifecycle-automated` clause (5): "every Certificate's
issuer must be ACME". That is broader than the spec it serves:
- R-TLS-01 and docs/08 concern *externally reachable* services;
- ADR-0006 says "mTLS between workloads is a separate concern … not covered here";
- a public CA cannot issue for a `*.svc.cluster.local` name or a client-auth certificate.
- The new clause (5b) exempts a Certificate annotated
`homelab.keithjackson.dev/tls-scope: workload-mtls` **only if it proves it**:
- no Gateway listener serves its Secret;
- every DNS name and the commonName are a bare label or `*.svc[.cluster.local]`;
- it has no IP, URI or email SANs;
- its issuer is a namespaced `selfSigned`/`ca` Issuer in its own namespace;
- a `ca` issuer's Secret is produced by a likewise-exempt tracked Certificate.
- Exemptions are printed, and clause (9), the renewal arithmetic, still applies.
- **Negative-tested:** a public DNS name, and a Gateway serving `buildkitd-tls`, each fail the
check with a specific message.
- **`infra/gitea/buildkit-client-secret.sh`** (operator action): copies `ca.crt`, `tls.crt` and
`tls.key` from the Secret `buildkit/buildkit-client-ci` into the Actions secrets `BUILDKIT_CA`,
`BUILDKIT_CLIENT_CERT` and `BUILDKIT_CLIENT_KEY`. Before writing anything it checks that:
- the cert verifies against the CA;
- the cert is a client-auth cert;
- the key matches the cert.
It never prints a value. `--check` reports whether the stored copy is older than the current
certificate.
### Verification moved into the Containerfiles
`docker run` cannot exist on BuildKit, so each Containerfile now names its shipped stage after the
image (`toolbox`, `agent-runner`, `conformance-runner`) and ends in a **`verify` test stage**,
`FROM <image> AS verify`. That stage runs every previous assertion at build time, as the image's
non-root uid (65532 for agent-runner and conformance-runner), in `RUN --network=none`. It is never
pushed.
- **Every existing assertion is preserved**, and the toolbox gained one it never had: every tool
on PATH, and five tools report their pinned versions.
- `probe-validation.sh` is `COPY`ed into the verify stage only, so it never ships.
- **Bug fixed on the way:** in the old `docker run` block, `cmd | grep -q … && echo ok` and
`[ … ] && echo` could never fail the script under `set -e` + `pipefail`. The "entrypoints refuse
arguments", "not root", "no API key baked in" and "HOME writable" checks asserted nothing. They
are real assertions now.
- **What a build step cannot reproduce, and how it is approximated** (also written in each stage's
header):
- **uid:** exact.
- **`--network none`:** exact, and each stage *proves* it. Its first check fails if DNS or a
TCP connection works. A negative control without `--network=none` fails that check.
- **`--read-only` rootfs:** not reproducible, because a RUN step always has a writable layer.
It is approximated by permission:
- `touch /should-fail` must fail;
- a `find / -xdev -writable ! -type l` inventory, run as the uid, must lie entirely under
`/workspace`, `/tmp` or `/var/tmp`. `! -type l` was needed: Fedora's `/usr/tmp` symlink
to `/var/tmp` tripped it.
The real control remains the pods' `readOnlyRootFilesystem`, which the admission policies
enforce.
- **`--tmpfs /workspace`:** approximated by the image's own `/workspace`, owned by 65532, which
is exactly what the pod's emptyDir replaces.
The opt-in `--verify-clone` still `run`s the image, so it remains local-only.
- **`build-image.sh` (all three):**
- verify is `build --target verify` and the image is `build --target <image>`;
- a new `--print-build-args` prints the pins as `KEY=VALUE`, which is what CI consumes. The
arg list therefore exists once;
- the local podman path still works.
- The L0 check `alert-rules-unit-tested` now looks for `"PROMTOOL_VERSION=$PROMTOOL_VERSION"` in
the toolbox's build-arg list.
### The pipeline (`.gitea/workflows/images.yaml` + `scripts/images.sh`)
- **The builder.** The driver downloads the pinned `buildx` and refuses a checksum mismatch. It
writes the client cert from the `BUILDKIT_*` secrets to a 0600 temp dir that is removed on exit,
creates a `remote` builder, and runs `buildx inspect --bootstrap`.
- **It fails loudly, never skips.** It stops with a clear message if:
- the secrets are absent;
- buildkitd is unreachable. The message lists the causes in order: not deployed yet (Flux
applies it only after merge), stale certificate, NetworkPolicy.
- **Pull request:** `buildx bake <image>-verify` for changed images. Nothing is pushed and no
registry credential is referenced. The `BUILDKIT_*` builder secrets *are* given to PR jobs,
because they must build.
- agent-runner is built FROM the toolbox built **in the same run** through a bake `target:`
context. That toolbox exists in no registry, so the toolbox is built as its base even when
its own inputs did not change.
- **Push to main / dispatch on main:** for each image, the verify target first, and only then the
image itself with `push=true`. Tags are `sha-<12 hex>` (immutable) and `main` (moving).
- The digest is read back from the registry and compared with the client's metadata.
- agent-runner builds FROM the toolbox digest just pushed, or else from the pinned
`AGENT_BASE_IMAGE` (the zero placeholder bootstraps).
- `BUILDX_NO_DEFAULT_ATTESTATIONS=1` keeps one manifest per tag, which is what admission,
Renovate and the retention script assume.
- **Unchanged from the first iteration:** change detection, publish idempotence,
`REGISTRY_PUSH_TOKEN` and its preflight, the Renovate markers, and `registry-retention.sh`. The
DinD prune step is gone, because nothing is built in DinD now.
### Unchanged from the first iteration (still true)
- **Digest references** carry `# renovate-digest: image=<repo> tag=main` markers:
- `platform/pm-agent/deployment.yaml` and `configmap.yaml`;
- `infra/agent/versions.env` `AGENT_BASE_IMAGE`, now the published toolbox by digest, zero
until the first publish;
- `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to zero.
The admission policies are untouched.
- **`renovate.json5`**:
- a new `custom.regex` manager for the markers;
- the `custom.regex` marker manager;
- `enabledManagers` `regex` → `custom.regex`;
- a packageRule that disables the `kubernetes` manager for these three repos. Otherwise it
would also see the digest-only lines and look them up as `latest`;
- an `own-image` label;
- `ignorePaths` narrowed from `conformance/**` to its subdirectories. `ignorePaths` applies
to custom managers too, so the blanket glob hid `conformance/versions.env`.
- **`renovate.yaml`**: `renovate-version: 44.167.0` pinned, plus a `RENOVATE_HOST_RULES` docker
rule for the Gitea registry, using the same `RENOVATE_TOKEN`, which now also needs
`read:package`. See below for why the version pin is a real fix and not tidiness.
- **`infra/gitea/registry-retention.sh`**: dry run by default, `--apply` to delete. This is an
operator action. Per image it keeps:
- `main` and `latest`;
- the 10 newest `sha-` tags;
- every tag whose digest is pinned anywhere in `origin/main`;
- every digest-named version.
- **`infra/gitea/conformance-trigger-kubeconfig.sh`**: the "runner image is digest-pinned"
precondition grepped the literal line `CONFORMANCE_RUNNER_IMAGE="${…_REPO}@${…_DIGEST}"`
for 64 hex digits, so it **could never pass**. It now sources the file, evaluates the value,
and rejects the zero placeholder.
- **`conformance/tests/l0-static/alert-rules-unit-tested/check.sh`**: its "the toolbox build
passes `--build-arg PROMTOOL_VERSION`" assertion now reads `infra/devcontainer/build-image.sh`
instead of `dev.sh`. The gate caught this on the first run, because the recipe had moved.
- **Docs**: `platform/pm-agent/README.md`, `runbooks/pm-agent.md` (two rows),
`docs/plans/pm-agent-issue-to-pr.md` (follow-up 1), `conformance/incluster/README.md`,
`infra/devcontainer/README.md`, and a new "How images are built" section in
`.gitea/workflows/README.md`.
- the `kubernetes` manager disabled for these three repos;
- `ignorePaths` narrowed so `conformance/versions.env` is visible.
### BLOCKER — the runner cannot build images, and fixing that is a security decision
**Measured on this PR's own CI run (images run 325): the job failed in 6 s** with
`failed to connect to the docker API at unix:///var/run/docker.sock ... no such file`. The job
container is given **no Docker daemon at all**, for two independent reasons, both read from the
exact versions deployed:
- act_runner **v0.2.11** `internal/app/cmd/daemon.go`: when `DOCKER_HOST` has a scheme other than
`unix`/`npipe` — here `tcp://localhost:2376`, the DinD sidecar — it sets
`cfg.Container.DockerHost = "-"`, which means "never mount into job containers".
- gitea/act **v0.261.2** (act_runner's `go.mod` replace) `pkg/container/docker_run.go`
`sanitizeConfig`: with `valid_volumes` empty (the runner has no config file, so the default
`[]`), `hostConfig.Binds = []string{}`. **Every** bind is dropped, the daemon socket included,
and so is any `volumes:` a workflow asks for.
I assumed the opposite while designing (from act's `getDockerDaemonSocketMountPath`, which never
gets the chance to run). That was wrong, and the PR run is what showed it.
**Why I did not fix it in this PR.** Any fix gives workflow code the DinD daemon. That daemon is
`privileged: true`, so `docker run --privileged -v /:/host` from a job is **root on the node**.
Today a malicious workflow — for example one pushed by a prompt-injected pm-agent worker, which
holds a repo write token — runs in an unprivileged container and holds no cluster credential.
After the fix it does not. That is a change to the security boundary, and it is the operator's
call. It also cannot be tested from a PR anyway: the runner only changes after merge + Flux.
**Options, for the operator:**
1. **Socket to every job (smallest change):** in `apps/wave-1/gitea-actions.yaml`,
- mount a shared `emptyDir` at `/var/run` in both the `dind` sidecar and the `runner`, so
dockerd's own `unix:///var/run/docker.sock` is visible at the same path in both;
- set the runner's `DOCKER_HOST=unix:///var/run/docker.sock`, and drop `DOCKER_TLS_VERIFY`
and `DOCKER_CERT_PATH`, because TLS over a unix socket breaks the client;
- add an act_runner config file (`CONFIG_FILE`) with `container.valid_volumes:
["/var/run/docker.sock"]` and `runner.file: /data/.runner`, so the existing registration
is kept.
Consequence: any workflow on any branch can reach node root.
2. **Option 1 on a SECOND runner** with its own label (e.g. `image-builder`) and its own DinD,
with `images.yaml` `runs-on` that label. The gate runner stays socket-less, and image builds
stop competing with the gate for the single slot and its 7 GB node disk. **It is not a
security boundary**: Gitea cannot restrict which workflow may target a label, so any branch
could still `runs-on: image-builder`.
3. **A builder that is not a privileged daemon**, e.g. a rootless BuildKit sidecar exposed to
jobs. That bounds a job to BuildKit's rootless sandbox instead of node root. It needs the
verification steps rewritten, because they *run* the image (as uid 65532, `--read-only`),
which BuildKit alone does not do. This is a larger piece of work.
Nothing in this PR has to change for options 1 or 2 except `runs-on` for option 2. The driver
already produced verified images with a stock docker 27 CLI (see *What I verified*).
`renovate.yaml` pins `renovate-version: 44.167.0` and adds a docker `RENOVATE_HOST_RULES` entry
using `RENOVATE_TOKEN`, which now needs `read:package`.
- **`infra/gitea/registry-retention.sh`** (dry run by default; operator action); the
`conformance-trigger-kubeconfig.sh` precondition fix; and the docs (pm-agent README, runbook,
plan, conformance README, devcontainer README, workflows README). The docs are updated again
for BuildKit, and there is a new `platform/buildkit/README.md`.
### Decisions and pushback
@@ -147,7 +214,8 @@ already produced verified images with a stock docker 27 CLI (see *What I verifie
- `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3.
The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight
pattern.
pattern. With BuildKit the credential reaches the daemon through the buildx session (a
0600 `config.json` in the job's temp dir), never through the cluster.
- **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate:
- `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` →
`custom.regex`;
@@ -195,89 +263,112 @@ already produced verified images with a stock docker 27 CLI (see *What I verifie
## What I verified
- **Local gate, pinned toolbox, from this worktree, committed HEAD.** Command:
`ENGINE=<shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`.
The shim adds the `.git` mount. Result: see the PR body for the rc and stage list. They are
recorded there rather than here because this entry is part of the tree being gated.
- **Local podman builds through the CI driver**, run as
`MODE=verify ENGINE=podman BASE_SHA=$(git merge-base origin/main HEAD) .gitea/workflows/scripts/images.sh`.
It returned rc 0 and built and verified all three images:
- toolbox: every tool on PATH, and the five pins matched;
- agent-runner: `PM_SELFTEST=1 pm-dispatch` OK, and the stdin-fed `probe-validation.sh`
gave "20 refusal(s), 0 failure(s)";
- conformance-runner: all seven `conformance-clone` refusals.
- **The docker-CLI code path**, i.e. what CI runs. I ran the same driver inside
`docker:27-cli` (`ENGINE=docker`, `DOCKER_BUILDKIT=0`) against podman's Docker-compatible API
socket. Exit code 0; all three images built and verified (selftest OK; probe OK, 20 refusals).
This covers the docker-specific calls (`image inspect` formats, `run -i` with stdin,
`--tmpfs`, `--read-only`, `--user`). It does **not** cover BuildKit or a real DinD daemon.
- **This PR's CI, run 325 (`images`): FAILED in 6 s**, because there is no Docker daemon in the
job container (see BLOCKER). Gate run 324 failed only on `approval-policy` (expected,
sensitive paths). Every other stage passed there, including `gitea-settings`.
- **Other driver paths, run locally:**
- agent-runner alone builds the toolbox as its base;
- an empty diff builds nothing;
- publish mode without a token dies before doing anything;
- `infra/agent/build-image.sh` with the zero placeholder dies with build-locally
instructions.
- **Renovate, with the pinned 44.167.0 image, locally:**
- `renovate-config-validator` exits 0;
- a `--platform=local` `extract` dry run finds all four marker references (custom regex manager,
`currentValue: main`, `currentDigest: sha256:000…`);
- a `lookup` dry run shows the `kubernetes`-manager duplicates as `skipReason: disabled` and the
marker deps taking the "unsupported/unversioned value main → getDigest(main)" path. That
lookup currently 404s because nothing is published yet.
- **Renovate 39.264.0 rejects main's current `renovate.json5`.** Validator output reproduced
above.
- **`registry-retention.sh` dry run against the live Gitea (read-only):** zero versions per image,
nothing to delete. The token exchange and the package listing both answered.
- **Live, read-only measurements** (kubectl and the Gitea API):
- the runner pod is on k3s-prod-1, which had **7 GB** nodefs available;
- DinD holds only the 1.54 GB act image; buildx is v0.20.1; the daemon is 27.5.1 on overlay2
with cgroup v2;
- Gitea `/data` is 9.7G, 1% used;
- the Gateway's Envoy (Cilium) has `streamIdleTimeout: 300s`, `maxStreamDuration: 0s`, and
**no route `timeout`** on the gitea route;
- Gitea's `LIMIT_SIZE_CONTAINER` is unset (default −1 = unlimited, `modules/setting/packages.go`).
**Proved locally, against the exact pinned images:**
- **Rootless buildkitd, configured as the manifest configures it.**
- Setup: the pinned `moby/buildkit:v0.34.1-rootless` digest, under rootless podman, with
`--cap-drop ALL --cap-add SETUID --cap-add SETGID`, seccomp unconfined and `--read-only`;
tmpfs at the same five paths; the ConfigMap's `buildkitd.toml`; mTLS on :1234.
- The image's `/etc/subuid` was overridden with `user:1:999` and `user:1001:64535`, because
rootless podman maps only 65,537 uids. rootlesskit's real `newuidmap` user-namespace path
still ran.
- It started, `buildctl debug workers` listed the OCI worker, and `debug workers -v` showed the
GC config applied (max 4.29 GB, reserved 1.07 GB).
- **The securityContext, item by item.** Each requirement was measured by taking it away:
- **default seccomp:** daemon starts, every RUN step fails → `Unconfined` is required;
- **no `SETUID`/`SETGID`:** rootlesskit dies with
`newuidmap … operation not permitted` → both capabilities are required;
- **`--read-only` rootfs:** works once the five emptyDirs exist. Its only failure was the
probe client's `~/.docker`, which is now an emptyDir;
- **minimal caps + Unconfined seccomp:** works, and a RUN step executes.
- **The full CI path, in the actual CI job image.** Setup: `ghcr.io/catthehacker/ubuntu:act-22.04`
running `.gitea/workflows/scripts/images.sh` in verify mode. It downloaded and sha256-checked
buildx v0.38.1, connected over mTLS, and bake-built all three `verify` targets from an empty
cache. rc 0, 1m32s total:
- `TOOLBOX VERIFIED` (42 s);
- `AGENT-RUNNER VERIFIED` (26 s): selftest OK, probe "20 refusal(s), 0 failure(s)", and built
FROM the in-run toolbox through the `target:` context, with the log showing
`[agent-runner-verify toolbox 1/6]` and `base=toolbox-in-this-run`;
- `CONFORMANCE-RUNNER VERIFIED` (21 s): all seven refusals;
- every stage printed `confirmed: no network`.
- **mTLS rejects what it should.** Tested with a real BuildKit API call (`buildctl debug workers`):
- no client cert → `remote error: tls: certificate required`;
- a client cert from another CA → rejected the same way;
- the valid cert → accepted.
Through the driver:
- another CA's client cert → rc 1;
- the wrong CA for the server → rc 1;
- an unreachable host → rc 1 in about 20 s, with the explanatory message;
- missing secrets → rc 1, naming the three secrets.
- **The local podman path**, `build-image.sh` for all three (rc 0).
- Negative control: a verify stage expecting a different promtool version printed
`DRIFT promtool want 9.9.9` and failed the build.
- Negative control: the `--network=none` check fails when the flag is removed.
- **`buildkit-client-secret.sh`'s sanity checks**, run offline on local certs. They tell apart:
- a good client cert;
- a server cert (not client-auth);
- a mismatched key;
- a cert from another CA.
**Measured live, read-only:**
- node user namespaces, Debian 13.7 and AppArmor (via an Ansible `sysctl`/`cat`);
- DinD's MASQUERADE rules;
- the live ClusterPolicies, namespaced Policies, PSA labels, and cert-manager v1.21.3 with its
ClusterIssuers;
- the act-runner pod labels.
**Renovate (unchanged since the first iteration):**
- the validator with 44.167.0: exit 0;
- `extract` finds all four markers;
- the `kubernetes` duplicates are `skipReason: disabled`;
- 39.264.0 rejects `main`'s config.
**Local gate, CI results:** see the PR body. They are recorded there because this entry is part of
the tree being gated.
## What I did NOT verify
- **Nothing has been pushed to the registry, by me or by CI.** The publish path has never run:
- the registry HEAD/digest read-back;
- **platform/buildkit has never run in the cluster.** It can't until it merges: agents may not
`kubectl apply`, and Flux reconciles only `main`. So this PR's own `images` job **cannot build
and is expected to fail at the builder preflight**, which is the designed loud failure. The
following are therefore all unproven in-cluster:
- kubelet admitting the pod;
- containerd honouring `appArmorProfile: Unconfined`;
- the five emptyDirs;
- cert-manager issuing the three certs;
- the CiliumNetworkPolicies.
- **AppArmor `Unconfined` is reasoned, not measured.** My test host runs SELinux, so the AppArmor
requirement comes from upstream's example plus the nodes having AppArmor enabled. If the pod
were to work under `RuntimeDefault` AppArmor, that line could be tightened; I did not test it.
- **The local userns mapping is narrower than the cluster's.** Locally it was 65,535 ids across
two ranges; in the cluster it is the image's own `100000:65536` against real host subuids. That
is the normal case, but it is the cluster's first test.
- **The NetworkPolicies are untested.** In particular:
- that `toEntities: [ingress]` admits the push to the Gateway VIP. The comment explains the
reasoning, but it is unmeasured;
- that world :443/:80 covers every download the builds make. A dnf mirror on another port
would fail a build visibly.
- **No push, anywhere.** The whole publish path is unexercised:
- `REGISTRY_PUSH_TOKEN` reaching buildkitd through the buildx session;
- `push=true`;
- the metadata digest compared with the registry read-back;
- agent-runner FROM `toolbox@sha256:`;
- the immutable-tag guard;
- "already published at this commit" idempotence;
- agent-runner `FROM <toolbox>@sha256:`;
- `docker login` with a PAT.
- idempotence.
The first merge to main is the first test, and it needs `REGISTRY_PUSH_TOKEN` to exist first.
- **Large-layer push through the Gateway is UNVERIFIED.** Envoy has no request-body cap and no
stream duration cap here. But with no route `timeout` set, Envoy's **default 15 s upstream
timeout** applies from the end of the request to the end of the response. That covers Gitea
finalising a blob: copying the upload into package storage on Longhorn. A ~250 MB compressed
layer should finish well inside 15 s, but I could not measure it without pushing. If the first
publish shows 504s on blob PUTs, set `timeouts.request` on the gitea HTTPRoute.
- **DinD fit is estimated, not measured** (no build has run there, see the BLOCKER):
- memory: 2Gi is shared by the job container and every build container;
- disk: the toolbox, agent-runner, conformance-runner, Fedora base and build cache should
total roughly 3 GB on top of the 1.5 GB act image, against 7 GB free on the node, with
`dind-storage` having no sizeLimit.
- **No image has been built in this runner's DinD**, because it cannot be done until the BLOCKER
is resolved. Two things are therefore unmeasured until then:
- DinD memory (2Gi) and disk fit;
- BuildKit (`buildx` v0.20.1 is on the daemon) as opposed to the legacy builder I tested
with.
- **Renovate has still never completed a run here.** Three things block it:
- #80, the `configurationFile` name;
- `RENOVATE_TOKEN` gaining `read:package`;
- this branch's version pin.
`renovatebot/github-action@v41.0.9` running a 44.x image is untested. The action only passes
the tag to `docker run`.
- **Pods cannot pull these images yet regardless.** The kubelet pull credential is PR #61
(`pm-agent-registry` dockerconfigjson, `read:package`), still open. Until it merges, a correct
digest still ends in ImagePullBackOff (401), as before.
- **`registry-retention.sh --apply` has never run**, and its keep logic has only run against an
empty registry.
- **`git diff` change detection on Gitea `push` events relies on `github.event.before`.** A
force-push or a first push falls back to building everything, which is intended but not
observed.
So are the 15 s Envoy route-timeout risk on blob finalisation, and the Gitea PVC (10Gi, shared
with repositories) absorbing ~2 GB per toolbox publish until retention runs.
- **Node disk under real builds.** The cache emptyDir (6Gi limit, 4 GB GC cap) sits on nodes that
had 7–9 GB free. It is untested under concurrent pressure.
- **Certificate rotation is designed, not exercised.** The server cert renews 30 days early but
buildkitd only reads it at start; the client copy needs the script re-run.
- **`--oci-worker-no-process-sandbox` is an accepted risk, not a mitigated one.** A build step can
ptrace the daemon and read its TLS key and cache. Anyone with the client cert, which means every
PR job, can run build steps.
- **Renovate has still never completed a run.** It is blocked on #80, on `RENOVATE_TOKEN` gaining
`read:package`, and on this branch. The `[email protected]` + 44.x image pairing is
untested.
- **Pods cannot pull these images until PR #61** (the kubelet pull secret) merges.
- **`registry-retention.sh --apply` has never run**; its keep logic has only run against an empty
registry.
@@ -0,0 +1,57 @@
---
date: 2026-10-29
branch: fix/renovate-config-filename
title: Renovate has failed every run since #79 — the workflow still named renovate.json
---
## What changed
**`.gitea/workflows/renovate.yaml`** — `configurationFile: renovate.json` → `renovate.json5`.
PR #79 (`fix/renovate-invalid-config`) renamed the config to `renovate.json5` and did not update
the one line in the workflow that names it. Measured 2026-10-29 from the Gitea Actions log of run
322 (task 513, schedule, 08:15 UTC):
Error: ENOENT: no such file or directory, stat '/workspace/Home-Network/home-network-k3s-impl/renovate.json'
Runs 315–322 (2026-10-22 → 10-29, every one listed) are all `failure` on the `schedule` event,
each failing in about one second. Renovate never started. `RENOVATE_TOKEN` **does** exist as a
repository Actions secret (listed by name via the API; value not read), so the token preflight
passes and this filename is the only blocker observed.
**`ci/lint.sh`** — a new assertion after yamllint: every non-comment `configurationFile:` key in
`.gitea/workflows/*.y*ml` must name a path that exists. This is valid YAML, so yamllint cannot see
it, and the gate stayed green through 11 days of failures.
**Second blocker, same PR:** the workflow did not pin a Renovate version, so
`renovatebot/[email protected]` ran its default, Renovate **39** (`action.yml`:
`renovate-version` `default: '39'`, read at that tag). `renovate.json5` uses `managerFilePatterns`,
which needs >= 40.2.0. Under 39.264.0 the validator rejects the config with five errors. This was
found and reproduced by the image-pipeline worker (PR #81). #79 had validated against `latest`. Pinned
`renovate-version: 44.167.0`, the version the config validates clean against per that worker.
## What I verified
- The root cause, from the run log quoted above, and that `origin/main` has `renovate.json5` and
no `renovate.json` (`git ls-tree`).
- The new guard, both ways, run standalone: on this branch it prints
`workflow configurationFile references resolve`, rc 0. With the line reverted to `renovate.json`
it prints `configurationFile 'renovate.json' does not exist in the repo`, rc 1. My first version
also matched its own explanatory comment in the workflow and failed. That is why the grep is
anchored to non-comment lines.
- Toolbox gate: result and rc are reported in the PR, not predicted here.
## What I did NOT verify
- **The 39.x rejection and the 44.167.0 pass were not reproduced by me.** I confirmed only the
action's default (`'39'`). The validation results are the PR #81 worker's.
- **Renovate has still never completed a run.** This removes the observed blocker. Others may
follow it. The test is a `workflow_dispatch` with `dry_run: true` after merge, and its log is
the first evidence that Renovate authenticates, extracts dependencies, and honours `.json5`.
- **The `enabledManagers: ["regex"]` vs `custom.regex` question** (#73, #77) is still open. The
same dry-run log answers it: look for the regex/custom manager's dependency-extraction count.
- **No alert for "Renovate keeps failing".** The guard prevents this specific cause from recurring.
Nothing notices a scheduled workflow that is red every day. That gap stays open.
- **Sensitive path.** `.gitea/**` and `ci/**` are both in `approval-policy.json`. CI will block
on `approval-policy` until the operator posts `/approve-override` as a review. The local gate
reports that stage as N/A.
@@ -0,0 +1,81 @@
---
date: 2026-10-30
branch: fix/gitea-admit-buildkit
title: The registry push 403 is Cilium refusing buildkitd's EGRESS at the Gateway — fixed in platform/buildkit, not in Gitea's ingress
---
## What changed
**`platform/buildkit/networkpolicy.yaml`** — `buildkitd-allow-egress` gains one rule:
`toEndpoints {io.kubernetes.pod.namespace: gitea, app.kubernetes.io/name: gitea}` on TCP 3000,
with a comment carrying the measurement. `platform/buildkit/README.md`'s policy row says the same.
**What did NOT change, deliberately:** `catalog/gitea.yaml` `inClusterClients` and
`apps/wave-0/gitea.yaml`. The brief proposed admitting `buildkit` there. The evidence below says
that is the wrong policy: Gitea's ingress is not what refuses the push, and adding it would only
open the direct ClusterIP path from build code to Gitea, which nothing uses.
### Why: the brief's hypothesis was half right
Images run 337 failed its push with `HEAD .../blobs/sha256:5855... 403 Forbidden: Access denied`.
The PM's hypothesis was that this was Cilium's Envoy denying the request, not Gitea. **That part
holds.** The hypothesis also said the denial came from Gitea's *ingress* policy, which does not
admit the `buildkit` namespace. **That part is wrong.** The denial is enforced on buildkitd's own
*egress*. Since Cilium 1.16.8/1.17.2 (CVE-2025-30162; prod runs `cilium:v1.20.1`), the Gateway's
L7 load balancer re-evaluates the source pod's egress policy against the identity of the backend
it resolves to. buildkitd's egress allowed `world` :443/:80 and the `ingress` entity. That gets
the TCP connection to Envoy, but no rule matches Gitea's backend identity, so Envoy answers
`403 Access denied` without forwarding the request. This is the failure commit `68690e5` fixed for
every SSO app's hop to Keycloak, and the fix has the same shape. Keycloak's ingress names no app
namespaces either.
## What I verified
All measurements are read-only, against `k3s-prod`, on 2026-10-30 around 05:27–05:37 UTC.
- **Gitea never saw the failed push.** Its router log for the previous 12h had 316 `/v2/` lines
and **0** that were `toolbox/blobs`, PUT, POST or PATCH. It also had **0** from buildkitd's pod
IP (`10.0.0.7`). In the same window it logged the CI job's own `HEAD .../toolbox/manifests/
sha-0dd57fbaf002 → 404` from the act-runner pod (`10.0.0.38`). Requests that came through the
Gateway reach Gitea and get logged. This one never arrived.
- **A reproduction with no credentials.** An unauthenticated `GET https://gitea.home.keithjackson.dev/v2/`
(Gateway VIP `192.168.2.20`) gave:
- from the buildkitd pod: **`403 Forbidden`**, with no matching line in Gitea's log;
- from the act-runner pod's `dind` container: **`401 Unauthorized`**, which is Gitea's normal
registry challenge, logged by Gitea as `GET /v2/ for 10.0.0.38:0, 401`.
No token was involved, so this cannot be a package-permission problem. The bot account and the
`REGISTRY_PUSH_TOKEN` were not the cause, and the operator does not need to change them.
- **Hubble names the enforcement point.** On `cilium-55jv8`, buildkitd's node:
`buildkit/buildkitd-859d67c8f8-x44nh (ingress) -> gateway/cilium-gateway-shared:443 http-request
DROPPED (HTTP/1.1 GET https://gitea.home.keithjackson.dev/v2/)`, followed by `http-response
FORWARDED (HTTP/1.1 403 0ms)`. In the JSON form, **`traffic_direction: EGRESS`**. Envoy wrote
the 403 itself in 0 ms.
- **Gitea's ingress does not need the source namespace.** The act-runner pod's namespace
(`gitea-actions`) is **not** in Gitea's ingress allows. Its NetworkPolicy
`default-deny-ingress-allow-egress` has an `{}` egress rule, so its egress is unrestricted. Its
requests through the Gateway still reach Gitea. `cilium-dbg bpf policy get` on Gitea's endpoint
(1876 on `cilium-9rk5x`) shows all Gateway traffic admitted as `reserved:ingress 3000/TCP`
(391 MB, 1.19 M packets) and no row for `gitea-actions`. The source-identity check therefore
happens on the source's egress, not on the destination's ingress.
- **The policy gap is visible in the BPF map.** `bpf policy get 83` (buildkitd) has these egress
rows and nothing else: `reserved:world` 80/443, `reserved:ingress ANY`, and kube-dns 53. There
is no Gitea identity.
- The edited YAML parses (three documents, names unchanged).
## What I did NOT verify
- **That this rule turns the 403 into a 401/200. UNVERIFIED.** That needs the policy applied, which
is a Flux/operator action. My confidence is high but not proven, and it rests on two things. The
Hubble verdict is EGRESS. Commit `68690e5` made the identical change for Keycloak and recorded
"Verified live: headlamp-ns -> keycloak discovery 403 -> 200".
- **The port restriction.** The rule names `3000`, Gitea's container port, which is the upstream
port Envoy connects to. The Keycloak precedent has no port. If the 403 survives, the first thing
to test is whether L7LB enforcement uses the frontend :443 instead. I expect it does not.
- **Run 337's own flows.** The Hubble ring buffer had already rotated past them, and buildkitd had
been restarted since (the pod was 18 minutes old). The reproduction above uses the same path, the
same pod spec and the same policy, but it is not the original request.
- Envoy's access logs in the `cilium-envoy` pods. Hubble's L7 flow already shows the drop, so I
did not read them.
- The push itself, the Renovate digest flow, and whether the bot team's `Packages=Write` is enough
for a first push that creates a new package. These only become testable after this lands.
+2 -2
View File
@@ -245,8 +245,8 @@ about runtime behaviour in this document is a design claim, not a measurement.
the credential decision now being local Gitea accounts with PATs, that means a
`dockerconfigjson` for an account whose token carries `read:package`. Only then does the
digest matter. **Superseded 2026-10-29:** the build-and-push is no longer an operator step.
`.gitea/workflows/images.yaml` builds and verifies the image with
`infra/agent/build-image.sh`, publishes it on a push to `main`, and Renovate opens the one
`.gitea/workflows/images.yaml` builds and verifies the image on rootless BuildKit
(`platform/buildkit/`), publishes it on a push to `main`, and Renovate opens the one
PR that replaces both placeholder digests (`decisions/2026-10-29-image-pipeline.md`). The
same two blockers sit in front of `conformance-runner`, which shares the registry path and
has never been pulled.
+81 -1
View File
@@ -50,7 +50,10 @@
# default is deliberately EMPTY so a build that forgets it fails loudly instead of
# resolving some other image named `agent-base`.
ARG AGENT_BASE_IMAGE=
FROM ${AGENT_BASE_IMAGE}
# Stage `agent-runner` is the image that ships; stage `verify` (end of file) is a TEST stage built
# FROM it that only runs assertions and is never pushed. build-image.sh and the CI pipeline
# build `--target verify` to prove the image and `--target agent-runner` for the image itself.
FROM ${AGENT_BASE_IMAGE} AS agent-runner
# Re-declared AFTER the FROM: an ARG from before the first FROM is not in scope in the
# build stage, so the provenance layer below would otherwise record an empty base.
@@ -208,3 +211,80 @@ WORKDIR /workspace
# paths above; an ENTRYPOINT here would either be ignored or prepend something the policy
# cannot see. `bash` is the interactive fallback for a human debugging the image locally.
CMD ["/bin/bash"]
# =============================================================================
# verify — a TEST stage, never shipped. Every assertion infra/agent/build-image.sh used to make
# with `run --user 65532 --read-only --tmpfs /workspace --tmpfs /tmp --network none` now runs
# here, at build time, because CI builds on rootless BuildKit where no container runtime exists.
#
# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE, stated so nobody over-reads a green verify:
# * uid 65532 ............ exact: USER is inherited from the agent-runner stage.
# * --network none ....... exact: RUN --network=none, and the first check proves it.
# * --read-only rootfs ... NOT reproducible: a RUN step always has a writable layer. It is
# APPROXIMATED by permission: as uid 65532, every path the process can
# write is inventoried and must lie under /workspace, /tmp or /var/tmp
# (the pod mounts emptyDirs on the first two; /var/tmp is read-only
# there, and TMPDIR points into /workspace so nothing should use it).
# The pod's readOnlyRootFilesystem is enforced by admission
# (platform/kyverno/admission/pm-agent-job-shape.yaml), not by this.
# * --tmpfs /workspace ... approximated by the image's own /workspace (owned 65532), which is
# what the pod's emptyDir replaces.
# The probe script is COPYed into THIS stage only, so it never ships in the image.
# =============================================================================
FROM agent-runner AS verify
COPY probe-validation.sh /opt/verify/probe-validation.sh
RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
echo "--- no network (the build-time stand-in for run --network none) ---"
if getent hosts registry.npmjs.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then
echo "UNEXPECTED: network reachable in the verify stage"; exit 1
fi
echo " confirmed: no network"
echo "--- id ---"; id
echo "--- HOME ---"; echo "HOME=${HOME} CLAUDE_CONFIG_DIR=${CLAUDE_CONFIG_DIR}"
echo "--- baked versions ---"; cat /etc/agent-runner-versions
echo "--- the gate toolchain the worker needs (CLAUDE.md §3) ---"
for t in git python3 bash yamllint check-jsonschema kyverno promtool chainsaw flux sops age kubectl jq diff; do
if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t";
else printf " MISSING %s\n" "$t"; missing=1; fi
done
[ -z "${missing:-}" ] || { echo "the gate cannot run in this image"; exit 1; }
echo "--- python can import the renderer dependency ---"
python3 -c "import yaml; print('PyYAML', yaml.__version__)"
echo "--- claude ---"; claude --version
echo "--- the two entrypoints refuse arguments ---"
# Each must REFUSE. (The old docker-run version wrote `cmd | grep -q ... && echo`, which under
# `set -e` + pipefail could never fail the script; these are real assertions.)
for e in pm-worker pm-dispatch; do
out="$(/usr/local/bin/$e --anything 2>&1 || true)"
grep -q "takes no arguments" <<<"$out" || { echo "UNEXPECTED: $e accepted an argument: $out"; exit 1; }
echo " $e: rejected"
done
echo "--- the dispatcher selftest (branch vocabulary, result envelopes, Job template, persona) ---"
PM_SELFTEST=1 /usr/local/bin/pm-dispatch
echo "--- the entrypoints are root-owned and NOT writable by the runtime uid ---"
for f in /usr/local/bin/pm-worker /usr/local/bin/pm-dispatch /etc/pm-agent/worker-job-template.yaml; do
stat -c " %n %U:%G %a" "$f"
if [ -w "$f" ]; then echo "UNEXPECTED: uid $(id -u) can write $f"; exit 1; fi
done
echo " confirmed: not writable by uid $(id -u)"
echo "--- the root filesystem is not writable by the runtime uid (expected: a failure) ---"
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi
echo " confirmed: / is not writable"
echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---"
stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \
| grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)"
if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf ' %s\n' ${stray} | head -20; exit 1; fi
echo " confirmed: no writable path outside /workspace, /tmp, /var/tmp"
echo "--- the two writable paths really are writable ---"
touch "${HOME}/.probe" /tmp/.probe || { echo "UNEXPECTED: ${HOME} or /tmp not writable"; exit 1; }
echo " confirmed: ${HOME} and /tmp are writable"
echo "--- uid is non-zero ---"
[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; }
echo " confirmed: uid $(id -u), not root"
echo "--- no Anthropic API key is baked into the image (it would outrank the subscription token) ---"
[ -z "${ANTHROPIC_API_KEY:-}${ANTHROPIC_AUTH_TOKEN:-}" ] || { echo "UNEXPECTED: an Anthropic key is baked in"; exit 1; }
echo " confirmed: neither is set"
echo "--- pm-worker's input validation, with the interpreter and script that run in the cluster ---"
PM_WORKER_PATH=/usr/local/bin/pm-worker /bin/bash /opt/verify/probe-validation.sh
echo "AGENT-RUNNER VERIFIED"
VERIFY
+39 -91
View File
@@ -2,7 +2,8 @@
###############################################################################
# infra/agent/build-image.sh — build the pinned `agent-runner` image and prove it works.
#
# ./build-image.sh # build + verify + print the image id
# ./build-image.sh # build the verify target, then the image
# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes)
# ./build-image.sh --no-verify # build only (skip the in-image proof)
# ENGINE=docker ./build-image.sh # build with docker instead of podman
# AGENT_BASE_IMAGE=home-k3s-dev:local ./build-image.sh
@@ -11,12 +12,12 @@
# IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this)
#
# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
# Publishing is done by CI: .gitea/workflows/images.yaml calls this script, and on a push to
# `main` it pushes what this script built AND verified to the Gitea registry, records the
# registry-returned digest in the job summary, and Renovate then opens the pull request that
# moves platform/pm-agent/ onto that digest. So this script's job is to build, and to prove
# the toolchain and both entrypoints work INSIDE the image as the non-root uid with the
# read-only root filesystem the admission policy forces — the same proof locally and in CI.
# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on
# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and
# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest
# in the job summary. Renovate then opens the pull request that moves platform/pm-agent/ onto
# that digest. This script is the LOCAL path (podman by default); its proof — the
# Containerfile's `verify` stage — is the same one CI runs.
#
# VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the pins this image adds
# (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds the
@@ -26,11 +27,10 @@
# kept current by Renovate. An AGENT_BASE_IMAGE in the environment overrides it — CI passes
# the toolbox it built in the same run, and a developer can pass `home-k3s-dev:local`.
#
# ENGINE PORTABILITY. CI runs this with ENGINE=docker against a Docker-in-Docker daemon, so
# nothing below may be podman-only (`image exists`, `--digestfile`, `{{.Digest}}`), and
# nothing may BIND-MOUNT a file from this checkout into a container: under DinD the daemon
# resolves bind sources on ITS filesystem, where this checkout does not exist, and silently
# mounts an empty directory instead. Files go in over stdin.
# NO `run` STEP. Verification used to `run` the image as uid 65532 with a read-only rootfs.
# CI has no container runtime (rootless BuildKit only), so the assertions moved into the
# Containerfile's `verify` stage and this script builds that target. `--print-build-args`
# prints the pins as KEY=VALUE for the CI pipeline; AGENT_BASE_IMAGE is the caller's choice.
###############################################################################
set -euo pipefail
@@ -39,9 +39,11 @@ REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
ENGINE="${ENGINE:-podman}"
VERIFY=1
PRINT_ARGS=0
for arg in "$@"; do
case "$arg" in
--no-verify) VERIFY=0 ;;
--print-build-args) PRINT_ARGS=1 ;;
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown argument: $arg" >&2; exit 2 ;;
esac
@@ -51,8 +53,6 @@ log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
info() { printf ' %s\n' "$*"; }
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
# --- pins: sourced, never invented -------------------------------------------
AGENT_ENV="${HERE}/versions.env"
[ -f "${AGENT_ENV}" ] || die "missing ${AGENT_ENV} — the pins live there, not here"
@@ -68,6 +68,17 @@ source "${AGENT_ENV}"
: "${AGENT_MODEL:?AGENT_MODEL not set in ${AGENT_ENV}}"
: "${AGENT_IMAGE_REPO:?AGENT_IMAGE_REPO not set in ${AGENT_ENV}}"
# THE build args, in ONE place; `--print-build-args` hands them to CI (which builds on rootless
# BuildKit and supplies its own base reference). AGENT_BASE_IMAGE is deliberately NOT in this
# list: the caller decides the base.
BUILD_ARGS=(
"NODEJS_MAJOR=${NODEJS_MAJOR}" "CLAUDE_CODE_VERSION=${CLAUDE_CODE_VERSION}"
"CLAUDE_CODE_MIN_VERSION=${CLAUDE_CODE_MIN_VERSION}" "AGENT_MODEL=${AGENT_MODEL}"
)
if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
# --- the base must exist, locally or in the registry ---------------------------
# Deliberately NOT built on demand. The toolbox is its own reviewed artefact with its own
# pins, and silently building it here would mean this script could change the toolchain a
@@ -103,89 +114,26 @@ info " model ${AGENT_MODEL}"
# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
LABEL_ARGS=()
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
"${ENGINE}" build "${LABEL_ARGS[@]}" \
--file "${HERE}/Containerfile" \
--tag "${IMAGE_REF}" \
--build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}" \
--build-arg "NODEJS_MAJOR=${NODEJS_MAJOR}" \
--build-arg "CLAUDE_CODE_VERSION=${CLAUDE_CODE_VERSION}" \
--build-arg "CLAUDE_CODE_MIN_VERSION=${CLAUDE_CODE_MIN_VERSION}" \
--build-arg "AGENT_MODEL=${AGENT_MODEL}" \
"${HERE}"
ARG_FLAGS=(--build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}")
for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
# --- verify: the point of the whole exercise ---------------------------------
# A build that merely succeeded is not evidence this image is usable. Prove the tools
# EXECUTE, as the NON-ROOT uid, with a READ-ONLY root filesystem and only the two volumes
# the admission policy admits — the exact shape a worker Job runs in. Anything that works
# here but not there is a 3am discovery.
# A build that merely succeeded is not evidence this image is usable. The proof is the
# Containerfile's `verify` stage: as the NON-ROOT uid, with no network, the toolchain executes,
# both entrypoints refuse arguments, the dispatcher selftest passes, the entrypoints are not
# writable, nothing outside /workspace and /tmp is writable, and pm-worker's input guards
# (probe-validation.sh) refuse every hostile input. It is a build target rather than a
# `run` so the SAME proof runs here and in CI, where no container runtime exists — see the
# stage's header for what a build step can and cannot reproduce of the pod's shape.
if [ "${VERIFY}" -eq 1 ]; then
log "Verifying inside the image (uid 65532, read-only rootfs, workspace+tmp only)"
"${ENGINE}" run --rm \
--user 65532:65532 \
--read-only \
--tmpfs /workspace:rw,mode=1777 \
--tmpfs /tmp:rw,mode=1777 \
--network none \
--entrypoint "" \
"${IMAGE_REF}" \
/bin/bash -euo pipefail -c '
echo "--- id ---"; id
echo "--- HOME ---"; echo "HOME=${HOME} CLAUDE_CONFIG_DIR=${CLAUDE_CONFIG_DIR}"
echo "--- baked versions ---"; cat /etc/agent-runner-versions
echo "--- the gate toolchain the worker needs (CLAUDE.md §3) ---"
for t in git python3 bash yamllint check-jsonschema kyverno promtool chainsaw flux sops age kubectl jq diff; do
if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t";
else printf " MISSING %s\n" "$t"; missing=1; fi
done
[ -z "${missing:-}" ] || { echo "the gate cannot run in this image"; exit 1; }
echo "--- python can import the renderer dependency ---"
python3 -c "import yaml; print(\"PyYAML\", yaml.__version__)"
echo "--- claude ---"; claude --version
echo "--- the two entrypoints refuse arguments ---"
/usr/local/bin/pm-worker --anything 2>&1 | grep -q "takes no arguments" && echo " pm-worker: rejected"
/usr/local/bin/pm-dispatch --anything 2>&1 | grep -q "takes no arguments" && echo " pm-dispatch: rejected"
echo "--- the dispatcher selftest (branch vocabulary, result envelopes, Job template, persona) ---"
PM_SELFTEST=1 /usr/local/bin/pm-dispatch
echo "--- the entrypoints are root-owned and NOT writable by the runtime uid ---"
for f in /usr/local/bin/pm-worker /usr/local/bin/pm-dispatch /etc/pm-agent/worker-job-template.yaml; do
stat -c " %n %U:%G %a" "$f"
if [ -w "$f" ]; then echo "UNEXPECTED: uid $(id -u) can write $f"; exit 1; fi
done
echo " confirmed: not writable by uid $(id -u)"
echo "--- rootfs is read-only (expected: a failure) ---"
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
echo " confirmed: / is read-only"
echo "--- the two writable paths really are writable ---"
touch "${HOME}/.probe" /tmp/.probe && echo " confirmed: ${HOME} and /tmp are writable"
echo "--- uid is non-zero ---"
[ "$(id -u)" -ne 0 ] && echo " confirmed: uid $(id -u), not root"
echo "--- no Anthropic API key is baked into the image (it would outrank the subscription token) ---"
[ -z "${ANTHROPIC_API_KEY:-}${ANTHROPIC_AUTH_TOKEN:-}" ] && echo " confirmed: neither is set"
'
# The worker's input guards, exercised in the image, with the interpreter and the script
# that will actually run in the cluster. The probe is mounted read-only rather than baked
# in: it is a build-time check, not something a worker Job has any business carrying.
log "Probing pm-worker's input validation inside the image"
#
# The probe goes in over STDIN, not as a bind mount: under Docker-in-Docker (CI) a bind
# source is resolved on the daemon's filesystem, where this file does not exist, and the
# container would get an empty directory. PM_WORKER_PATH points it at the baked worker,
# since the probe's own default (a sibling of the probe file) has no meaning on stdin.
"${ENGINE}" run --rm -i \
--user 65532:65532 \
--read-only \
--tmpfs /workspace:rw,mode=1777 \
--tmpfs /tmp:rw,mode=1777 \
--network none \
-e PM_WORKER_PATH=/usr/local/bin/pm-worker \
--entrypoint "" \
"${IMAGE_REF}" \
/bin/bash -s < "${HERE}/probe-validation.sh"
log "Verifying (Containerfile target: verify)"
"${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}"
log "Image verified"
fi
"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target agent-runner \
--file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}"
# --- result ------------------------------------------------------------------
# No digest is printed: a manifest digest does not exist until a registry has the manifest,
# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at
+5 -5
View File
@@ -16,14 +16,14 @@
# scrubbed environment and fake credentials, and if a guard ever stopped firing, the case
# would fail here rather than by really cloning something.
#
# Run offline, no cluster, no Gitea, no model. infra/agent/build-image.sh runs it inside the
# built image as well, where the interpreter and the script are the ones that will actually
# run in the cluster.
# Run offline, no cluster, no Gitea, no model. The Containerfile's `verify` stage runs it inside
# the built image as well (as uid 65532, no network), where the interpreter and the script are
# the ones that will actually run in the cluster — locally via build-image.sh and in CI.
###############################################################################
set -uo pipefail
# `:-$0` because build-image.sh feeds this script over STDIN (`bash -s`), where BASH_SOURCE is
# empty and `set -u` would abort on it; PM_WORKER_PATH is always set in that case.
# `:-$0` keeps this safe if it is ever fed over stdin (`bash -s`), where BASH_SOURCE is empty
# and `set -u` would abort on it.
HERE="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)"
WORKER="${PM_WORKER_PATH:-${HERE}/pm-worker}"
[ -r "$WORKER" ] || { echo "cannot read ${WORKER}" >&2; exit 2; }
+49 -1
View File
@@ -8,7 +8,10 @@
# packages are retained as harmless disk-image utilities. Fedora base is pinned for
# reproducibility.
ARG FEDORA_VERSION=44
FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION}
# Stage `toolbox` is the image that ships. Stage `verify` (end of file) is a TEST stage: it is
# built FROM the finished image and only runs assertions, and it is never pushed. Build with
# `--target toolbox` for the image and `--target verify` to prove it (build-image.sh does both).
FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} AS toolbox
ARG OPENTOFU_VERSION
ARG AGE_VERSION
@@ -95,3 +98,48 @@ RUN { echo "opentofu=${OPENTOFU_VERSION}"; echo "age=${AGE_VERSION}"; echo "sops
WORKDIR /work
CMD ["/bin/bash"]
# =============================================================================
# verify — a TEST stage, never shipped. It runs every assertion against the finished image AT
# BUILD TIME, so CI needs no container runtime to prove the image: the image pipeline builds
# this target on rootless BuildKit (platform/buildkit/), where `docker run` does not exist.
# `RUN --network=none` reproduces the old `run --network none` exactly, and the first check
# below proves it (it fails if any network is reachable).
# =============================================================================
FROM toolbox AS verify
ARG PROMTOOL_VERSION
ARG KUBECTL_VERSION
ARG CHAINSAW_VERSION
ARG KYVERNO_VERSION
ARG SOPS_VERSION
RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
echo "--- no network (the build-time stand-in for run --network none) ---"
if getent hosts registry.fedoraproject.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then
echo "UNEXPECTED: network reachable in the verify stage"; exit 1
fi
echo " confirmed: no network"
echo "--- baked versions ---"; cat /etc/dev-toolchain-versions
echo "--- every tool is on PATH ---"
for t in tofu age age-keygen sops flux kubectl helm kyverno kyverno-json chainsaw yq \
yamllint check-jsonschema promtool ansible-playbook git make jq diff python3 \
wg qrencode; do
if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t";
else printf " MISSING %s\n" "$t"; missing=1; fi
done
[ -z "${missing:-}" ] || { echo "toolbox is missing tools"; exit 1; }
echo "--- version-sensitive tools report their pins ---"
: "${PROMTOOL_VERSION:?}" "${KUBECTL_VERSION:?}" "${CHAINSAW_VERSION:?}" "${KYVERNO_VERSION:?}" "${SOPS_VERSION:?}"
check() { # name pin actual-output
if printf "%s" "$3" | grep -qF "$2"; then printf " ok %-10s %s\n" "$1" "$2";
else printf " DRIFT %-10s want %s, got: %s\n" "$1" "$2" "$(printf "%s" "$3" | head -1)"; drift=1; fi
}
check promtool "${PROMTOOL_VERSION}" "$(promtool --version 2>&1)"
check kubectl "${KUBECTL_VERSION}" "$(kubectl version --client 2>&1)"
check chainsaw "${CHAINSAW_VERSION}" "$(chainsaw version 2>&1)"
check kyverno "${KYVERNO_VERSION}" "$(kyverno version 2>&1)"
check sops "${SOPS_VERSION}" "$(sops --version --disable-version-check 2>&1 || sops --version 2>&1)"
[ -z "${drift:-}" ] || { echo "a tool does not report its pinned version"; exit 1; }
echo "--- python can import what the renderer and the gate need ---"
python3 -c "import yaml; print(' PyYAML', yaml.__version__)"
echo "TOOLBOX VERIFIED"
VERIFY
+5 -3
View File
@@ -36,9 +36,11 @@ cd infra/devcontainer
### The published toolbox
`./dev.sh build` delegates to [`build-image.sh`](build-image.sh), which is the **one** build
recipe: CI runs the same script (with its in-image verification — every tool executes, and the
version-sensitive ones report their `versions.env` pins) and, on a push to `main` that changes
`./dev.sh build` delegates to [`build-image.sh`](build-image.sh). The recipe is the
`Containerfile` plus that script's build-arg list, and the proof is the Containerfile's `verify`
stage (every tool on `PATH`, the version-sensitive ones report their `versions.env` pins, no
network). CI builds the same Containerfile on rootless BuildKit with the args from
`build-image.sh --print-build-args`, and, on a push to `main` that changes
`Containerfile`, `versions.env` or `build-image.sh`, **publishes** the result as
`gitea.home.keithjackson.dev/home-network/toolbox` (tags `sha-<commit>` and `main`) —
see [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). `agent-runner` is
+37 -61
View File
@@ -6,12 +6,14 @@
# ./build-image.sh --no-verify # build only
# IMAGE=foo:bar ./build-image.sh # build under another local name
# ENGINE=docker ./build-image.sh # build with docker instead of podman
# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes)
#
# ONE BUILD DEFINITION, TWO CALLERS. `dev.sh build` calls this for local development, and the
# CI image pipeline (.gitea/workflows/images.yaml -> .gitea/workflows/scripts/images.sh) calls
# it to build the image it then PUBLISHES as gitea.home.keithjackson.dev/home-network/toolbox.
# Keeping the build args in exactly one place is what makes "the toolbox CI publishes" and "the
# toolbox you gate with locally" the same recipe.
# ONE BUILD DEFINITION, TWO CALLERS. `dev.sh build` calls this for local development. The CI
# image pipeline (.gitea/workflows/images.yaml -> .gitea/workflows/scripts/images.sh) builds the
# SAME Containerfile on rootless BuildKit (platform/buildkit/) — no container engine exists in a
# CI job — taking its build args from `--print-build-args`, and PUBLISHES the result as
# gitea.home.keithjackson.dev/home-network/toolbox. The Containerfile (including its `verify`
# test stage) and the arg list here are the one recipe both use.
#
# THIS SCRIPT NEVER PUSHES. Publishing is CI's job (images.yaml, on push to main), not a
# workstation's: an image that exists only because someone ran `podman push` from a laptop has
@@ -26,9 +28,11 @@ ENGINE="${ENGINE:-podman}"
IMAGE="${IMAGE:-home-k3s-dev:local}"
VERIFY=1
PRINT_ARGS=0
for arg in "$@"; do
case "$arg" in
--no-verify) VERIFY=0 ;;
--print-build-args) PRINT_ARGS=1 ;;
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown argument: $arg" >&2; exit 2 ;;
esac
@@ -37,72 +41,44 @@ done
log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
# shellcheck disable=SC1091
source "${HERE}/versions.env"
# THE build args, in ONE place. `--print-build-args` hands exactly this list to the CI pipeline
# (.gitea/workflows/scripts/images.sh), which builds on rootless BuildKit rather than calling
# "$ENGINE build" — so CI and a local build cannot drift apart on a pin.
BUILD_ARGS=(
"FEDORA_VERSION=$FEDORA_VERSION" "OPENTOFU_VERSION=$OPENTOFU_VERSION" "AGE_VERSION=$AGE_VERSION"
"SOPS_VERSION=$SOPS_VERSION" "FLUX_VERSION=$FLUX_VERSION" "KUBECTL_VERSION=$KUBECTL_VERSION"
"HELM_VERSION=$HELM_VERSION" "KYVERNO_VERSION=$KYVERNO_VERSION"
"KYVERNO_JSON_VERSION=$KYVERNO_JSON_VERSION" "CHAINSAW_VERSION=$CHAINSAW_VERSION"
"YAMLLINT_VERSION=$YAMLLINT_VERSION" "YQ_VERSION=$YQ_VERSION"
"CHECK_JSONSCHEMA_VERSION=$CHECK_JSONSCHEMA_VERSION" "PROMTOOL_VERSION=$PROMTOOL_VERSION"
)
if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
ARG_FLAGS=()
for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
LABEL_ARGS=()
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
log "Building ${IMAGE}"
"$ENGINE" build "${LABEL_ARGS[@]}" \
--build-arg FEDORA_VERSION="$FEDORA_VERSION" \
--build-arg OPENTOFU_VERSION="$OPENTOFU_VERSION" \
--build-arg AGE_VERSION="$AGE_VERSION" \
--build-arg SOPS_VERSION="$SOPS_VERSION" \
--build-arg FLUX_VERSION="$FLUX_VERSION" \
--build-arg KUBECTL_VERSION="$KUBECTL_VERSION" \
--build-arg HELM_VERSION="$HELM_VERSION" \
--build-arg KYVERNO_VERSION="$KYVERNO_VERSION" \
--build-arg KYVERNO_JSON_VERSION="$KYVERNO_JSON_VERSION" \
--build-arg CHAINSAW_VERSION="$CHAINSAW_VERSION" \
--build-arg YAMLLINT_VERSION="$YAMLLINT_VERSION" \
--build-arg YQ_VERSION="$YQ_VERSION" \
--build-arg CHECK_JSONSCHEMA_VERSION="$CHECK_JSONSCHEMA_VERSION" \
--build-arg PROMTOOL_VERSION="$PROMTOOL_VERSION" \
-t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}"
# --- verify ------------------------------------------------------------------
# A build that succeeded is not evidence the toolbox can run the gate. Every tool the gate and
# the runbooks call must EXECUTE, and the version-sensitive ones must report the version
# versions.env pins — promtool above all, because ci/versions.env asserts a promtool/Prometheus
# lockstep and a drifted engine evaluates alert-rule tests differently (see versions.env).
# --- verify --------------------------------------------------------------------
# The assertions live in the Containerfile's `verify` stage (every tool on PATH, the
# version-sensitive ones report their versions.env pins — promtool above all, because
# ci/versions.env asserts a promtool/Prometheus lockstep — and no network), so the SAME proof
# runs here under podman and in CI under rootless BuildKit. A failing assertion fails the build.
if [ "${VERIFY}" -eq 1 ]; then
log "Verifying the toolchain inside ${IMAGE}"
"$ENGINE" run --rm --network none --entrypoint "" \
-e PROMTOOL_VERSION="${PROMTOOL_VERSION}" \
-e KUBECTL_VERSION="${KUBECTL_VERSION}" \
-e CHAINSAW_VERSION="${CHAINSAW_VERSION}" \
-e KYVERNO_VERSION="${KYVERNO_VERSION}" \
-e SOPS_VERSION="${SOPS_VERSION}" \
"$IMAGE" /bin/bash -euo pipefail -c '
echo "--- baked versions ---"; cat /etc/dev-toolchain-versions
echo "--- every tool is on PATH ---"
for t in tofu age age-keygen sops flux kubectl helm kyverno kyverno-json chainsaw yq \
yamllint check-jsonschema promtool ansible-playbook git make jq diff python3 \
wg qrencode; do
if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t";
else printf " MISSING %s\n" "$t"; missing=1; fi
done
[ -z "${missing:-}" ] || { echo "toolbox is missing tools"; exit 1; }
echo "--- version-sensitive tools report their pins ---"
check() { # name pin actual-output
if printf "%s" "$3" | grep -qF "$2"; then printf " ok %-10s %s\n" "$1" "$2";
else printf " DRIFT %-10s want %s, got: %s\n" "$1" "$2" "$(printf "%s" "$3" | head -1)"; drift=1; fi
}
check promtool "${PROMTOOL_VERSION}" "$(promtool --version 2>&1)"
check kubectl "${KUBECTL_VERSION}" "$(kubectl version --client 2>&1)"
check chainsaw "${CHAINSAW_VERSION}" "$(chainsaw version 2>&1)"
check kyverno "${KYVERNO_VERSION}" "$(kyverno version 2>&1)"
check sops "${SOPS_VERSION}" "$(sops --version --disable-version-check 2>&1 || sops --version 2>&1)"
[ -z "${drift:-}" ] || { echo "a tool does not report its pinned version"; exit 1; }
echo "--- python can import what the renderer and the gate need ---"
python3 -c "import yaml; print(\" PyYAML\", yaml.__version__)"
'
log "Verifying (Containerfile target: verify)"
"$ENGINE" build "${ARG_FLAGS[@]}" --target verify -f "${HERE}/Containerfile" "${HERE}"
log "Toolbox verified"
fi
log "Building ${IMAGE}"
"$ENGINE" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target toolbox \
-t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}"
log "Built ${IMAGE}"
printf ' image id %s\n' "$("$ENGINE" image inspect --format '{{.Id}}' "$IMAGE")"
+104
View File
@@ -0,0 +1,104 @@
#!/usr/bin/env bash
# Copy the image builder's CLIENT certificate from the cluster into the three repository Actions
# secrets the image pipeline reads: BUILDKIT_CA, BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY.
#
# ./infra/gitea/buildkit-client-secret.sh # copy and verify
# ./infra/gitea/buildkit-client-secret.sh --check # compare only; changes nothing
#
# RUNNING IT IS AN OPERATOR ACTION. It writes repository secrets.
#
# WHERE THE CERTIFICATE COMES FROM. platform/buildkit/certificates.yaml has cert-manager issue
# `buildkit-client-ci` from a CA that exists only for the builder; buildkitd admits a connection
# only with a certificate signed by that CA. CI jobs hold NO cluster credential (R-SEC-02), so they
# cannot read the Secret themselves: this script is the one bridge, run by a human who has both
# a kube context and the Gitea credential.
#
# WHEN TO RUN IT
# 1. once, after Flux has applied platform/buildkit (Kustomization `buildkit` Ready, and
# `kubectl -n buildkit get certificate buildkit-client-ci` READY=True);
# 2. after every renewal of that certificate (duration 1y, renewed 30 days early). Until then
# the image pipeline fails LOUDLY at its builder preflight — it never skips.
# `--check` tells you whether the copy is stale.
#
# NOTHING SECRET IS PRINTED. The key goes from `kubectl get secret` into the Gitea secrets API
# through shell variables and nowhere else. The script prints only names, sizes and the
# certificate's subject and expiry (public data).
#
# NEEDS: kubectl with a context for the cluster (default k3s-prod), jq, curl, openssl, base64,
# and the Gitea credential `git` already has stored for the host — the same requirements as
# infra/gitea/conformance-trigger-kubeconfig.sh, run from the same place.
set -euo pipefail
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
NS=buildkit; SECRET=buildkit-client-ci; KCTX="${KCTX:-k3s-prod}"
CHECK=0
for a in "$@"; do
case "$a" in
--check) CHECK=1 ;;
-h|--help) sed -n '2,/^set -euo/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown argument: $a" >&2; exit 2 ;;
esac
done
die() { echo "ERROR: $*" >&2; exit 1; }
for t in kubectl jq curl openssl base64 git; do command -v "$t" >/dev/null || die "missing required tool: $t"; done
echo "==> reading Secret ${NS}/${SECRET} (context ${KCTX}; values not shown)"
js="$(kubectl --context "$KCTX" -n "$NS" get secret "$SECRET" -o json)" \
|| die "cannot read ${NS}/${SECRET} — is platform/buildkit deployed and the Certificate Ready?"
CA="$(jq -r '.data["ca.crt"] // empty' <<<"$js" | base64 -d)"
CRT="$(jq -r '.data["tls.crt"] // empty' <<<"$js" | base64 -d)"
KEY="$(jq -r '.data["tls.key"] // empty' <<<"$js" | base64 -d)"
unset js
[ -n "$CA" ] && [ -n "$CRT" ] && [ -n "$KEY" ] || die "${NS}/${SECRET} lacks ca.crt/tls.crt/tls.key"
# Sanity before anything is written: the leaf is a CLIENT cert issued by this CA, and the key
# belongs to it. A wrong pairing would otherwise surface only as a TLS failure in CI.
openssl verify -CAfile <(printf '%s\n' "$CA") <(printf '%s\n' "$CRT") >/dev/null 2>&1 \
|| die "tls.crt does not verify against ca.crt"
openssl x509 -noout -ext extendedKeyUsage <<<"$CRT" 2>/dev/null | grep -q "TLS Web Client Authentication" \
|| die "tls.crt is not a client-auth certificate"
[ "$(openssl x509 -noout -pubkey <<<"$CRT" | openssl sha256)" = "$(openssl pkey -pubout <<<"$KEY" 2>/dev/null | openssl sha256)" ] \
|| die "tls.key does not match tls.crt"
echo " $(openssl x509 -noout -subject <<<"$CRT") / $(openssl x509 -noout -enddate <<<"$CRT")"
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"; unset creds
[ -n "$U" ] && [ -n "$P" ] || die "no stored credential for $HOST"
echo "==> authenticating to Gitea as '${U}' (secret not shown)"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
API="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets"
if [ "$CHECK" -eq 1 ]; then
# Gitea never returns a secret's value, only its name and creation time. "Stale" therefore
# means: the secret was created before the certificate's current notBefore.
names="$(api "$API" | jq -c '[.[]? | {name, created_at}]')"
nb="$(date -u -d "$(openssl x509 -noout -startdate <<<"$CRT" | cut -d= -f2)" +%s)"
rc=0
for n in BUILDKIT_CA BUILDKIT_CLIENT_CERT BUILDKIT_CLIENT_KEY; do
c="$(jq -r --arg n "$n" '.[] | select(.name==$n) | .created_at' <<<"$names")"
if [ -z "$c" ]; then echo " MISSING $n"; rc=1
elif [ "$(date -u -d "$c" +%s)" -lt "$nb" ]; then echo " STALE $n (stored $c, certificate re-issued since)"; rc=1
else echo " current $n (stored $c)"; fi
done
exit "$rc"
fi
put() { # name value
local code
code="$(api -o /dev/null -w '%{http_code}' -X PUT -H 'Content-Type: application/json' \
-d "$(jq -n --arg d "$2" '{data:$d}')" "${API}/$1")"
[ "$code" = 201 ] || [ "$code" = 204 ] || die "PUT $1 failed: HTTP ${code}"
echo " stored $1 (${#2} bytes, value not shown)"
}
echo "==> storing repository Actions secrets"
put BUILDKIT_CA "$CA"
put BUILDKIT_CLIENT_CERT "$CRT"
put BUILDKIT_CLIENT_KEY "$KEY"
unset CA CRT KEY
# VERIFY BY READING BACK (names only — Gitea never returns values). curl -sS exits 0 on an HTTP
# error, which is how set-ci-token.sh once claimed a store that never happened.
names="$(api "$API" | jq -r '[.[]?.name] | join(" ")')"
for n in BUILDKIT_CA BUILDKIT_CLIENT_CERT BUILDKIT_CLIENT_KEY; do
grep -qw "$n" <<<"$names" || die "stored $n but it is NOT listed (${names}) — refusing to claim success"
done
echo "==> stored and VERIFIED present. Re-run the images workflow (workflow_dispatch) to use them."
+74
View File
@@ -0,0 +1,74 @@
# platform/buildkit — rootless BuildKit, the CI image builder
The builder behind [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). It
builds `toolbox`, `agent-runner` and `conformance-runner`, and on `main` pushes them to the
Gitea registry. Flux reconciles it as the Kustomization `buildkit`
([`clusters/base/buildkit.yaml`](../../clusters/base/buildkit.yaml), `dependsOn: platform`).
## Why it exists (operator decision on PR #81, 2026-10-29)
CI jobs run as containers on the act runner's privileged DinD sidecar, and **cannot reach any
Docker daemon**. act_runner 0.2.11 forces `docker_host: "-"` for a `tcp://` DinD, and gitea/act
strips every bind mount when `valid_volumes` is empty. Measured on PR #81's first run. Giving jobs
that daemon would make every branch push root on the node. So the rule is: **no job may reach a
privileged daemon.** Builds go to this rootless daemon over mTLS instead.
## Shape
| Piece | What it is |
|---|---|
| `deployment.yaml` | `moby/buildkit:v0.34.1-rootless`, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The `securityContext` is commented line by line with how each item was measured |
| `certificates.yaml` | cert-manager: a self-signed CA used only here, plus the server cert `buildkitd-tls` and the CI client cert `buildkit-client-ci` |
| `service.yaml` | ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`, mTLS only. Never on the Gateway |
| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and, for the registry push, the Gateway (`ingress` entity) **plus** Gitea's backend identity on :3000 — Cilium's Gateway re-checks this pod's egress against the backend it proxies to, and without that rule answers `403 Access denied` itself (measured 2026-10-30, `decisions/2026-10-30-gitea-admit-buildkit.md`) |
| `configmap.yaml` | `buildkitd.toml`: cache GC capped at 4 GB, because the cache is an `emptyDir` on node disk |
| `versions.env` | The CI side: pinned `buildx` (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource |
## Admission
**No Kyverno exception is needed.**
- Every ClusterPolicy that is actually applied (the list in
[`platform-config/kustomization.yaml`](../../platform-config/kustomization.yaml)) matches only
namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`. This namespace, like
`gitea-actions`, does not carry that label.
- The job-shape policies (`pm-agent-job-shape`, `conformance-trigger-job-shape`) match only their
own namespaces.
- There is no cluster-wide Pod Security default. This namespace is deliberately left without a
`pod-security` label, because PSS `baseline` forbids the `Unconfined` seccomp and AppArmor
profiles that rootless BuildKit needs.
## Prerequisites (operator, in order)
1. **Unprivileged user namespaces on the nodes.** Already true. Measured 2026-10-29 on all three
Debian 13.7 nodes (kernel 7.1.8): `kernel.unprivileged_userns_clone = 1`,
`user.max_user_namespaces = 127206`, and no Ubuntu-style AppArmor userns restriction.
2. **Merge.** Flux applies this directory. Check that the Kustomization `buildkit` is Ready and
that all three Certificates in namespace `buildkit` are READY.
3. **Copy the client certificate to CI.** Run
[`infra/gitea/buildkit-client-secret.sh`](../../infra/gitea/buildkit-client-secret.sh).
Re-run it yearly, after renewal; `--check` reports whether the copy is stale.
4. **Restart buildkitd after the server certificate renews.** It reads the certificate at start,
and cert-manager renews it 30 days before expiry, so a restart inside that window is needed.
Until step 3 is done, the `images` workflow fails at its builder preflight with a message naming
the cause. It never skips.
## What a build step can do
The daemon runs with `--oci-worker-no-process-sandbox`, so build steps share its PID namespace.
buildkitd warns about this itself at startup.
- **A build step CAN** signal or ptrace the daemon and read what it can read, including this
pod's TLS key and the build cache.
- **A build step CANNOT** reach the node. Everything runs as uid 1000 in a user namespace,
unprivileged, with no host mounts.
- **Who can run one:** anyone holding the client certificate, i.e. every CI job, including pull
requests.
## Local equivalent
The pinned image runs under rootless podman with the same restrictions:
`--cap-drop ALL --cap-add SETUID --cap-add SETGID --security-opt seccomp=unconfined --read-only`,
plus tmpfs at the same five paths. That is how the `securityContext` was measured. Rootless podman
maps only 65,537 uids, so the image's `/etc/subuid` (`user:100000:65536`) must be overridden with
two ranges that fit: `user:1:999` and `user:1001:64535`.
+115
View File
@@ -0,0 +1,115 @@
---
# mTLS for buildkitd, issued by cert-manager from a CA that exists ONLY for this purpose.
#
# buildkit-selfsigned (Issuer) --signs--> buildkit-ca (CA Certificate, 10y)
# buildkit-ca (Issuer) --signs--> buildkitd-tls (server, the daemon presents it)
# --signs--> buildkit-client-ci (client, CI presents it)
#
# buildkitd verifies EVERY client certificate against this CA (--tlscacert), so holding a client
# certificate signed by it is what "may build" means. Nothing else in the cluster trusts this CA,
# and it trusts nothing else.
#
# THE CLIENT CERTIFICATE REACHES CI AS ACTIONS SECRETS, not through the cluster: CI jobs hold no
# cluster credential (R-SEC-02). infra/gitea/buildkit-client-secret.sh copies ca.crt/tls.crt/
# tls.key from the Secret below into the repository secrets BUILDKIT_CA, BUILDKIT_CLIENT_CERT and
# BUILDKIT_CLIENT_KEY without printing them. Re-run it after every renewal (see renewBefore).
#
# WHY A PRIVATE CA AND NOT LET'S ENCRYPT. The platform's TLS rule (R-TLS-01/02, docs/08,
# ADR-0006: ACME DNS-01) is about EXTERNALLY REACHABLE services; ADR-0006 itself says "mTLS between
# workloads is a separate concern … not covered here", and a public CA cannot issue for a
# `*.svc.cluster.local` name or a client-auth certificate anyway. buildkitd is ClusterIP-only. Each
# Certificate below is therefore annotated `homelab.keithjackson.dev/tls-scope: workload-mtls`,
# which the L0 check tls-lifecycle-automated accepts ONLY if the certificate proves it: no
# Gateway serves its Secret, every name is cluster-internal, and the issuer is a namespaced
# selfSigned/CA Issuer whose CA is itself issued here. Issuance and renewal stay unattended.
#
# ROTATION. cert-manager renews both leaves `renewBefore` their expiry and rewrites the Secrets.
# * server: buildkitd reads its certificate at START, so the pod must restart inside the 30-day
# renewal window (a node reboot by kured, a Deployment rollout, or a manual restart).
# * client: the Actions secrets are a COPY; re-run the script. Until then CI fails loudly at
# its TLS preflight (.gitea/workflows/scripts/images.sh), never silently.
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: buildkit-selfsigned
namespace: buildkit
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: buildkit-ca
namespace: buildkit
annotations:
homelab.keithjackson.dev/tls-scope: workload-mtls # see the header: R-TLS-01 is about external TLS
spec:
isCA: true
commonName: buildkit-ca
secretName: buildkit-ca
duration: 87600h # 10 years: rotating the CA means re-issuing everything and re-copying
privateKey:
algorithm: ECDSA
size: 256
issuerRef:
name: buildkit-selfsigned
kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: buildkit-ca
namespace: buildkit
spec:
ca:
secretName: buildkit-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: buildkitd-tls
namespace: buildkit
annotations:
homelab.keithjackson.dev/tls-scope: workload-mtls
spec:
secretName: buildkitd-tls
commonName: buildkitd.buildkit.svc.cluster.local
dnsNames:
- buildkitd.buildkit.svc.cluster.local # what CI dials (platform/buildkit/versions.env)
- buildkitd.buildkit.svc
- buildkitd
duration: 8760h
renewBefore: 720h
usages:
- server auth
- digital signature
privateKey:
algorithm: ECDSA
size: 256
rotationPolicy: Always
issuerRef:
name: buildkit-ca
kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: buildkit-client-ci
namespace: buildkit
annotations:
homelab.keithjackson.dev/tls-scope: workload-mtls
spec:
secretName: buildkit-client-ci
commonName: gitea-actions-image-pipeline
duration: 8760h
renewBefore: 720h
usages:
- client auth
- digital signature
privateKey:
algorithm: ECDSA
size: 256
rotationPolicy: Always
issuerRef:
name: buildkit-ca
kind: Issuer
+24
View File
@@ -0,0 +1,24 @@
---
# buildkitd.toml. Validated against the pinned image (v0.34.1-rootless): `buildctl debug workers
# -v` reports "Maximum used space: 4.29GB / Reserved space: 1.07GB" for the default policy.
#
# WHY 4GB. The cache lives in an emptyDir on the NODE's disk (deployment.yaml), and the nodes had
# 7-9 GB free when this was written. The largest image (toolbox) is ~1.4 GB uncompressed; the
# three images share their Fedora base. The emptyDir's sizeLimit sits above this cap, so GC
# trims the cache well before the kubelet would evict the pod for exceeding it.
apiVersion: v1
kind: ConfigMap
metadata:
name: buildkitd-config
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
data:
buildkitd.toml: |
[worker.oci]
enabled = true
gc = true
maxUsedSpace = "4GB"
reservedSpace = "1GB"
[worker.containerd]
enabled = false
+142
View File
@@ -0,0 +1,142 @@
---
# buildkitd, ROOTLESS. The builder behind .gitea/workflows/images.yaml; see README.md.
#
# THE SECURITY CONTEXT IS THE BOUNDARY, AND EVERY LINE OF IT WAS MEASURED (2026-10-29, the pinned
# image under podman with the same restrictions; decisions/2026-10-29-image-pipeline.md):
# * privileged: false — never. This is the whole point of the operator's decision on PR #81:
# the alternative was handing CI the privileged DinD daemon, i.e. root on the node.
# * runAsUser/Group 1000 — the image's `user`; rootlesskit maps it to uid 0 INSIDE a user
# namespace only. The nodes allow that (measured: kernel.unprivileged_userns_clone = 1,
# user.max_user_namespaces = 127206 on all three Debian 13 nodes).
# * capabilities: drop ALL, add SETUID + SETGID — the image's newuidmap/newgidmap are NOT
# setuid; they carry FILE capabilities, which need SETUID/SETGID in the bounding set.
# Measured: with drop ALL and nothing added, rootlesskit fails "newuidmap ... operation not
# permitted" and the daemon never starts.
# * allowPrivilegeEscalation: true — required for the same reason: no_new_privs (what `false`
# sets) stops a file capability from ever taking effect. It does NOT grant privileges the
# bounding set above does not contain.
# * seccompProfile: Unconfined — measured: with the default seccomp profile the daemon starts
# but every RUN step fails (the runtime cannot create the build step's namespaces/mounts).
# * appArmorProfile: Unconfined — upstream's documented requirement (buildkit
# examples/kubernetes, *.rootless.yaml); the nodes run AppArmor (enabled=Y) and containerd's
# default profile denies mount. NOT measured here: the test host runs SELinux, not AppArmor.
# * readOnlyRootFilesystem: true — measured to work with the five emptyDirs below.
#
# --oci-worker-no-process-sandbox: build steps share the daemon's PID namespace instead of getting
# their own (creating one needs an unmasked /proc, i.e. more privilege). Consequence, stated by
# buildkitd itself at startup: a build step can signal or ptrace the daemon and read what the
# daemon can read — including this pod's TLS key and the build cache. So a build step is trusted
# at the level of "may impersonate this builder", which is the level anyone holding the client
# certificate already has. It can NOT reach the node: everything still runs as uid 1000 in a user
# namespace, unprivileged, with no host mounts.
apiVersion: apps/v1
kind: Deployment
metadata:
name: buildkitd
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
spec:
replicas: 1
strategy:
type: Recreate # one builder, one cache; never two daemons fighting over a node's disk
selector:
matchLabels:
app.kubernetes.io/name: buildkitd
template:
metadata:
labels:
app.kubernetes.io/name: buildkitd
spec:
automountServiceAccountToken: false # the builder needs no Kubernetes API at all
enableServiceLinks: false
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000 # makes the 0440 TLS Secret below readable by uid 1000, and nobody else
containers:
- name: buildkitd
# Pinned by digest (the multi-arch index). The kubernetes manager in renovate.json5
# tracks platform/**, so Renovate proposes bumps; the CI client (buildx, pinned in
# versions.env) talks to any recent daemon.
image: docker.io/moby/buildkit:v0.34.1-rootless@sha256:f68752a5a94acfcdad2d07867b7075601a421638e17fc5f28efc1c814d0c4f89
args:
- --addr
- unix:///run/user/1000/buildkit/buildkitd.sock # local only: the probes below
- --addr
- tcp://0.0.0.0:1234 # CI, mTLS only
- --tlscacert
- /certs/ca.crt # client certs are VERIFIED against this CA: no cert, no build
- --tlscert
- /certs/tls.crt
- --tlskey
- /certs/tls.key
- --config
- /etc/buildkit/buildkitd.toml
- --oci-worker-no-process-sandbox
ports:
- name: buildkit
containerPort: 1234
protocol: TCP
readinessProbe:
exec:
command: ["buildctl", "debug", "workers"]
initialDelaySeconds: 5
periodSeconds: 30
livenessProbe:
exec:
command: ["buildctl", "debug", "workers"]
initialDelaySeconds: 30
periodSeconds: 60
timeoutSeconds: 10
securityContext:
privileged: false
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
allowPrivilegeEscalation: true
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
add: ["SETUID", "SETGID"]
seccompProfile:
type: Unconfined
appArmorProfile:
type: Unconfined
resources:
# Build steps (dnf, npm, pip) run INSIDE this container's cgroup, so the limit covers
# the daemon and the build together.
requests:
cpu: 250m
memory: 512Mi
ephemeral-storage: 2Gi
limits:
memory: 4Gi
volumeMounts:
- { name: certs, mountPath: /certs, readOnly: true }
- { name: config, mountPath: /etc/buildkit, readOnly: true }
- { name: cache, mountPath: /home/user/.local/share/buildkit }
- { name: run, mountPath: /run/user }
- { name: tmpdir, mountPath: /home/user/.local/tmp } # the image's TMPDIR
- { name: tmp, mountPath: /tmp }
- { name: clicfg, mountPath: /home/user/.docker } # buildctl (the probes) writes here
volumes:
- name: certs
secret:
secretName: buildkitd-tls # cert-manager: ca.crt + tls.crt + tls.key
defaultMode: 0440
- name: config
configMap:
name: buildkitd-config
- name: cache
emptyDir:
sizeLimit: 6Gi # GC caps use at 4GB (configmap.yaml); this is the backstop
- name: run
emptyDir: {}
- name: tmpdir
emptyDir: {}
- name: tmp
emptyDir: {}
- name: clicfg
emptyDir: {}
+13
View File
@@ -0,0 +1,13 @@
---
# platform/buildkit — rootless BuildKit, the CI image builder. Reconciled by the Flux
# Kustomization `buildkit` (clusters/base/buildkit.yaml). See README.md.
# versions.env is NOT a resource: it pins the CI-side client (buildx) and the address CI dials.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- certificates.yaml
- configmap.yaml
- deployment.yaml
- service.yaml
- networkpolicy.yaml
+26
View File
@@ -0,0 +1,26 @@
---
# buildkit — the image builder for CI (.gitea/workflows/images.yaml). Operator decision on PR #81
# (2026-10-29): images are built by a ROOTLESS BuildKit daemon reached over mTLS, and NO CI job
# may reach a privileged daemon. See README.md for the whole design.
#
# A NAMESPACE OF ITS OWN, deliberately NOT `gitea-actions`:
# * The act-runner namespace exists to hold a PRIVILEGED DinD sidecar. Putting the builder
# beside it would put a TLS server key next to a privileged container for no benefit.
# * Its NetworkPolicies, Secrets and quota then belong to one component, and the ingress rule
# ("only the act-runner pod may connect") can be written from the outside.
#
# NO `app-namespace` LABEL, like gitea-actions: this is CI plumbing, not a user-facing app. That
# is also why NO Kyverno exception is needed — every applied ClusterPolicy
# (platform-config/kustomization.yaml) matches only namespaces labelled
# homelab.keithjackson.dev/app-namespace: "true", and none of the job-shape policies match here.
#
# NO pod-security.kubernetes.io LABEL, on purpose. Rootless BuildKit needs seccomp AND AppArmor
# `Unconfined` (measured — see deployment.yaml), which PSS `baseline` forbids; there is no
# cluster-wide PSA default here, so an unlabelled namespace admits it. It is NOT privileged and
# the deployment's securityContext is the boundary — read it before relaxing anything.
apiVersion: v1
kind: Namespace
metadata:
name: buildkit
labels:
homelab.keithjackson.dev/ci: buildkit
+109
View File
@@ -0,0 +1,109 @@
---
# Default-deny for the namespace, then two narrow allows.
#
# WHO MAY CONNECT — AND WHY THE ACT-RUNNER POD IS THE RIGHT SOURCE (measured 2026-10-29).
# CI jobs do not run as pods. act_runner starts each job as a container on its DinD sidecar, on a
# docker bridge network INSIDE the act-runner pod's network namespace, and dockerd masquerades it:
# kubectl -n gitea-actions exec deploy/act-runner -c dind -- iptables -t nat -S POSTROUTING
# -A POSTROUTING -s 172.18.0.0/16 ! -o br-c0f13d041ad6 -j MASQUERADE
# -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
# So a job's connection leaves with the act-runner POD's IP and therefore Cilium's identity for
# that pod (namespace gitea-actions, app.kubernetes.io/name=act-runner). That is exactly what the
# ingress rule names. It cannot distinguish one job from another, or a job from act_runner
# itself — per-job authentication is the mTLS client certificate, not this policy. What the
# policy adds is that NOTHING ELSE in the cluster can even open the port.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: buildkit
labels:
homelab.keithjackson.dev/baseline: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress: []
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: buildkitd-allow-ci-ingress
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: buildkitd
ingress:
- fromEndpoints:
- matchLabels:
io.kubernetes.pod.namespace: gitea-actions
app.kubernetes.io/name: act-runner
toPorts:
- ports:
- { port: "1234", protocol: TCP }
---
# WHAT A BUILD NEEDS TO REACH. Build steps download (dnf mirrors, GitHub releases, dl.k8s.io,
# npm, PyPI, Fedora/Docker registries) and the result is pushed to the Gitea registry.
# * world 443/80: the downloads. 80 because Fedora's mirror list includes plain-http mirrors
# (dnf verifies package GPG signatures, not transport).
# * ingress (the Gateway's Envoy, reserved:ingress): the push. gitea.home.keithjackson.dev
# resolves in-cluster to the Gateway VIP (split-horizon CoreDNS), and Cilium hands that
# traffic to the Gateway, whose identity is `ingress` — the same identity gitea's own
# ingress allow admits.
# * gitea's BACKEND identity, :3000 only: ALSO the push, and the half that `ingress` alone does
# not cover. Cilium >= 1.16.8/1.17.2 (CVE-2025-30162; this cluster runs 1.20.1) makes the
# Gateway's Envoy re-evaluate THIS pod's egress policy against the identity of the backend it
# resolves the request to (enforce_policy_on_l7lb). `toEntities: ingress` gets the TCP
# connection to Envoy; without this rule Envoy then refuses the HTTP request itself with
# `403 Access denied`, and Gitea never sees it. Measured 2026-10-30 (images run 337 failed
# this way): Hubble on buildkitd's node logged `buildkit/buildkitd (ingress) ->
# gateway/cilium-gateway-shared:443 http-request DROPPED`, traffic_direction EGRESS, and
# Gitea's router log had no entry for it — while the same unauthenticated GET /v2/ from the
# act-runner pod (egress unrestricted) reached Gitea and got its normal 401. It is the same
# fix catalog/render.py applies for every SSO app's hop to Keycloak (commit 68690e5).
# Gitea's INGRESS needs nothing: the request arrives there as `reserved:ingress`, already
# admitted. Do not add buildkit to catalog/gitea.yaml's inClusterClients — that would open
# the ClusterIP path from build code to Gitea, which nothing here uses.
# Nothing else in the cluster: not the API server, not other pods. A build step is code from a
# pull request.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: buildkitd-allow-egress
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: buildkitd
egress:
- toEntities:
- world
toPorts:
- ports:
- { port: "443", protocol: TCP }
- { port: "80", protocol: TCP }
- toEntities:
- ingress
- toEndpoints:
- matchLabels:
io.kubernetes.pod.namespace: gitea
app.kubernetes.io/name: gitea
toPorts:
- ports:
- { port: "3000", protocol: TCP }
+19
View File
@@ -0,0 +1,19 @@
---
# The address CI dials: tcp://buildkitd.buildkit.svc.cluster.local:1234 (platform/buildkit/
# versions.env), mTLS only. ClusterIP — never exposed on the Gateway.
apiVersion: v1
kind: Service
metadata:
name: buildkitd
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: buildkitd
ports:
- name: buildkit
port: 1234
targetPort: buildkit
protocol: TCP
+15
View File
@@ -0,0 +1,15 @@
# platform/buildkit/versions.env — the CI side of the builder. Sourced by
# .gitea/workflows/scripts/images.sh. NOT a Kubernetes resource (kustomization.yaml omits it).
#
# The CLIENT. CI drives the daemon with `buildx` (remote driver) rather than bare `buildctl`
# because the image pipeline needs `bake`'s `target:` contexts: on a pull request agent-runner is
# built FROM the toolbox built in the same run, which never exists in any registry.
# Pinned with the sha256 from the release's own checksums.txt
# (https://github.com/docker/buildx/releases/download/v0.38.1/checksums.txt); the pipeline
# refuses a download that does not match. Bump BOTH lines together.
BUILDX_VERSION=0.38.1
BUILDX_SHA256=e25d5cd675092bb5d1d03632fdc46cdf3fede7b514dc5f023a105e00d3ce5ad4
# The DAEMON's address and the name its certificate carries (platform/buildkit/certificates.yaml).
BUILDKIT_HOST=tcp://buildkitd.buildkit.svc.cluster.local:1234
BUILDKIT_SERVERNAME=buildkitd.buildkit.svc.cluster.local
+3 -3
View File
@@ -47,9 +47,9 @@ Both are deliberate, and both fail **visibly** rather than silently:
1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
`PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder until the first
publish. `.gitea/workflows/images.yaml` builds `agent-runner` with
`infra/agent/build-image.sh` (the same script, and the same in-image verification, you can
run locally), and on a push to `main` publishes it to
publish. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
(`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that
`infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to
`gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
Renovate then follows the `main` tag through the `# renovate-digest:` marker above **both**
lines and moves them together in one pull request. Nobody builds-and-pushes this image from