From e5391098302e7709e64b779fc1dbcb8b1551c778 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Thu, 29 Oct 2026 21:27:44 -0500 Subject: [PATCH 1/4] feat(images): build, verify and publish the three images in CI; Renovate carries the digests Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a workstation build/push. Nothing had ever been pushed to the Gitea registry. - .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as sha- + main, reading the digest back from the registry. Only changed images build. - Each image's own build-image.sh does the build and in-image verification, now engine-neutral (DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh. - Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex manager for them. The worker Job template is deliberately untracked (it would loop). - renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION. - Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh. Co-Authored-By: Claude Opus 5.5 --- .gitea/workflows/README.md | 33 ++ .gitea/workflows/images.yaml | 174 +++++++++++ .gitea/workflows/renovate.yaml | 18 +- .gitea/workflows/scripts/images.sh | 291 ++++++++++++++++++ conformance/incluster/README.md | 11 +- conformance/incluster/build-image.sh | 84 ++--- .../alert-rules-unit-tested/check.sh | 13 +- conformance/versions.env | 39 ++- decisions/2026-10-29-image-pipeline.md | 222 +++++++++++++ docs/plans/pm-agent-issue-to-pr.md | 10 +- infra/agent/build-image.sh | 140 +++++---- infra/agent/probe-validation.sh | 4 +- infra/agent/versions.env | 14 +- infra/agent/worker-job-template.yaml | 6 + infra/devcontainer/README.md | 16 + infra/devcontainer/build-image.sh | 108 +++++++ infra/devcontainer/dev.sh | 20 +- infra/gitea/conformance-trigger-kubeconfig.sh | 11 +- infra/gitea/registry-retention.sh | 130 ++++++++ platform/pm-agent/README.md | 21 +- platform/pm-agent/configmap.yaml | 13 +- platform/pm-agent/deployment.yaml | 7 +- renovate.json5 | 95 +++++- runbooks/pm-agent.md | 4 +- 24 files changed, 1274 insertions(+), 210 deletions(-) create mode 100644 .gitea/workflows/images.yaml create mode 100755 .gitea/workflows/scripts/images.sh create mode 100644 decisions/2026-10-29-image-pipeline.md create mode 100755 infra/devcontainer/build-image.sh create mode 100755 infra/gitea/registry-retention.sh diff --git a/.gitea/workflows/README.md b/.gitea/workflows/README.md index 952e6ce..c42412c 100644 --- a/.gitea/workflows/README.md +++ b/.gitea/workflows/README.md @@ -115,14 +115,47 @@ CI. That is a real coverage gap, not a phase. | variable | `REQUIRE_CLUSTER_TESTS` | `'true'` (Phase 1+) makes unverified cluster-needing touched reqs a hard block | | secret | `CONFORMANCE_TRIGGER_KUBECONFIG` | narrow creds to create/await the in-cluster Chainsaw Job (not cluster-admin) | +## How images are built + +This repo's three images — `toolbox` (`infra/devcontainer/`), `agent-runner` (`infra/agent/`, +built `FROM` the toolbox) and `conformance-runner` (`conformance/incluster/`) — are built, +verified and published by [`images.yaml`](images.yaml), never from a workstation +(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh); each +image is built by **its own** `build-image.sh`, the same script a developer runs locally, so the +in-image verification (non-root uid, read-only rootfs, tool presence, `PM_SELFTEST=1 +pm-dispatch`, the `conformance-clone` refusals) exists once. + +| Trigger | What happens | +|---|---| +| pull request touching an image's inputs | build + verify only. Nothing is pushed and **no registry credential is referenced** | +| push to `main` touching an image's inputs | build + verify + publish to `gitea.home.keithjackson.dev/home-network/` as `sha-` (immutable) and `main` (moving); the **registry-returned** digest is in the job summary | +| `workflow_dispatch` | the same as a push when run on `main` (all images by default); verify-only elsewhere | + +Only images whose inputs changed are built. The publish job **never writes to git**: Renovate +follows each image's `main` tag through a `# renovate-digest:` marker next to every digest +reference (`platform/pm-agent/`, `infra/agent/versions.env`, `conformance/versions.env`) and +opens the bump PR, which goes through the gate like anything else. A PR opened with the Actions +token would not trigger the required gate, which is why it is Renovate's job and not this one's. + +Credential: the repository secret **`REGISTRY_PUSH_TOKEN`** (scope `write:package`, owned by an +account with package write on the org). The automatic Actions token cannot push in Gitea 1.24.3 +— the reasoning, with source citations, is in the header of `images.yaml`. Retention: +[`infra/gitea/registry-retention.sh`](../../infra/gitea/registry-retention.sh), an operator +action, keeps every digest git pins plus the newest tags. + +Cost on this runner: the job shares the single act_runner slot with the gate, and the DinD store +is a node `emptyDir`; the job prunes it on exit. + ## Files ``` .gitea/workflows/ merge-gate.yaml the pipeline (lint → l0 → l1 → conformance → decision) + images.yaml build/verify/publish this repo's container images (see above) scripts/ install-tools.sh installs the pinned tools per stage (from ci/versions.env) decide.sh the R-TEST-03 merge decision over job results + images.sh the image pipeline's driver (what to build, base, push, digest) README.md this file ``` diff --git a/.gitea/workflows/images.yaml b/.gitea/workflows/images.yaml new file mode 100644 index 0000000..3d027bc --- /dev/null +++ b/.gitea/workflows/images.yaml @@ -0,0 +1,174 @@ +# Container images — built, verified and published by CI, never from a workstation. +# +# ═══════════════════════════════════════════════════════════════════════════════════════════════ +# WHY THIS FILE EXISTS (operator rule, 2026-10-29) +# ═══════════════════════════════════════════════════════════════════════════════════════════════ +# "Container image creation, hosting and lifecycle must be part of CI/CD and the codebase, never a +# one-off build/push from a workstation." Before this workflow, three images existed only as +# workstation builds and NOTHING had ever been pushed to the Gitea registry: the Home-Network org's +# package list was empty, platform/pm-agent/ pinned an all-zero digest and sat in ImagePullBackOff, +# and conformance/versions.env carried an empty digest. Both build scripts said "publishing is an +# operator action" — a step with no log, no recorded inputs and no verification anyone could read. +# +# WHAT IT DOES — the logic is in scripts/images.sh; this file only wires triggers and secrets. +# pull_request -> build + VERIFY only. No push, and no registry credential is referenced. +# push to main -> build + verify + PUBLISH to gitea.home.keithjackson.dev/home-network/ +# dispatch -> same as a push when run on main; verify-only on any other ref. +# Images: toolbox (infra/devcontainer), agent-runner (infra/agent, FROM toolbox) and +# conformance-runner (conformance/incluster). Only images whose inputs changed are built. +# +# THE DIGEST LIFECYCLE — this workflow NEVER writes to git. +# 1. publish pushes `sha-` (immutable) and `main` (moving), and prints the digest the +# REGISTRY returns in the job summary. +# 2. Renovate (renovate.yaml) follows the `main` tag of each image through the +# `# renovate-digest:` markers next to every digest reference, and opens the bump PR. +# 3. That PR goes through the gate like any other change, and Flux deploys it. +# The bump PR is Renovate's and not this job's because a PR opened with the Actions token does not +# trigger workflows, so it could never pass the REQUIRED gate check; Renovate's token is a real +# user's. +# +# THE PUSH CREDENTIAL — `REGISTRY_PUSH_TOKEN`, a repository Actions secret, referenced BY NAME. +# The automatic Actions token CANNOT be used. Gitea 1.24.3 source, verified: +# * services/auth/basic.go (Basic.Verify) turns a task token into the pseudo-user +# `gitea-actions` (ActionsUserID -2) and sets NO token scope; +# * routers/api/packages/container/container.go (Authenticate, the /v2/token exchange every +# docker client uses) then requires read:package / write:package / all scope and answers 401; +# * services/context/package.go (determineAccessMode) has `// TODO: ActionUser permission check` +# and no branch for that user, so even past login it would get at most READ on a public org. +# So: a personal access token with scope `write:package`, minted by an account that has package +# WRITE on the Home-Network org (org owner, or a team with the Packages unit at write). A dedicated +# bot account in a packages-only team is the least-privilege choice. The username sent with it is +# not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER` +# is optional and only labels the login. +# +# THE RUNNER (apps/wave-1/gitea-actions.yaml): act_runner 0.2.11, one job at a time, Docker +# executor on a privileged `docker:27-dind` sidecar whose image store is an emptyDir on the node +# (no sizeLimit) and whose memory limit is 2Gi — shared by the job container AND every build +# container. The toolbox is ~1.4 GB uncompressed and agent-runner adds ~0.3 GB on top of it, so the +# cleanup step below is not optional: without it each run would leave ~3 GB on a node that had +# 7 GB free when this was written, and the next eviction would take the runner with it. +name: images + +on: + pull_request: + branches: [main] + paths: + - infra/devcontainer/Containerfile + - infra/devcontainer/versions.env + - infra/devcontainer/build-image.sh + - infra/agent/** + - conformance/incluster/Containerfile + - conformance/incluster/conformance-clone + - conformance/incluster/build-image.sh + - .gitea/workflows/images.yaml + - .gitea/workflows/scripts/images.sh + push: + branches: [main] + paths: + - infra/devcontainer/Containerfile + - infra/devcontainer/versions.env + - infra/devcontainer/build-image.sh + - infra/agent/** + - conformance/incluster/Containerfile + - conformance/incluster/conformance-clone + - conformance/incluster/build-image.sh + - .gitea/workflows/images.yaml + - .gitea/workflows/scripts/images.sh + workflow_dispatch: + inputs: + images: + description: "Which images: 'all', or a space-separated subset of: toolbox agent-runner conformance-runner" + required: false + default: "all" + +concurrency: + # Never cancel a publish half-way: a pushed `sha-` tag with no matching `main` move is + # recoverable (a rerun skips what is already published) but pointless to provoke. + group: images-${{ github.ref }} + cancel-in-progress: false + +jobs: + images: + runs-on: ubuntu-latest + # Measured on the PR that introduced this file — see its decisions entry. Generous because a + # cold DinD store downloads the Fedora base and every pinned tool on each run. + timeout-minutes: 75 + steps: + - name: Checkout + uses: actions/checkout@v4 + with: + fetch-depth: 0 # the per-image change detection diffs against the base commit + + # ── pull requests: build + verify, NO credential ─────────────────────────────────────── + - name: Build and verify (pull request — nothing is pushed) + if: github.event_name == 'pull_request' + env: + MODE: verify + IMAGES: auto + PR_BASE_REF: ${{ github.base_ref }} + run: | + set -euo pipefail + BASE_SHA="$(git merge-base "origin/${PR_BASE_REF}" HEAD)" + export BASE_SHA + ./.gitea/workflows/scripts/images.sh + + # ── non-main dispatch: verify only ────────────────────────────────────────────────────── + - name: Build and verify (dispatch off main — nothing is pushed) + if: github.event_name == 'workflow_dispatch' && github.ref != 'refs/heads/main' + env: + MODE: verify + IMAGES: ${{ inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images }} + run: ./.gitea/workflows/scripts/images.sh + + # ── main: publish ─────────────────────────────────────────────────────────────────────── + - name: Preflight - the push token must exist, and its absence must be LOUD + if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main' + env: + REGISTRY_PUSH_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }} + run: | + set -euo pipefail + if [ -z "${REGISTRY_PUSH_TOKEN:-}" ]; then + echo "::error::REGISTRY_PUSH_TOKEN is not set on this repository." >&2 + cat >&2 <<'MSG' + + Images cannot be published without a registry credential, and this step fails rather + than skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would + make "nothing was published" look exactly like "published". That is how this repo went + a month with images that existed only on one workstation. + + WHY NOT THE BUILT-IN ACTIONS TOKEN: Gitea 1.24.3 gives it no package scope at the + registry's /v2/token exchange (routers/api/packages/container/container.go, + Authenticate) — see the header of .gitea/workflows/images.yaml. + + TO FIX: as an account with package WRITE on the Home-Network org (ideally a dedicated + bot in a packages-only team), create an access token with scope `write:package` and + nothing else, then add it to this repository as the Actions secret + REGISTRY_PUSH_TOKEN. Optionally set the Actions variable REGISTRY_PUSH_USER to that + account's name. Do not commit the token anywhere. Then re-run this workflow + (workflow_dispatch on main, images: all). + MSG + exit 1 + fi + echo "REGISTRY_PUSH_TOKEN is present (value not printed)." + + - name: Build, verify and publish (main) + if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main' + env: + MODE: publish + IMAGES: ${{ github.event_name == 'workflow_dispatch' && (inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images) || 'auto' }} + BASE_SHA: ${{ github.event.before }} + REGISTRY_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }} + REGISTRY_USER: ${{ vars.REGISTRY_PUSH_USER }} + run: ./.gitea/workflows/scripts/images.sh + + # ── always: give the DinD store back ──────────────────────────────────────────────────── + # The runner's image store is a node emptyDir with no sizeLimit (see header). Everything this + # job built or pulled is removed; `prune -a` cannot remove the job's own image, because the + # job container is still running on it. + - name: Clean the DinD image store + if: always() + run: | + docker logout gitea.home.keithjackson.dev >/dev/null 2>&1 || true + docker image prune -af >/dev/null || true + docker builder prune -af >/dev/null || true + docker system df || true diff --git a/.gitea/workflows/renovate.yaml b/.gitea/workflows/renovate.yaml index c313aac..f635ebf 100644 --- a/.gitea/workflows/renovate.yaml +++ b/.gitea/workflows/renovate.yaml @@ -60,8 +60,9 @@ # entry's "What I did NOT verify" — that gap is stated, not papered over. # # OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE: -# `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository` and -# `read:user`. Set it as an Actions secret on this repository. Nothing in this repo mints, +# `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository`, +# `read:user` and (since 2026-10-29) `read:package`. Set it as an Actions secret on this +# repository. Nothing in this repo mints, # stores or references its value; it is consumed by name only. # Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a # visible red is the correct state for "automation configured but not credentialed". @@ -139,8 +140,9 @@ jobs: seven files claimed "Renovate maintains this pin". TO FIX: create a Gitea access token for the bot account with scopes - `write:repository` and `read:user`, then add it to this repository as the Actions - secret RENOVATE_TOKEN. Do not commit it anywhere. + `write:repository`, `read:user` and `read:package` (the last one lets Renovate read + this repo's own images in the Gitea registry), then add it to this repository as the + Actions secret RENOVATE_TOKEN. Do not commit it anywhere. MSG exit 1 fi @@ -179,6 +181,14 @@ jobs: RENOVATE_REQUIRE_CONFIG: "required" RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }} LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }} + # ADDED 2026-10-29: registry read access for this repo's OWN images (toolbox, + # agent-runner, conformance-runner — published by .gitea/workflows/images.yaml and + # followed through `# renovate-digest:` markers). The Gitea registry answers 401 to + # anonymous reads, and Renovate scopes the platform token to hostType `gitea`, which a + # docker lookup never matches — so without this rule every digest lookup fails. Same + # token; it therefore needs scope `read:package` in addition to the ones listed in the + # preflight above. Gitea ignores the username for token auth (services/auth/basic.go). + RENOVATE_HOST_RULES: '[{"hostType":"docker","matchHost":"gitea.home.keithjackson.dev","username":"renovate","password":"${{ secrets.RENOVATE_TOKEN }}"}]' - name: Summary if: always() diff --git a/.gitea/workflows/scripts/images.sh b/.gitea/workflows/scripts/images.sh new file mode 100755 index 0000000..2d83fb4 --- /dev/null +++ b/.gitea/workflows/scripts/images.sh @@ -0,0 +1,291 @@ +#!/usr/bin/env bash +############################################################################### +# .gitea/workflows/scripts/images.sh — the image pipeline's driver (images.yaml calls this). +# +# Builds, verifies and (on main) PUBLISHES the three images this repo ships: +# +# toolbox infra/devcontainer/ the pinned dev/gate toolchain +# agent-runner infra/agent/ FROM toolbox; the PM dispatcher + worker +# conformance-runner conformance/incluster/ the in-cluster L2/L3 Chainsaw runner +# +# Each image is built by ITS OWN build-image.sh — the same script a developer runs locally — +# so the in-image verification (non-root uid, read-only rootfs, tool presence, pm-dispatch +# selftest, conformance-clone refusals, ...) is written once and runs in both places. This +# file only decides WHAT to build, in which order, FROM which base, and where it goes. +# +# MODES (env MODE): +# verify build + verify, never push, holds NO credential. Pull requests. +# publish build + verify + push to ${REGISTRY}/${NAMESPACE}/ with tags +# `sha-<12 hex>` (immutable: never re-pushed) and `main` (moving; Renovate follows +# it). The digest recorded is the one the REGISTRY returns, read back after the push. +# +# WHAT GETS BUILT (env IMAGES): `auto` (default) diffs BASE_SHA..HEAD_SHA against each image's +# inputs; otherwise a space-separated list. A change to the pipeline itself (images.yaml, this +# file) builds everything in verify mode, and nothing extra in publish mode: re-publishing +# unchanged inputs would only mint new digests and churn Renovate PRs. +# +# THE agent-runner BASE — the one ordering rule, and the reason this is not three jobs: +# * toolbox built in this run -> agent-runner builds FROM it (in publish mode, FROM the +# digest the registry just returned for it). +# * verify mode, toolbox not changed -> the toolbox is built anyway as a base. A pull request +# holds no registry credential, so it cannot pull the published one. +# * publish mode, toolbox not changed -> AGENT_BASE_IMAGE from infra/agent/versions.env, +# i.e. the published toolbox digest that Renovate keeps current. If that is still the +# all-zero placeholder the toolbox has never been published, so it is built and published +# first (this is how the very first run bootstraps). +# +# LOOP-FREEDOM. Renovate turns each publish into a digest-bump PR. Those bumps land in +# platform/pm-agent/ (no image input), conformance/versions.env (no image input) and +# infra/agent/versions.env (agent-runner input -> ONE more agent-runner build, whose own bump +# lands in platform/pm-agent/ and stops). The worker Job template is deliberately NOT tracked: +# it is baked INTO agent-runner, so bumping it would rebuild agent-runner, mint a new digest, +# bump it again, forever. pm-dispatch overwrites that field at runtime anyway. +############################################################################### +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" +cd "${REPO_ROOT}" + +MODE="${MODE:-verify}" +IMAGES="${IMAGES:-auto}" +ENGINE="${ENGINE:-docker}" +REGISTRY="${REGISTRY:-gitea.home.keithjackson.dev}" +NAMESPACE="${NAMESPACE:-home-network}" +HEAD_SHA="${HEAD_SHA:-$(git rev-parse HEAD)}" +BASE_SHA="${BASE_SHA:-}" +SUMMARY="${GITHUB_STEP_SUMMARY:-/dev/null}" +ZERO_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000" + +ALL_IMAGES=(toolbox agent-runner conformance-runner) +SHA_TAG="sha-${HEAD_SHA:0:12}" +MOVING_TAG="main" + +log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; } +info() { printf ' %s\n' "$*"; } +die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; printf -- '- **ERROR**: %s\n' "$*" >>"${SUMMARY}"; exit 1; } + +case "${MODE}" in verify|publish) ;; *) die "MODE must be verify or publish, got '${MODE}'" ;; esac + +# --- inputs per image: a change under any of these rebuilds that image --------------------- +# conformance-runner takes its chainsaw/kubectl/fedora pins from infra/devcontainer/versions.env. +# conformance/versions.env is NOT an input (it only RECORDS the published digest), and must not +# become one: Renovate writes it after every publish, and an input there would loop. +inputs() { + case "$1" in + toolbox) echo "infra/devcontainer/Containerfile infra/devcontainer/versions.env infra/devcontainer/build-image.sh" ;; + agent-runner) echo "infra/agent/" ;; + conformance-runner) echo "conformance/incluster/Containerfile conformance/incluster/conformance-clone conformance/incluster/build-image.sh infra/devcontainer/versions.env" ;; + esac +} +PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh" + +declare -A WANT=() BUILT=() DIGEST=() SECS=() NOTE=() + +changed_any() { # -> 0 if BASE_SHA..HEAD_SHA touches any + # shellcheck disable=SC2086 + [ -n "$(git diff --name-only "${BASE_SHA}" "${HEAD_SHA}" -- "$@")" ] +} + +select_images() { + if [ "${IMAGES}" != "auto" ]; then + for i in ${IMAGES}; do + case " ${ALL_IMAGES[*]} " in *" ${i} "*) WANT[$i]=1; NOTE[$i]="requested" ;; *) die "unknown image '${i}'" ;; esac + done + return + fi + if [ -z "${BASE_SHA}" ] || [[ "${BASE_SHA}" =~ ^0+$ ]] || ! git cat-file -e "${BASE_SHA}^{commit}" 2>/dev/null; then + info "no usable diff base ('${BASE_SHA}'): building everything" + for i in "${ALL_IMAGES[@]}"; do WANT[$i]=1; NOTE[$i]="no diff base"; done + return + fi + info "diff base ${BASE_SHA} .. ${HEAD_SHA}" + local i + for i in "${ALL_IMAGES[@]}"; do + # shellcheck disable=SC2046 + if changed_any $(inputs "$i"); then WANT[$i]=1; NOTE[$i]="inputs changed"; fi + done + # shellcheck disable=SC2086 + if [ "${MODE}" = verify ] && changed_any ${PIPELINE_FILES}; then + for i in "${ALL_IMAGES[@]}"; do [ -n "${WANT[$i]:-}" ] || { WANT[$i]=1; NOTE[$i]="pipeline changed"; }; done + fi +} + +# --- registry helpers (publish mode only) ---------------------------------------------------- +REG_TOKEN="" +registry_token() { + [ -n "${REG_TOKEN}" ] && return 0 + REG_TOKEN="$(curl -fsS --config <(printf 'user = "%s:%s"\n' "${REGISTRY_USER}" "${REGISTRY_TOKEN}") \ + "https://${REGISTRY}/v2/token?service=container_registry&scope=*" | jq -r '.token // empty')" \ + || die "could not obtain a registry bearer token from https://${REGISTRY}/v2/token" + [ -n "${REG_TOKEN}" ] || die "registry returned no bearer token" +} +# Prints the manifest digest the REGISTRY serves for :; nothing if it is absent (404). +# Any other answer (401, 5xx, no digest header) is fatal: "cannot tell" must never be read as +# "not published", or the immutable-tag guard would be bypassed by an auth hiccup. +registry_digest() { + registry_token + local hdrs code digest + hdrs="$(curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "${REG_TOKEN}") \ + -H 'Accept: application/vnd.oci.image.manifest.v1+json' \ + -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \ + -H 'Accept: application/vnd.oci.image.index.v1+json' \ + -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \ + "https://${REGISTRY}/v2/${NAMESPACE}/$1/manifests/$2" | tr -d '\r')" \ + || die "registry unreachable while reading $1:$2" + code="$(awk 'NR==1{print $2}' <<<"${hdrs}")" + digest="$(awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' <<<"${hdrs}" | tail -1)" + case "${code}" in + 200) [[ "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]] || die "registry answered 200 for $1:$2 with no usable digest" + echo "${digest}" ;; + 404) ;; + *) die "registry answered HTTP ${code} for $1:$2 — cannot tell whether it exists" ;; + esac +} + +local_ref() { echo "${REGISTRY}/${NAMESPACE}/$1:ci-${HEAD_SHA:0:12}"; } + +build_one() { # [agent base] + local img="$1" base="${2:-}" t0 tag + tag="ci-${HEAD_SHA:0:12}" + t0=$(date +%s) + log "BUILD + VERIFY ${img}" + case "${img}" in + toolbox) + ENGINE="${ENGINE}" IMAGE="$(local_ref toolbox)" BUILD_REVISION="${HEAD_SHA}" \ + ./infra/devcontainer/build-image.sh ;; + agent-runner) + ENGINE="${ENGINE}" IMAGE_TAG="${tag}" AGENT_BASE_IMAGE="${base}" BUILD_REVISION="${HEAD_SHA}" \ + ./infra/agent/build-image.sh ;; + conformance-runner) + ENGINE="${ENGINE}" IMAGE_TAG="${tag}" BUILD_REVISION="${HEAD_SHA}" \ + ./conformance/incluster/build-image.sh ;; + esac + BUILT[$img]=1 + SECS[$img]=$(( $(date +%s) - t0 )) + info "${img}: built and verified in ${SECS[$img]}s" +} + +push_one() { # -> sets DIGEST[image] to the registry-returned digest + local img="$1" repo src existing pushed got t0 + repo="${REGISTRY}/${NAMESPACE}/${img}" + src="$(local_ref "${img}")" + t0=$(date +%s) + log "PUSH ${img} -> ${repo}:{${SHA_TAG},${MOVING_TAG}}" + existing="$(registry_digest "${img}" "${SHA_TAG}")" + if [ -n "${existing}" ]; then + # Immutable tag: never overwrite. A rebuild is not byte-reproducible, so re-pushing would + # silently change what `sha-` means after something may already have pinned it. + die "${repo}:${SHA_TAG} already exists (${existing}). Tags here are immutable; this run must not + replace it. (publish skips images that already exist at this commit BEFORE building them, + so reaching this line means the tag appeared mid-run.)" + fi + "${ENGINE}" tag "${src}" "${repo}:${SHA_TAG}" + "${ENGINE}" tag "${src}" "${repo}:${MOVING_TAG}" + pushed="$("${ENGINE}" push "${repo}:${SHA_TAG}" | tee /dev/stderr | grep -oE 'digest: sha256:[0-9a-f]{64}' | tail -1 | cut -d' ' -f2)" + "${ENGINE}" push "${repo}:${MOVING_TAG}" >/dev/null + # The AUTHORITATIVE digest is what the registry serves, read back — not what the client + # printed. They should agree; if they do not, nothing downstream may trust either. + got="$(registry_digest "${img}" "${SHA_TAG}")" + [ -n "${got}" ] || die "pushed ${repo}:${SHA_TAG} but the registry does not serve it" + [ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] \ + || die "${img}: client reported ${pushed} but the registry serves ${got}" + [ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \ + || die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})" + DIGEST[$img]="${got}" + SECS[$img]=$(( ${SECS[$img]:-0} + $(date +%s) - t0 )) + info "${img}: registry digest ${got}" +} + +# --- main --------------------------------------------------------------------------------- +log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}" +command -v "${ENGINE}" >/dev/null || die "${ENGINE} is not on PATH in the job container" +"${ENGINE}" info >/dev/null 2>&1 \ + || die "the ${ENGINE} daemon is not reachable from the job container — the runner did not + hand it a Docker socket (act_runner container.docker_host). Nothing can be built." +"${ENGINE}" version 2>/dev/null | sed -n '1,12p' || true +"${ENGINE}" system df || true + +if [ "${MODE}" = publish ]; then + : "${REGISTRY_TOKEN:?REGISTRY_TOKEN is required in publish mode}" + REGISTRY_USER="${REGISTRY_USER:-ci-image-publisher}" + printf '%s' "${REGISTRY_TOKEN}" | "${ENGINE}" login "${REGISTRY}" -u "${REGISTRY_USER}" --password-stdin >/dev/null \ + || die "docker login ${REGISTRY} failed — check REGISTRY_PUSH_TOKEN (scope write:package, owner has package write on the org)" +fi + +select_images + +# Publish idempotence: an image already published at THIS commit is not rebuilt. This is what +# makes a re-run after a half-finished publish safe — and it is the only way a rerun can +# finish, because the immutable sha tag must never be pushed twice. +if [ "${MODE}" = publish ]; then + for i in "${ALL_IMAGES[@]}"; do + [ -n "${WANT[$i]:-}" ] || continue + d="$(registry_digest "${i}" "${SHA_TAG}")" + if [ -n "${d}" ]; then + info "${i}: already published at ${SHA_TAG} (${d}); not rebuilding" + unset "WANT[$i]"; DIGEST[$i]="${d}"; NOTE[$i]="already published at this commit" + fi + done +fi + +# Decide the agent-runner base (see header). +AGENT_BASE="" +if [ -n "${WANT[agent-runner]:-}" ]; then + # shellcheck disable=SC1091 + PINNED_BASE="$(. ./infra/agent/versions.env; echo "${AGENT_BASE_IMAGE}")" + if [ "${MODE}" = verify ] && [ -z "${WANT[toolbox]:-}" ]; then + WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)" + elif [ "${MODE}" = publish ] && [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then + case "${PINNED_BASE}" in + *@"${ZERO_DIGEST}") WANT[toolbox]=1; NOTE[toolbox]="never published; required as agent-runner's base" ;; + *@sha256:*) AGENT_BASE="${PINNED_BASE}" ;; + *) die "infra/agent/versions.env AGENT_BASE_IMAGE='${PINNED_BASE}' is not digest-pinned; CI will not build on a floating base" ;; + esac + fi +fi + +yn() { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; } +for i in "${ALL_IMAGES[@]}"; do info "plan: ${i}: $(yn "${WANT[$i]:-}" BUILD skip) (${NOTE[$i]:-inputs unchanged})"; done + +if [ -n "${WANT[toolbox]:-}" ]; then + build_one toolbox + if [ "${MODE}" = publish ]; then push_one toolbox; fi +fi +if [ -n "${WANT[agent-runner]:-}" ]; then + if [ -z "${AGENT_BASE}" ]; then + if [ "${MODE}" = publish ]; then + AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}" + else + AGENT_BASE="$(local_ref toolbox)" + fi + fi + info "agent-runner base: ${AGENT_BASE}" + build_one agent-runner "${AGENT_BASE}" + if [ "${MODE}" = publish ]; then push_one agent-runner; fi +fi +if [ -n "${WANT[conformance-runner]:-}" ]; then + build_one conformance-runner + if [ "${MODE}" = publish ]; then push_one conformance-runner; fi +fi + +# --- summary ------------------------------------------------------------------------------ +{ + echo "## Images (${MODE})" + echo + echo "commit \`${HEAD_SHA}\`" + echo + echo "| image | built + verified | seconds | published digest (registry-returned) | note |" + echo "|---|---|---|---|---|" + for i in "${ALL_IMAGES[@]}"; do + d="${DIGEST[$i]:-}" + echo "| \`${i}\` | $(yn "${BUILT[$i]:-}" yes no) | ${SECS[$i]:--} | $(yn "${d}" "\`${REGISTRY}/${NAMESPACE}/${i}@${d}\`" -) | ${NOTE[$i]:-inputs unchanged} |" + done + echo + if [ "${MODE}" = publish ]; then + echo "Tags pushed: \`${SHA_TAG}\` (immutable) and \`${MOVING_TAG}\` (moving)." + echo "Nothing here writes to git. Renovate opens the digest-bump PR on its next run." + else + echo "Verify mode: nothing was pushed and no registry credential was present." + fi +} >>"${SUMMARY}" +log "done" diff --git a/conformance/incluster/README.md b/conformance/incluster/README.md index 5728e4b..3d306dc 100644 --- a/conformance/incluster/README.md +++ b/conformance/incluster/README.md @@ -61,10 +61,13 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02). > digest-pinned to the conformance-runner image, `command` exactly > `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and > `env`/`envFrom` were the same shape of hole and are closed in the same policy. -- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the - image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it - prints the exact `podman push --digestfile …` command for the operator instead. See - `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin. +- `build-image.sh` — builds the above and **verifies the toolchain by running it** inside the + image as uid 65532 with `--read-only`. It **never pushes**, and neither does anyone from a + workstation: CI (`.gitea/workflows/images.yaml`) runs this same script, and on a push to + `main` publishes the verified image under `sha-` and `main`. Renovate then moves + `CONFORMANCE_RUNNER_IMAGE_DIGEST` in `conformance/versions.env` onto the registry-returned + digest in a pull request (that file is a sensitive path, so the bump needs an + administrator's `/approve-override`). `--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the default verify stays offline so an air-gapped build is fully checked, not half-checked). diff --git a/conformance/incluster/build-image.sh b/conformance/incluster/build-image.sh index 54a8d28..c836484 100755 --- a/conformance/incluster/build-image.sh +++ b/conformance/incluster/build-image.sh @@ -1,37 +1,41 @@ #!/usr/bin/env bash ############################################################################### # conformance/incluster/build-image.sh — build the pinned `conformance-runner` -# image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1. +# image and prove it works. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1. # -# ./build-image.sh # build + verify + print the digest and push command +# ./build-image.sh # build + verify + print the image id # ./build-image.sh --no-verify # build only (skip the in-image tool proof) # ./build-image.sh --verify-clone # ALSO really clone this repo with conformance-clone. # # Opt-in because it needs the Gitea host reachable; the # # default verify is offline so an air-gapped build is # # still fully checked rather than half-checked. # ENGINE=docker ./build-image.sh # build with docker instead of podman +# IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this) # -# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is -# not an agent's to hold, and the pushed digest is what a fail-closed admission policy -# then trusts. So this builds locally, proves the toolchain works INSIDE the image as the -# non-root uid the policy demands, and prints the exact `podman push` line for a human. +# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either. +# Publishing is done by CI: .gitea/workflows/images.yaml calls this script, and on a push to +# `main` it pushes what this script built AND verified to the Gitea registry, records the +# registry-returned digest in the job summary, and Renovate then opens the pull request that +# moves conformance/versions.env onto that digest. So this script's job is to build and to +# prove the toolchain works INSIDE the image as the non-root uid the policy demands — the same +# proof locally and in CI. It runs under ENGINE=docker in CI (Docker-in-Docker), so nothing +# below may be podman-only. # # VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the # single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same # pins the toolbox gate runs with. Nothing in this file invents a version. # -# THE DIGEST CAVEAT — read it before pasting a digest into anything. -# The digest printed here is the LOCAL manifest digest. It is normally identical to the -# registry digest (podman pushes the manifest it already has rather than recompressing), -# but that is a property of the push, not a guarantee. So the push command below uses -# `--digestfile`: the digest the registry actually returns is the authoritative one, and -# that is the one that goes into conformance/versions.env. +# NO DIGEST IS PRINTED. A manifest digest does not exist until a registry holds the manifest, +# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at +# push time (.gitea/workflows/scripts/images.sh) and Renovate carries it into +# conformance/versions.env. ############################################################################### set -euo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_ROOT="$(cd "${HERE}/../.." && pwd)" ENGINE="${ENGINE:-podman}" +TAG_OVERRIDE="${IMAGE_TAG:-}" VERIFY=1 VERIFY_CLONE=0 @@ -64,9 +68,9 @@ source "${TOOLCHAIN_ENV}" # gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path # is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here. IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner" -# The tag is build convenience only (a human-readable handle for the push). Admission -# ignores it: only the digest is ever referenced from a Job. -IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}" +# The tag is a local handle only. CI re-tags what it publishes (git SHA + `main`), and +# admission ignores tags entirely: only the digest is ever referenced from a Job. +IMAGE_TAG="${TAG_OVERRIDE:-chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}}" IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}" log "Building ${IMAGE_REF}" @@ -76,7 +80,11 @@ info " CHAINSAW ${CHAINSAW_VERSION}" info " KUBECTL ${KUBECTL_VERSION}" info " FEDORA ${FEDORA_VERSION}" -"${ENGINE}" build \ +# Provenance: CI passes the commit it is building, so a pulled image names its source tree. +LABEL_ARGS=() +[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}") + +"${ENGINE}" build "${LABEL_ARGS[@]}" \ --file "${HERE}/Containerfile" \ --tag "${IMAGE_REF}" \ --build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \ @@ -184,46 +192,12 @@ if [ "${VERIFY_CLONE}" -eq 1 ]; then log "conformance-clone verified against the real repository" fi -# --- digest ------------------------------------------------------------------ -DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")" +# --- result ------------------------------------------------------------------ IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")" SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")" log "Built ${IMAGE_REF}" -info "local manifest digest ${DIGEST}" -info "image id ${IMAGE_ID}" -info "size $(( SIZE / 1024 / 1024 )) MiB" - -cat < sha256:.... THIS is the authoritative digest, not the local one above. - - # conformance/versions.env - CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)" - -WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml -is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not -${IMAGE_REPO}@sha256:. A tag reference is denied, -and so is any other repository path. - -TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard -\`${IMAGE_REPO}@sha256:*\` to be replaced with the -LITERAL digest once the image exists, closing the "anyone who can push to this path" gap. -That is a separate, human-reviewed PR against the security boundary. -================================================================================ -EOF +info "image id ${IMAGE_ID}" +info "size $(( SIZE / 1024 / 1024 )) MiB" +info "Not pushed. CI publishes this image from main (.gitea/workflows/images.yaml) and" +info "Renovate moves conformance/versions.env onto the registry-returned digest." diff --git a/conformance/tests/l0-static/alert-rules-unit-tested/check.sh b/conformance/tests/l0-static/alert-rules-unit-tested/check.sh index eb31f03..ae7c593 100755 --- a/conformance/tests/l0-static/alert-rules-unit-tested/check.sh +++ b/conformance/tests/l0-static/alert-rules-unit-tested/check.sh @@ -29,7 +29,8 @@ # THEN (1) promtool is a PINNED member of the toolchain: PROMTOOL_VERSION is declared in BOTH # infra/devcontainer/versions.env and ci/versions.env and the two AGREE (the repo's # lockstep rule), the Containerfile takes it as a build ARG, installs it from the -# pinned prometheus release and records it for provenance, dev.sh passes it, and the +# pinned prometheus release and records it for provenance, build-image.sh (the one +# recipe `dev.sh build` and CI both use) passes it, and the # CI runner installs it for the l0 stage; # (2) the promtool ACTUALLY IN USE reports exactly that pinned version — a unit test is # evidence only if the engine matches the one prod evaluates rules with; @@ -246,10 +247,12 @@ check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text, "infra/devcontainer/Containerfile does not record promtool in /etc/dev-toolchain-versions — " "that provenance line is how an operator tells a stale image from a current one") -DEVSH = ROOT / "infra/devcontainer/dev.sh" -check(DEVSH.is_file() and "--build-arg PROMTOOL_VERSION=" in DEVSH.read_text(), - "infra/devcontainer/dev.sh does not pass --build-arg PROMTOOL_VERSION — the ARG would be " - "empty and the build would fetch a bogus URL") +# The build recipe moved from dev.sh into build-image.sh on 2026-10-29, so that `dev.sh build` +# and the CI image pipeline (.gitea/workflows/images.yaml) share ONE recipe. Check the recipe. +BUILDSH = ROOT / "infra/devcontainer/build-image.sh" +check(BUILDSH.is_file() and "--build-arg PROMTOOL_VERSION=" in BUILDSH.read_text(), + "infra/devcontainer/build-image.sh does not pass --build-arg PROMTOOL_VERSION — the ARG " + "would be empty and the build would fetch a bogus URL") INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh" if INSTALL.is_file(): diff --git a/conformance/versions.env b/conformance/versions.env index 731d469..8325387 100644 --- a/conformance/versions.env +++ b/conformance/versions.env @@ -40,32 +40,29 @@ PYTHON_MIN="3.9" # gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:* # Any other path, and any TAG reference, is DENIED at admission. Do not "tidy" these values. CONFORMANCE_RUNNER_IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner" -# Build handle only. Admission never sees the tag; a Job must reference the digest. +# Local build handle only. Admission never sees the tag; a Job must reference the digest. +# CI publishes under `sha-` and `main` instead (.gitea/workflows/images.yaml). CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44" # # ############################################################################ -# # DIGEST DELIBERATELY EMPTY — THE IMAGE HAS NOT BEEN PUBLISHED YET. # +# # THE DIGEST IS WRITTEN BY RENOVATE, NOT BY HAND. # # ############################################################################ -# The image has been BUILT and its toolchain verified locally (non-root uid 65532, -# read-only rootfs: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4, chainsaw 0.2.12, -# and `./run.sh --layer l2 --no-cluster` completing with rc 0). It has NOT been pushed: -# pushing needs a registry credential, and publishing the artifact a fail-closed security -# boundary then trusts is an OPERATOR action, not an agent's. +# CI builds and verifies the image (non-root uid 65532, read-only rootfs, the +# conformance-clone refusals) and, on a push to main, publishes it and prints the digest the +# REGISTRY returned (.gitea/workflows/images.yaml). Renovate follows the image's `main` tag +# through the marker below and opens the pull request that moves this value. Nobody pushes +# this image from a workstation, and nobody pastes a digest here. # -# OPERATOR, to complete the pin: -# 1. ./conformance/incluster/build-image.sh # rebuild, verify, print the commands -# 2. podman login gitea.home.keithjackson.dev -# 3. podman push --digestfile /tmp/conformance-runner.digest \ -# "${CONFORMANCE_RUNNER_IMAGE_REPO}:${CONFORMANCE_RUNNER_IMAGE_TAG}" -# 4. Set CONFORMANCE_RUNNER_IMAGE_DIGEST below to `cat /tmp/conformance-runner.digest` -# — the digest the REGISTRY returned, not the local manifest digest the build printed. -# 5. Land steps 4 in a PR (this file is GitOps input, not a live edit). -# -# UNTIL THEN: this is empty on purpose, NOT a placeholder to be guessed at. An empty value -# makes any consumer that renders the Job fail visibly; a fake or stale digest would instead -# produce an ImagePullBackOff that looks like a cluster fault. Do not invent one. -CONFORMANCE_RUNNER_IMAGE_DIGEST="" -CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_DIGEST:+${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}}" +# ALL-ZERO UNTIL THE FIRST PUBLISH. Before 2026-10-29 this was EMPTY, so that a consumer +# rendering the Job would fail visibly. It is now the same impossible all-zero digest +# platform/pm-agent/ uses, because Renovate can only replace a digest that is there to match. +# A consumer MUST treat the all-zero value as "not published" (no consumer exists yet: the +# L2/L3 trigger is unwired, docs/plans/l2-l3-conformance-ci-wiring.md). Note also that +# chainsaw-job.yaml's @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ takes the 64 hex digits WITHOUT the +# `sha256:` prefix this value carries — the trigger must strip it. +# renovate-digest: image=gitea.home.keithjackson.dev/home-network/conformance-runner tag=main +CONFORMANCE_RUNNER_IMAGE_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000" +CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}" # # AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for # its `@sha256:*` wildcard to be replaced with the LITERAL digest once the image exists — diff --git a/decisions/2026-10-29-image-pipeline.md b/decisions/2026-10-29-image-pipeline.md new file mode 100644 index 0000000..7fa2979 --- /dev/null +++ b/decisions/2026-10-29-image-pipeline.md @@ -0,0 +1,222 @@ +--- +date: 2026-10-29 +branch: feat/image-pipeline +title: "Images are built, verified and published by CI; Renovate carries the digests into git" +--- + +## What changed + +Operator rule (2026-10-29): container image creation, hosting and lifecycle belong to CI/CD and +the codebase, never to a one-off workstation build/push. Before this branch the Home-Network org's +package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and +both `build-image.sh` scripts said "publishing is an operator action". + +- **`.gitea/workflows/images.yaml`** + **`.gitea/workflows/scripts/images.sh`** (the logic). + - Pull request: build and verify only. No push, and the step references no secret at all. + - Push to `main`, or `workflow_dispatch` on `main`: build, verify, then push to + `gitea.home.keithjackson.dev/home-network/{toolbox,agent-runner,conformance-runner}`. + Tags are `sha-<12 hex>` (immutable: the driver refuses to overwrite one) and `main` (moving). + The digest is read back from the registry with a HEAD on the manifest, compared with what the + client printed, and written to the job summary. + - Only images whose inputs changed are built. A re-run on the same commit skips images that are + already published. + - agent-runner builds FROM the toolbox built in the same run (on main, FROM its just-pushed + digest). Otherwise: + - in verify mode the toolbox is built anyway as its base, because a PR holds no credential; + - in publish mode it uses `AGENT_BASE_IMAGE` from `infra/agent/versions.env`; + - if that is still the zero placeholder, the toolbox is built and published first. This is + how the first run on main bootstraps. + - An always-run step prunes the DinD image store. +- **One build recipe per image, reused by CI.** + - New `infra/devcontainer/build-image.sh`. `dev.sh build` now delegates to it. It has an + in-image verification step that the toolbox never had: every tool is on PATH, and + promtool, kubectl, chainsaw, kyverno and sops report their pins. + - `infra/agent/build-image.sh` and `conformance/incluster/build-image.sh` were made + engine-neutral, because CI runs `ENGINE=docker` against DinD: + - `podman image exists` became `image inspect`, with a pull fallback; + - `{{.Digest}}` and `--digestfile` are gone. No local digest is printed, since only the + registry's digest is meaningful; + - `probe-validation.sh` is fed over **stdin** instead of a bind mount. Under DinD a bind + source is resolved on the daemon's filesystem, which would silently mount an empty + directory; + - `probe-validation.sh` itself gained `${BASH_SOURCE[0]:-$0}` so it survives `bash -s` + under `set -u`. + - `AGENT_BASE_IMAGE` and `IMAGE_TAG` can be overridden from the environment, and + `BUILD_REVISION` adds an `org.opencontainers.image.revision` label. + - The "publishing is an operator action" text was replaced throughout. +- **Digest references.** Each now carries a `# renovate-digest: image= tag=main` marker + on the line above it: + - `platform/pm-agent/deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`), which share + one dependency, so they bump in one PR; + - `infra/agent/versions.env` `AGENT_BASE_IMAGE`, which was `home-k3s-dev:local` and is now + the published toolbox by digest (zero placeholder until the first publish); + - `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to the + zero placeholder because Renovate can only replace a digest it can match. + + The admission policies are **untouched**; the `@sha256:` form is preserved. +- **`renovate.json5`**: + - a new `custom.regex` manager for the markers; + - `enabledManagers` `regex` → `custom.regex`; + - a packageRule that disables the `kubernetes` manager for these three repos. Otherwise it + would also see the digest-only lines and look them up as `latest`; + - an `own-image` label; + - `ignorePaths` narrowed from `conformance/**` to its subdirectories. `ignorePaths` applies + to custom managers too, so the blanket glob hid `conformance/versions.env`. +- **`renovate.yaml`**: `renovate-version: 44.167.0` pinned, plus a `RENOVATE_HOST_RULES` docker + rule for the Gitea registry, using the same `RENOVATE_TOKEN`, which now also needs + `read:package`. See below for why the version pin is a real fix and not tidiness. +- **`infra/gitea/registry-retention.sh`**: dry run by default, `--apply` to delete. This is an + operator action. Per image it keeps: + - `main` and `latest`; + - the 10 newest `sha-` tags; + - every tag whose digest is pinned anywhere in `origin/main`; + - every digest-named version. +- **`infra/gitea/conformance-trigger-kubeconfig.sh`**: the "runner image is digest-pinned" + precondition grepped the literal line `CONFORMANCE_RUNNER_IMAGE="${…_REPO}@${…_DIGEST}"` + for 64 hex digits, so it **could never pass**. It now sources the file, evaluates the value, + and rejects the zero placeholder. +- **`conformance/tests/l0-static/alert-rules-unit-tested/check.sh`**: its "the toolbox build + passes `--build-arg PROMTOOL_VERSION`" assertion now reads `infra/devcontainer/build-image.sh` + instead of `dev.sh`. The gate caught this on the first run, because the recipe had moved. +- **Docs**: `platform/pm-agent/README.md`, `runbooks/pm-agent.md` (two rows), + `docs/plans/pm-agent-issue-to-pr.md` (follow-up 1), `conformance/incluster/README.md`, + `infra/devcontainer/README.md`, and a new "How images are built" section in + `.gitea/workflows/README.md`. + +### Decisions and pushback + +- **The push credential is a PAT secret, `REGISTRY_PUSH_TOKEN`; the Actions token cannot push.** + Checked against Gitea **v1.24.3** source: + - `services/auth/basic.go` `Basic.Verify` turns a task token into the pseudo-user + `gitea-actions` (ID −2) and sets no token scope; + - `routers/api/packages/container/container.go` `Authenticate` (the `/v2/token` exchange every + docker client performs) requires `read:package`/`write:package`/`all` scope, so it answers + 401. The Actions token cannot even *pull*; + - `services/context/package.go` `determineAccessMode` has `// TODO: ActionUser permission check` + and no branch for that user. Even past login it would get at most read on a public org; + - `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3. + + The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight + pattern. +- **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate: + - `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` → + `custom.regex`; + - that migration runs on every repository run (`lib/workers/repository/init/merge.ts` → + `migrateAndValidate`), not only in the validator; + - `lib/modules/manager/index.ts` `getEnabledManagersList` accepts either spelling; + - `custom.regex` exists since 37.51.1 (renovatebot/renovate#24079). +- **The open question hid a bigger bug: the Renovate config on main is invalid for the Renovate + the workflow actually runs.** + - `renovatebot/github-action@v41.0.9` defaults `renovate-version` to `39` + (`action.yml`, `src/docker.ts`), which is 39.264.0. + - The validator used in #79 was `ghcr.io/renovatebot/renovate` with no tag, i.e. `latest`, + 44.167.0. + - **Measured:** `renovate:39.264.0 renovate-config-validator` on main's `renovate.json5` gives + `ERROR: Found errors in configuration` with five errors, all `managerFilePatterns`. + `fileMatch` was renamed in 40.2.0. + - So once #80 fixes the filename, every run would still die at `CONFIG_VALIDATION`. + - Fixed by pinning `renovate-version: 44.167.0`. This branch's config validates clean on it. + The only `--strict` complaint is a pending migration of the pre-existing top-level + `"docker": {"enabled": true}` block, which I left alone. +- **Unworkable as briefed: tracking the worker Job template's digest.** + - `infra/agent/worker-job-template.yaml` is `COPY`ed into agent-runner. A Renovate bump there + changes agent-runner's inputs, CI publishes a new digest, Renovate bumps it again, and so on + forever. + - `pm-dispatch` overwrites that field from `PM_WORKER_IMAGE` at runtime + (`pm-dispatch.py:649`), so nothing is lost by leaving it all-zero. It is commented as + deliberately untracked. + - For the same loop reason, `conformance/versions.env` is **not** an input of the + conformance-runner build, and a pipeline-only change does not republish on main. +- **Changed from the brief: retention is a script, not a Gitea cleanup rule.** + - Gitea 1.24.3 has **no API** for cleanup rules (none in `routers/api/v1/api.go`; web + session+CSRF forms only, in `routers/web/web.go`). So a rule cannot be created, or + drift-asserted, as code. **`ci/gitea-settings.sh` was therefore not extended.** A cheaper + proxy (count versions per image) would need `read:package` on `SETTINGS_API_TOKEN`, which + it lacks. + - A built-in rule is also **unsafe here**. Gitea stores a manifest as a file of the TAG + version it was pushed under (`routers/api/packages/container/manifest.go` + `createPackageAndVersion`), and resolves a by-digest pull from those files + (`models/packages/container/search.go`). Deleting the tag a deployed digest was pushed under + breaks the `@sha256:` pull, and the rule's "newest N" knows nothing about what git pins. The + script keeps every digest git pins. +- **Auto-merge.** Digest bumps of these images auto-merge on a green gate (the existing + `digest` rule). The exception is the conformance-runner bump: `conformance/versions.env` is a + sensitive path, so it waits for an administrator's `/approve-override` every time. + +## What I verified + +- **Local gate, pinned toolbox, from this worktree, committed HEAD.** Command: + `ENGINE= DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`. + The shim adds the `.git` mount. Result: see the PR body for the rc and stage list. They are + recorded there rather than here because this entry is part of the tree being gated. +- **Local podman builds through the CI driver**, run as + `MODE=verify ENGINE=podman BASE_SHA=$(git merge-base origin/main HEAD) .gitea/workflows/scripts/images.sh`. + It returned rc 0 and built and verified all three images: + - toolbox: every tool on PATH, and the five pins matched; + - agent-runner: `PM_SELFTEST=1 pm-dispatch` OK, and the stdin-fed `probe-validation.sh` + gave "20 refusal(s), 0 failure(s)"; + - conformance-runner: all seven `conformance-clone` refusals. +- **Other driver paths, run locally:** + - agent-runner alone builds the toolbox as its base; + - an empty diff builds nothing; + - publish mode without a token dies before doing anything; + - `infra/agent/build-image.sh` with the zero placeholder dies with build-locally + instructions. +- **Renovate, with the pinned 44.167.0 image, locally:** + - `renovate-config-validator` exits 0; + - a `--platform=local` `extract` dry run finds all four marker references (custom regex manager, + `currentValue: main`, `currentDigest: sha256:000…`); + - a `lookup` dry run shows the `kubernetes`-manager duplicates as `skipReason: disabled` and the + marker deps taking the "unsupported/unversioned value main → getDigest(main)" path. That + lookup currently 404s because nothing is published yet. +- **Renovate 39.264.0 rejects main's current `renovate.json5`.** Validator output reproduced + above. +- **`registry-retention.sh` dry run against the live Gitea (read-only):** zero versions per image, + nothing to delete. The token exchange and the package listing both answered. +- **Live, read-only measurements** (kubectl and the Gitea API): + - the runner pod is on k3s-prod-1, which had **7 GB** nodefs available; + - DinD holds only the 1.54 GB act image; buildx is v0.20.1; the daemon is 27.5.1 on overlay2 + with cgroup v2; + - Gitea `/data` is 9.7G, 1% used; + - the Gateway's Envoy (Cilium) has `streamIdleTimeout: 300s`, `maxStreamDuration: 0s`, and + **no route `timeout`** on the gitea route; + - Gitea's `LIMIT_SIZE_CONTAINER` is unset (default −1 = unlimited, `modules/setting/packages.go`). + +## What I did NOT verify + +- **Nothing has been pushed to the registry, by me or by CI.** The publish path has never run: + - the registry HEAD/digest read-back; + - the immutable-tag guard; + - "already published at this commit" idempotence; + - agent-runner `FROM @sha256:`; + - `docker login` with a PAT. + + The first merge to main is the first test, and it needs `REGISTRY_PUSH_TOKEN` to exist first. +- **Large-layer push through the Gateway is UNVERIFIED.** Envoy has no request-body cap and no + stream duration cap here. But with no route `timeout` set, Envoy's **default 15 s upstream + timeout** applies from the end of the request to the end of the response. That covers Gitea + finalising a blob: copying the upload into package storage on Longhorn. A ~250 MB compressed + layer should finish well inside 15 s, but I could not measure it without pushing. If the first + publish shows 504s on blob PUTs, set `timeouts.request` on the gitea HTTPRoute. +- **DinD fit is estimated, not measured** (until the PR run reports): + - memory: 2Gi is shared by the job container and every build container; + - disk: the toolbox, agent-runner, conformance-runner, Fedora base and build cache should + total roughly 3 GB on top of the 1.5 GB act image, against 7 GB free on the node, with + `dind-storage` having no sizeLimit. +- **That act_runner hands the job container a Docker socket is inferred** from act_runner's + DOCKER_HOST handling, not measured. The driver fails loudly if it does not. +- **Renovate has still never completed a run here.** Three things block it: + - #80, the `configurationFile` name; + - `RENOVATE_TOKEN` gaining `read:package`; + - this branch's version pin. + + `renovatebot/github-action@v41.0.9` running a 44.x image is untested. The action only passes + the tag to `docker run`. +- **Pods cannot pull these images yet regardless.** The kubelet pull credential is PR #61 + (`pm-agent-registry` dockerconfigjson, `read:package`), still open. Until it merges, a correct + digest still ends in ImagePullBackOff (401), as before. +- **`registry-retention.sh --apply` has never run**, and its keep logic has only run against an + empty registry. +- **`git diff` change detection on Gitea `push` events relies on `github.event.before`.** A + force-push or a first push falls back to building everything, which is intended but not + observed. diff --git a/docs/plans/pm-agent-issue-to-pr.md b/docs/plans/pm-agent-issue-to-pr.md index ad87805..9f18152 100644 --- a/docs/plans/pm-agent-issue-to-pr.md +++ b/docs/plans/pm-agent-issue-to-pr.md @@ -244,10 +244,12 @@ about runtime behaviour in this document is a design claim, not a measurement. (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with the credential decision now being local Gitea accounts with PATs, that means a `dockerconfigjson` for an account whose token carries `read:package`. Only then does the - digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both - placeholder digests. Expect the build itself to need iteration too: the nodejs module - stream and the npm install have never run against this base. The same two blockers sit in - front of `conformance-runner`, which shares the registry path and has never been pulled. + digest matter. **Superseded 2026-10-29:** the build-and-push is no longer an operator step. + `.gitea/workflows/images.yaml` builds and verifies the image with + `infra/agent/build-image.sh`, publishes it on a push to `main`, and Renovate opens the one + PR that replaces both placeholder digests (`decisions/2026-10-29-image-pipeline.md`). The + same two blockers sit in front of `conformance-runner`, which shares the registry path and + has never been pulled. 2. **Create the credentials** per `pm-agent-credentials.sops.yaml.example`, and **verify the bot's permission is `write`, not `admin`** — that measurement is the control that keeps the agent out of its own approval gate. diff --git a/infra/agent/build-image.sh b/infra/agent/build-image.sh index eb65585..e854f4e 100755 --- a/infra/agent/build-image.sh +++ b/infra/agent/build-image.sh @@ -1,26 +1,36 @@ #!/usr/bin/env bash ############################################################################### -# infra/agent/build-image.sh — build the pinned `agent-runner` image and print its digest. +# infra/agent/build-image.sh — build the pinned `agent-runner` image and prove it works. # -# ./build-image.sh # build + verify + print the digest and push command +# ./build-image.sh # build + verify + print the image id # ./build-image.sh --no-verify # build only (skip the in-image proof) # ENGINE=docker ./build-image.sh # build with docker instead of podman +# AGENT_BASE_IMAGE=home-k3s-dev:local ./build-image.sh +# # build on a LOCAL toolbox instead of the pinned one +# # (`infra/devcontainer/dev.sh build` makes that tag) +# IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this) # -# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is not -# an agent's to hold, and the pushed digest is what a fail-closed admission policy then -# trusts. So this builds locally, proves the toolchain and both entrypoints work INSIDE the -# image as the non-root uid with the read-only root filesystem the policy forces, and prints -# the exact push line for a human. +# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either. +# Publishing is done by CI: .gitea/workflows/images.yaml calls this script, and on a push to +# `main` it pushes what this script built AND verified to the Gitea registry, records the +# registry-returned digest in the job summary, and Renovate then opens the pull request that +# moves platform/pm-agent/ onto that digest. So this script's job is to build, and to prove +# the toolchain and both entrypoints work INSIDE the image as the non-root uid with the +# read-only root filesystem the admission policy forces — the same proof locally and in CI. # -# VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the four pins this image -# adds (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds -# the platform toolchain the base already carries. Nothing in this file invents a version. +# VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the pins this image adds +# (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds the +# platform toolchain the base already carries. Nothing in this file invents a version. # -# THE DIGEST CAVEAT — read it before pasting a digest into anything. The digest printed here -# is the LOCAL manifest digest. It is normally identical to the registry digest, but that is -# a property of the push rather than a guarantee, so the push command below uses -# `--digestfile`: the digest the REGISTRY returns is the authoritative one, and that is the -# one that goes into platform/pm-agent/configmap.yaml. +# THE BASE. AGENT_BASE_IMAGE in versions.env is the PUBLISHED toolbox, pinned by digest and +# kept current by Renovate. An AGENT_BASE_IMAGE in the environment overrides it — CI passes +# the toolbox it built in the same run, and a developer can pass `home-k3s-dev:local`. +# +# ENGINE PORTABILITY. CI runs this with ENGINE=docker against a Docker-in-Docker daemon, so +# nothing below may be podman-only (`image exists`, `--digestfile`, `{{.Digest}}`), and +# nothing may BIND-MOUNT a file from this checkout into a container: under DinD the daemon +# resolves bind sources on ITS filesystem, where this checkout does not exist, and silently +# mounts an empty directory instead. Files go in over stdin. ############################################################################### set -euo pipefail @@ -46,8 +56,11 @@ command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGI # --- pins: sourced, never invented ------------------------------------------- AGENT_ENV="${HERE}/versions.env" [ -f "${AGENT_ENV}" ] || die "missing ${AGENT_ENV} — the pins live there, not here" +BASE_OVERRIDE="${AGENT_BASE_IMAGE:-}" +TAG_OVERRIDE="${IMAGE_TAG:-}" # shellcheck disable=SC1090 source "${AGENT_ENV}" +[ -z "${BASE_OVERRIDE}" ] || AGENT_BASE_IMAGE="${BASE_OVERRIDE}" : "${AGENT_BASE_IMAGE:?AGENT_BASE_IMAGE not set in ${AGENT_ENV}}" : "${NODEJS_MAJOR:?NODEJS_MAJOR not set in ${AGENT_ENV}}" : "${CLAUDE_CODE_VERSION:?CLAUDE_CODE_VERSION not set in ${AGENT_ENV}}" @@ -55,16 +68,28 @@ source "${AGENT_ENV}" : "${AGENT_MODEL:?AGENT_MODEL not set in ${AGENT_ENV}}" : "${AGENT_IMAGE_REPO:?AGENT_IMAGE_REPO not set in ${AGENT_ENV}}" -# --- the base must already exist --------------------------------------------- -# Deliberately NOT built on demand. The toolbox build is its own reviewed artefact with its -# own pins, and silently building it here would mean this script could change the toolchain -# a worker gates with as a side effect of rebuilding the agent image. -if ! "${ENGINE}" image exists "${AGENT_BASE_IMAGE}" 2>/dev/null; then - die "base image '${AGENT_BASE_IMAGE}' is not present. Build the pinned toolbox first: - ${REPO_ROOT}/infra/devcontainer/dev.sh build" +# --- the base must exist, locally or in the registry --------------------------- +# Deliberately NOT built on demand. The toolbox is its own reviewed artefact with its own +# pins, and silently building it here would mean this script could change the toolchain a +# worker gates with as a side effect of rebuilding the agent image. +case "${AGENT_BASE_IMAGE}" in + *@sha256:0000000000000000000000000000000000000000000000000000000000000000) + die "AGENT_BASE_IMAGE is still the all-zero placeholder in ${AGENT_ENV}: the toolbox has + not been published yet. Either let CI publish it (.gitea/workflows/images.yaml) and + merge Renovate's digest bump, or build on a local toolbox: + ${REPO_ROOT}/infra/devcontainer/dev.sh build + AGENT_BASE_IMAGE=home-k3s-dev:local $0" ;; +esac +if ! "${ENGINE}" image inspect "${AGENT_BASE_IMAGE}" >/dev/null 2>&1; then + info "base ${AGENT_BASE_IMAGE} is not local; pulling it" + "${ENGINE}" pull "${AGENT_BASE_IMAGE}" >/dev/null \ + || die "base image '${AGENT_BASE_IMAGE}' is neither local nor pullable (registry login?). + To build on a local toolbox instead: + ${REPO_ROOT}/infra/devcontainer/dev.sh build + AGENT_BASE_IMAGE=home-k3s-dev:local $0" fi -IMAGE_TAG="claude${CLAUDE_CODE_VERSION}-node${NODEJS_MAJOR}" +IMAGE_TAG="${TAG_OVERRIDE:-claude${CLAUDE_CODE_VERSION}-node${NODEJS_MAJOR}}" IMAGE_REF="${AGENT_IMAGE_REPO}:${IMAGE_TAG}" log "Building ${IMAGE_REF}" @@ -75,7 +100,11 @@ info " node ${NODEJS_MAJOR}" info " claude ${CLAUDE_CODE_VERSION} (floor ${CLAUDE_CODE_MIN_VERSION})" info " model ${AGENT_MODEL}" -"${ENGINE}" build \ +# Provenance: CI passes the commit it is building, so a pulled image names its source tree. +LABEL_ARGS=() +[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}") + +"${ENGINE}" build "${LABEL_ARGS[@]}" \ --file "${HERE}/Containerfile" \ --tag "${IMAGE_REF}" \ --build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}" \ @@ -138,63 +167,32 @@ if [ "${VERIFY}" -eq 1 ]; then # that will actually run in the cluster. The probe is mounted read-only rather than baked # in: it is a build-time check, not something a worker Job has any business carrying. log "Probing pm-worker's input validation inside the image" - "${ENGINE}" run --rm \ + # + # The probe goes in over STDIN, not as a bind mount: under Docker-in-Docker (CI) a bind + # source is resolved on the daemon's filesystem, where this file does not exist, and the + # container would get an empty directory. PM_WORKER_PATH points it at the baked worker, + # since the probe's own default (a sibling of the probe file) has no meaning on stdin. + "${ENGINE}" run --rm -i \ --user 65532:65532 \ --read-only \ --tmpfs /workspace:rw,mode=1777 \ --tmpfs /tmp:rw,mode=1777 \ --network none \ - --volume "${HERE}/probe-validation.sh:/probe-validation.sh:ro,Z" \ + -e PM_WORKER_PATH=/usr/local/bin/pm-worker \ --entrypoint "" \ "${IMAGE_REF}" \ - /bin/bash /probe-validation.sh + /bin/bash -s < "${HERE}/probe-validation.sh" log "Image verified" fi -# --- digest ------------------------------------------------------------------ -DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")" +# --- result ------------------------------------------------------------------ +# No digest is printed: a manifest digest does not exist until a registry has the manifest, +# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at +# push time and prints it in the job summary (.gitea/workflows/scripts/images.sh). SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")" - log "Built ${IMAGE_REF}" -info "local manifest digest ${DIGEST}" -info "size $(( SIZE / 1024 / 1024 )) MiB" - -cat < sha256:.... THIS is the authoritative digest, not the local one above. - - # platform/pm-agent/configmap.yaml - PM_WORKER_IMAGE: "${AGENT_IMAGE_REPO}@\$(cat /tmp/agent-runner.digest)" - -WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/pm-agent-job-shape.yaml is a -fail-closed Enforce policy. It DENIES any worker Job whose image is not -${AGENT_IMAGE_REPO}@sha256: — a tag reference is denied, -and so is any other repository path. The dispatcher also refuses to start if -PM_WORKER_IMAGE carries no digest, so the failure names its own cause. - -THE SAME IMAGE RUNS THE DISPATCHER. platform/pm-agent/deployment.yaml pins it too, so a -new digest is a two-line change in one pull request: the Deployment's image and -PM_WORKER_IMAGE. They must be the same digest — the dispatcher and its workers share the -committed Job template and the env-name contract, and a mismatched pair would disagree -about it. - -TIGHTEN THE BOUNDARY AFTERWARDS: the policy admits the wildcard -\`${AGENT_IMAGE_REPO}@sha256:*\`, so anyone who can push to that registry path can publish -a different digest. Replacing the wildcard with the LITERAL digest closes that gap; it is a -one-line change, reviewed in a pull request, which is exactly where a change to the -security boundary belongs. -================================================================================ -EOF +info "image id $("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")" +info "size $(( SIZE / 1024 / 1024 )) MiB" +info "Not pushed. CI publishes this image from main (.gitea/workflows/images.yaml); see" +info "platform/pm-agent/README.md for how a new digest reaches the cluster." diff --git a/infra/agent/probe-validation.sh b/infra/agent/probe-validation.sh index 042d004..d888700 100755 --- a/infra/agent/probe-validation.sh +++ b/infra/agent/probe-validation.sh @@ -22,7 +22,9 @@ ############################################################################### set -uo pipefail -HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# `:-$0` because build-image.sh feeds this script over STDIN (`bash -s`), where BASH_SOURCE is +# empty and `set -u` would abort on it; PM_WORKER_PATH is always set in that case. +HERE="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)" WORKER="${PM_WORKER_PATH:-${HERE}/pm-worker}" [ -r "$WORKER" ] || { echo "cannot read ${WORKER}" >&2; exit 2; } diff --git a/infra/agent/versions.env b/infra/agent/versions.env index bd0a136..676d66c 100644 --- a/infra/agent/versions.env +++ b/infra/agent/versions.env @@ -9,10 +9,16 @@ # truth and, the first time the two drifted, the worker's "green" would stop meaning the # gate's green. So this file pins ONLY what the toolbox does not carry. -# The toolbox image the runner is layered on. Built by `infra/devcontainer/dev.sh build` -# (tag `home-k3s-dev:local`); infra/agent/build-image.sh asserts it exists rather than -# silently building a different base. -AGENT_BASE_IMAGE=home-k3s-dev:local +# The toolbox image the runner is layered on: the PUBLISHED toolbox, by digest. CI builds and +# publishes it (.gitea/workflows/images.yaml) and Renovate moves this line onto each new digest +# of its `main` tag via the marker below, in a pull request — nobody edits it by hand. +# +# ALL-ZERO UNTIL THE FIRST PUBLISH. The first run of the image pipeline on main sees this +# placeholder, builds and publishes the toolbox first, and builds agent-runner FROM that fresh +# digest. For a LOCAL build on a local toolbox, override it: +# infra/devcontainer/dev.sh build && AGENT_BASE_IMAGE=home-k3s-dev:local infra/agent/build-image.sh +# renovate-digest: image=gitea.home.keithjackson.dev/home-network/toolbox tag=main +AGENT_BASE_IMAGE=gitea.home.keithjackson.dev/home-network/toolbox@sha256:0000000000000000000000000000000000000000000000000000000000000000 # Node.js — the runtime for the Claude Code CLI, installed from the Fedora base's module # stream. A major version, not a patch: the CLI supports the active LTS line and the diff --git a/infra/agent/worker-job-template.yaml b/infra/agent/worker-job-template.yaml index 6687282..62d043b 100644 --- a/infra/agent/worker-job-template.yaml +++ b/infra/agent/worker-job-template.yaml @@ -80,6 +80,12 @@ spec: # Replaced by the dispatcher with the digest-pinned image. A tag here would be # DENIED by admission; the dispatcher refuses a non-digest PM_WORKER_IMAGE at # startup so the error names the real cause. + # + # DELIBERATELY ALL-ZERO, AND DELIBERATELY NOT TRACKED BY RENOVATE. This file is + # baked INTO agent-runner, so it cannot carry agent-runner's own digest: bumping it + # would change the image's inputs, CI would publish a new digest, Renovate would bump + # it again — forever. pm-dispatch overwrites this field from PM_WORKER_IMAGE + # (platform/pm-agent/configmap.yaml, which IS tracked) on every Job it creates. image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000 imagePullPolicy: IfNotPresent # The pinned entrypoint, with NO args. Args on a pinned command are diff --git a/infra/devcontainer/README.md b/infra/devcontainer/README.md index 99cb8fd..3cfbaa3 100644 --- a/infra/devcontainer/README.md +++ b/infra/devcontainer/README.md @@ -34,6 +34,22 @@ cd infra/devcontainer ``` `ENGINE=docker ./dev.sh` to use docker instead of rootless podman. +### The published toolbox + +`./dev.sh build` delegates to [`build-image.sh`](build-image.sh), which is the **one** build +recipe: CI runs the same script (with its in-image verification — every tool executes, and the +version-sensitive ones report their `versions.env` pins) and, on a push to `main` that changes +`Containerfile`, `versions.env` or `build-image.sh`, **publishes** the result as +`gitea.home.keithjackson.dev/home-network/toolbox` (tags `sha-` and `main`) — +see [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). `agent-runner` is +built `FROM` that published digest (`infra/agent/versions.env`, kept current by Renovate). +Nobody pushes this image from a workstation. To use the published one locally instead of +building, after `podman login gitea.home.keithjackson.dev` (a token with `read:package`): + +```bash +IMAGE=gitea.home.keithjackson.dev/home-network/toolbox:main ./dev.sh run make -C ci gate +``` + ## Choosing a cluster — `CLUSTER=dev|prod` ```bash diff --git a/infra/devcontainer/build-image.sh b/infra/devcontainer/build-image.sh new file mode 100755 index 0000000..9467a8e --- /dev/null +++ b/infra/devcontainer/build-image.sh @@ -0,0 +1,108 @@ +#!/usr/bin/env bash +############################################################################### +# infra/devcontainer/build-image.sh — build the pinned dev `toolbox` image and prove it works. +# +# ./build-image.sh # build + verify (tag: home-k3s-dev:local) +# ./build-image.sh --no-verify # build only +# IMAGE=foo:bar ./build-image.sh # build under another local name +# ENGINE=docker ./build-image.sh # build with docker instead of podman +# +# ONE BUILD DEFINITION, TWO CALLERS. `dev.sh build` calls this for local development, and the +# CI image pipeline (.gitea/workflows/images.yaml -> .gitea/workflows/scripts/images.sh) calls +# it to build the image it then PUBLISHES as gitea.home.keithjackson.dev/home-network/toolbox. +# Keeping the build args in exactly one place is what makes "the toolbox CI publishes" and "the +# toolbox you gate with locally" the same recipe. +# +# THIS SCRIPT NEVER PUSHES. Publishing is CI's job (images.yaml, on push to main), not a +# workstation's: an image that exists only because someone ran `podman push` from a laptop has +# no build log, no recorded inputs and no verification anyone else can read. +# +# VERSIONS ARE NOT DEFINED HERE. infra/devcontainer/versions.env is the single source of truth. +############################################################################### +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ENGINE="${ENGINE:-podman}" +IMAGE="${IMAGE:-home-k3s-dev:local}" + +VERIFY=1 +for arg in "$@"; do + case "$arg" in + --no-verify) VERIFY=0 ;; + -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;; + *) echo "unknown argument: $arg" >&2; exit 2 ;; + esac +done + +log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; } +die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; } + +command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)" + +# shellcheck disable=SC1091 +source "${HERE}/versions.env" + +# Provenance: CI passes the commit it is building, so a pulled image names its source tree. +LABEL_ARGS=() +[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}") + +log "Building ${IMAGE}" +"$ENGINE" build "${LABEL_ARGS[@]}" \ + --build-arg FEDORA_VERSION="$FEDORA_VERSION" \ + --build-arg OPENTOFU_VERSION="$OPENTOFU_VERSION" \ + --build-arg AGE_VERSION="$AGE_VERSION" \ + --build-arg SOPS_VERSION="$SOPS_VERSION" \ + --build-arg FLUX_VERSION="$FLUX_VERSION" \ + --build-arg KUBECTL_VERSION="$KUBECTL_VERSION" \ + --build-arg HELM_VERSION="$HELM_VERSION" \ + --build-arg KYVERNO_VERSION="$KYVERNO_VERSION" \ + --build-arg KYVERNO_JSON_VERSION="$KYVERNO_JSON_VERSION" \ + --build-arg CHAINSAW_VERSION="$CHAINSAW_VERSION" \ + --build-arg YAMLLINT_VERSION="$YAMLLINT_VERSION" \ + --build-arg YQ_VERSION="$YQ_VERSION" \ + --build-arg CHECK_JSONSCHEMA_VERSION="$CHECK_JSONSCHEMA_VERSION" \ + --build-arg PROMTOOL_VERSION="$PROMTOOL_VERSION" \ + -t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}" + +# --- verify ------------------------------------------------------------------ +# A build that succeeded is not evidence the toolbox can run the gate. Every tool the gate and +# the runbooks call must EXECUTE, and the version-sensitive ones must report the version +# versions.env pins — promtool above all, because ci/versions.env asserts a promtool/Prometheus +# lockstep and a drifted engine evaluates alert-rule tests differently (see versions.env). +if [ "${VERIFY}" -eq 1 ]; then + log "Verifying the toolchain inside ${IMAGE}" + "$ENGINE" run --rm --network none --entrypoint "" \ + -e PROMTOOL_VERSION="${PROMTOOL_VERSION}" \ + -e KUBECTL_VERSION="${KUBECTL_VERSION}" \ + -e CHAINSAW_VERSION="${CHAINSAW_VERSION}" \ + -e KYVERNO_VERSION="${KYVERNO_VERSION}" \ + -e SOPS_VERSION="${SOPS_VERSION}" \ + "$IMAGE" /bin/bash -euo pipefail -c ' + echo "--- baked versions ---"; cat /etc/dev-toolchain-versions + echo "--- every tool is on PATH ---" + for t in tofu age age-keygen sops flux kubectl helm kyverno kyverno-json chainsaw yq \ + yamllint check-jsonschema promtool ansible-playbook git make jq diff python3 \ + wg qrencode; do + if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t"; + else printf " MISSING %s\n" "$t"; missing=1; fi + done + [ -z "${missing:-}" ] || { echo "toolbox is missing tools"; exit 1; } + echo "--- version-sensitive tools report their pins ---" + check() { # name pin actual-output + if printf "%s" "$3" | grep -qF "$2"; then printf " ok %-10s %s\n" "$1" "$2"; + else printf " DRIFT %-10s want %s, got: %s\n" "$1" "$2" "$(printf "%s" "$3" | head -1)"; drift=1; fi + } + check promtool "${PROMTOOL_VERSION}" "$(promtool --version 2>&1)" + check kubectl "${KUBECTL_VERSION}" "$(kubectl version --client 2>&1)" + check chainsaw "${CHAINSAW_VERSION}" "$(chainsaw version 2>&1)" + check kyverno "${KYVERNO_VERSION}" "$(kyverno version 2>&1)" + check sops "${SOPS_VERSION}" "$(sops --version --disable-version-check 2>&1 || sops --version 2>&1)" + [ -z "${drift:-}" ] || { echo "a tool does not report its pinned version"; exit 1; } + echo "--- python can import what the renderer and the gate need ---" + python3 -c "import yaml; print(\" PyYAML\", yaml.__version__)" + ' + log "Toolbox verified" +fi + +log "Built ${IMAGE}" +printf ' image id %s\n' "$("$ENGINE" image inspect --format '{{.Id}}' "$IMAGE")" diff --git a/infra/devcontainer/dev.sh b/infra/devcontainer/dev.sh index d86561c..29701d6 100755 --- a/infra/devcontainer/dev.sh +++ b/infra/devcontainer/dev.sh @@ -72,22 +72,10 @@ esac CLUSTER_KUBECONFIG="${REPO_ROOT}/infra/ansible/.cache/${CLUSTER}.kubeconfig" build() { - "$ENGINE" build \ - --build-arg FEDORA_VERSION="$FEDORA_VERSION" \ - --build-arg OPENTOFU_VERSION="$OPENTOFU_VERSION" \ - --build-arg AGE_VERSION="$AGE_VERSION" \ - --build-arg SOPS_VERSION="$SOPS_VERSION" \ - --build-arg FLUX_VERSION="$FLUX_VERSION" \ - --build-arg KUBECTL_VERSION="$KUBECTL_VERSION" \ - --build-arg HELM_VERSION="$HELM_VERSION" \ - --build-arg KYVERNO_VERSION="$KYVERNO_VERSION" \ - --build-arg KYVERNO_JSON_VERSION="$KYVERNO_JSON_VERSION" \ - --build-arg CHAINSAW_VERSION="$CHAINSAW_VERSION" \ - --build-arg YAMLLINT_VERSION="$YAMLLINT_VERSION" \ - --build-arg YQ_VERSION="$YQ_VERSION" \ - --build-arg CHECK_JSONSCHEMA_VERSION="$CHECK_JSONSCHEMA_VERSION" \ - --build-arg PROMTOOL_VERSION="$PROMTOOL_VERSION" \ - -t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}" + # The build recipe lives in build-image.sh so that CI (.gitea/workflows/images.yaml), which + # builds and PUBLISHES this image, and a local `dev.sh build` cannot drift apart. --no-verify + # keeps the local loop as fast as it always was; CI runs the verification. + IMAGE="$IMAGE" ENGINE="$ENGINE" "${HERE}/build-image.sh" --no-verify } run() { diff --git a/infra/gitea/conformance-trigger-kubeconfig.sh b/infra/gitea/conformance-trigger-kubeconfig.sh index 4b7e30d..9016409 100755 --- a/infra/gitea/conformance-trigger-kubeconfig.sh +++ b/infra/gitea/conformance-trigger-kubeconfig.sh @@ -252,11 +252,16 @@ step "preconditions — repo side (these block ONLY arming ${VARIABLE}; the secr # R1: the runner image is built and digest-pinned. Without it the Job cannot run run.sh at # all (the stock chainsaw image has no bash/python3/kubectl/git) — plan §3 item 1. +# EVALUATED, not grepped: the line is `CONFORMANCE_RUNNER_IMAGE="${..._REPO}@${..._DIGEST}"`, so +# its literal text never contains 64 hex digits and the grep this replaced could never pass. The +# all-zero digest is the "not published yet" placeholder (Renovate replaces it after CI's first +# publish, .gitea/workflows/images.yaml) and must not count as pinned. VERSIONS="${REPO_ROOT}/conformance/versions.env" -if grep -Eq '^[[:space:]]*CONFORMANCE_RUNNER_IMAGE=.*@sha256:[0-9a-f]{64}' "$VERSIONS" 2>/dev/null; then - pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env" +runner_img="$( (set +u; . "$VERSIONS" >/dev/null 2>&1; printf '%s' "${CONFORMANCE_RUNNER_IMAGE:-}") )" +if [[ "$runner_img" =~ @sha256:[0-9a-f]{64}$ ]] && [[ ! "$runner_img" =~ @sha256:0{64}$ ]]; then + pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env (${runner_img##*@})" else - fail_arm "conformance/versions.env has no digest-pinned CONFORMANCE_RUNNER_IMAGE (plan §3 item 1) — the Job cannot run ./run.sh" + fail_arm "conformance/versions.env has no PUBLISHED digest for CONFORMANCE_RUNNER_IMAGE (got '${runner_img}') — the image pipeline has not published it yet, or Renovate's bump is unmerged (plan §3 item 1)" fi # R2: the Job template is the real thing, not the skeleton: the admitted image AND an diff --git a/infra/gitea/registry-retention.sh b/infra/gitea/registry-retention.sh new file mode 100755 index 0000000..e62f6c0 --- /dev/null +++ b/infra/gitea/registry-retention.sh @@ -0,0 +1,130 @@ +#!/usr/bin/env bash +# Registry retention for this repo's OWN images (toolbox, agent-runner, conformance-runner): +# keep what is deployed and what is recent, delete the rest. DRY RUN unless --apply. +# +# ./infra/gitea/registry-retention.sh # show what would be deleted +# ./infra/gitea/registry-retention.sh --apply # delete it +# KEEP=15 ./infra/gitea/registry-retention.sh # keep the 15 newest sha- tags (default 10) +# +# RUNNING IT IS AN OPERATOR ACTION. It deletes registry content, so it is not wired into CI and +# no agent runs it with --apply. +# +# WHY A SCRIPT, AND NOT GITEA'S BUILT-IN "CLEANUP RULE" — measured against Gitea 1.24.3's source: +# +# 1. There is NO API for cleanup rules. routers/api/v1/api.go has no such route; the rules live +# only behind the web UI's session+CSRF forms (routers/web/web.go, org settings +# `/packages/rules/add`). A rule therefore cannot be created, read or drift-checked as code — +# which is also why ci/gitea-settings.sh cannot assert one exists. +# 2. A built-in rule would DELETE WHAT IS DEPLOYED. Every reference to these images is +# digest-only (`@sha256:...`, required by admission). Gitea stores a manifest as a +# file OF THE TAG VERSION it was pushed under (routers/api/packages/container/manifest.go +# createPackageAndVersion), and resolves a by-digest pull by searching those files +# (models/packages/container/search.go). So deleting the tag a digest was pushed under makes +# that digest unpullable — MANIFEST_UNKNOWN, i.e. ImagePullBackOff on the next reschedule. +# The built-in rule keeps "the newest N" and a name pattern, and knows nothing about which +# digest git currently pins. Renovate's bump PRs can lag, so "newest N" is not a guarantee. +# +# WHAT IS KEPT, per image: +# * the `main` and `latest` tags; +# * the KEEP newest `sha-` tags (by creation time); +# * EVERY tag whose manifest digest is referenced anywhere in the tracked tree at REF +# (default origin/main) — i.e. whatever Flux is deploying, and the toolbox digest +# agent-runner builds FROM; +# * every version whose NAME is a digest (`sha256:...`): those are index children, and Gitea's +# own cleanup handles them (services/packages/container/cleanup.go ShouldBeSkipped). +# Everything else is deleted. Gitea's `cleanup_packages` cron then reclaims the unreferenced +# blobs (default: daily, blobs older than 24h) — space comes back the day after, not instantly. +# +# THE STORAGE IT PROTECTS. Packages live on the SAME 10Gi Longhorn PVC as every repository and +# Gitea's database (catalog/gitea.yaml). The toolbox is ~1.4 GB uncompressed and every main push +# that touches its inputs adds a complete, non-reproducible copy. Unpruned, the registry is what +# fills the volume the git host lives on. +# +# CREDENTIAL: the one `git` has stored for the host (as the other scripts here). It needs +# `read:package` to list and `write:package` to --apply, and the account needs package write on +# the org. The secret is never printed. +set -euo pipefail +HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; NS=home-network +IMAGES=(toolbox agent-runner conformance-runner) +KEEP="${KEEP:-10}" +REF="${REF:-origin/main}" +APPLY=0 +for a in "$@"; do + case "$a" in + --apply) APPLY=1 ;; + -h|--help) sed -n '2,/^set -euo/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;; + *) echo "unknown argument: $a" >&2; exit 2 ;; + esac +done +[[ "$KEEP" =~ ^[0-9]+$ ]] && [ "$KEEP" -ge 1 ] || { echo "KEEP must be a positive integer" >&2; exit 2; } +command -v jq >/dev/null || { echo "jq is required" >&2; exit 1; } +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" + +git -C "$REPO_ROOT" fetch -q origin main 2>/dev/null || echo " (could not fetch origin/main; using the local ${REF})" >&2 +git -C "$REPO_ROOT" rev-parse -q --verify "${REF}^{commit}" >/dev/null || { echo "no such ref: ${REF}" >&2; exit 1; } + +creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)" +U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")" +[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; } +echo "authenticating as '${U}' (secret not shown)" +api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; } + +REG_TOKEN="$(api "https://${HOST}/v2/token?service=container_registry" | jq -r '.token // empty')" +[ -n "$REG_TOKEN" ] || { echo "registry token exchange failed (does the credential have read:package?)" >&2; exit 1; } +manifest_digest() { # -> digest, or nothing + curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "$REG_TOKEN") \ + -H 'Accept: application/vnd.oci.image.manifest.v1+json' \ + -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \ + -H 'Accept: application/vnd.oci.image.index.v1+json' \ + -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \ + "https://${HOST}/v2/${NS}/$1/manifests/$2" \ + | tr -d '\r' | awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' | tail -1 +} + +TOTAL_DEL=0 +for img in "${IMAGES[@]}"; do + echo + echo "==> ${NS}/${img}" + # Digests git pins at REF. The marker convention puts the digest on the line after + # `# renovate-digest: image=`; the plain `@sha256:` grep catches any other use. + repo="${HOST}/${NS}/${img}" + pinned="$( { git -C "$REPO_ROOT" grep -h -A1 -F "renovate-digest: image=${repo} " "$REF" -- . 2>/dev/null || true + git -C "$REPO_ROOT" grep -h -F "${repo}@sha256:" "$REF" -- . 2>/dev/null || true; } \ + | grep -oE 'sha256:[0-9a-f]{64}' | grep -v 'sha256:0\{64\}' | sort -u || true)" + echo " pinned at ${REF}: $(wc -w <<<"$pinned") digest(s)" + + versions="[]"; page=1 + while :; do + chunk="$(api "https://${HOST}/api/v1/packages/${OWNER}?type=container&q=${img}&limit=50&page=${page}")" + jq -e 'type=="array"' >/dev/null <<<"$chunk" || { echo " package list failed: $(head -c 200 <<<"$chunk")" >&2; exit 1; } + [ "$(jq length <<<"$chunk")" -gt 0 ] || break + versions="$(jq -c --argjson a "$versions" --arg n "$img" '$a + [ .[] | select(.name==$n) | {version, created_at} ]' <<<"$chunk")" + page=$((page + 1)) + done + echo " versions: $(jq length <<<"$versions")" + + keep_sha="$(jq -r '[ .[] | select(.version|startswith("sha-")) ] | sort_by(.created_at) | reverse | .[:'"$KEEP"'][] .version' <<<"$versions")" + while IFS= read -r v; do + [ -n "$v" ] || continue + case "$v" in + main|latest) echo " keep ${v} (moving tag)"; continue ;; + sha256:*) echo " keep ${v} (digest-named version; Gitea's cleanup owns those)"; continue ;; + esac + if grep -qxF "$v" <<<"$keep_sha"; then echo " keep ${v} (among the ${KEEP} newest)"; continue; fi + d="$(manifest_digest "$img" "$v")" + if [ -z "$d" ]; then echo " keep ${v} (could not read its digest — never delete what cannot be checked)"; continue; fi + if grep -qxF "$d" <<<"$pinned"; then echo " keep ${v} (${d} is pinned at ${REF})"; continue; fi + if [ "$APPLY" -eq 1 ]; then + code="$(api -o /dev/null -w '%{http_code}' -X DELETE \ + "https://${HOST}/api/v1/packages/${OWNER}/container/${img}/$(jq -rn --arg v "$v" '$v|@uri')")" + if [ "$code" = 204 ]; then echo " DELETED ${v} (${d})"; TOTAL_DEL=$((TOTAL_DEL + 1)); + else echo " FAILED ${v} HTTP ${code}" >&2; fi + else + echo " would delete ${v} (${d})"; TOTAL_DEL=$((TOTAL_DEL + 1)) + fi + done < <(jq -r '.[].version' <<<"$versions") +done + +echo +if [ "$APPLY" -eq 1 ]; then echo "deleted ${TOTAL_DEL} version(s). Blobs are reclaimed by Gitea's cleanup_packages cron." +else echo "DRY RUN: ${TOTAL_DEL} version(s) would be deleted. Re-run with --apply to delete them."; fi diff --git a/platform/pm-agent/README.md b/platform/pm-agent/README.md index b201f37..5247aff 100644 --- a/platform/pm-agent/README.md +++ b/platform/pm-agent/README.md @@ -45,11 +45,18 @@ waive the sensitive-path human review on its own pull request. Both are deliberate, and both fail **visibly** rather than silently: -1. **The image does not exist.** `platform/pm-agent/deployment.yaml` and - `PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder. Build it with - `infra/agent/build-image.sh` (which verifies it and refuses to push), push it as the - operator, then update **both** places with the digest `podman push --digestfile` returned, - in one pull request. Until then: `ImagePullBackOff`. +1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and + `PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder until the first + publish. `.gitea/workflows/images.yaml` builds `agent-runner` with + `infra/agent/build-image.sh` (the same script, and the same in-image verification, you can + run locally), and on a push to `main` publishes it to + `gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-` and `main`. + Renovate then follows the `main` tag through the `# renovate-digest:` marker above **both** + lines and moves them together in one pull request. Nobody builds-and-pushes this image from + a workstation, and nobody pastes a digest. Until the first publish, and until that PR + merges: `ImagePullBackOff`. The one operator prerequisite on the publishing side is the + `REGISTRY_PUSH_TOKEN` Actions secret (see the header of `.gitea/workflows/images.yaml`); + without it the publish job fails loudly rather than skipping. **Two things have to be true before any digest will pull, and neither is declared in this repository.** Both were measured after PR #57 merged — this service is the first workload @@ -78,8 +85,8 @@ Both are deliberate, and both fail **visibly** rather than silently: account whose token carries `read:package`, referenced from the Deployment and the committed worker Job template — not built yet, because minting that token is yours. - The same two apply to `conformance-runner`: same registry path, same push-then-pin model, - never pulled either. + The same two apply to `conformance-runner`: same registry path, same publish-then-Renovate + model, never pulled either. 2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off — which Gitea account to create, which scopes, and why it must *not* be an administrator — diff --git a/platform/pm-agent/configmap.yaml b/platform/pm-agent/configmap.yaml index 16de902..61104a1 100644 --- a/platform/pm-agent/configmap.yaml +++ b/platform/pm-agent/configmap.yaml @@ -52,12 +52,13 @@ data: # contract. A tag here is refused at startup (and would be DENIED by admission anyway). # # PLACEHOLDER UNTIL THE IMAGE IS PUBLISHED. The all-zero digest below is deliberately - # impossible: `infra/agent/build-image.sh` builds and verifies the image, the OPERATOR - # pushes it, and the authoritative digest from `--digestfile` replaces this value in a - # reviewed pull request (the same two-step the conformance-runner image follows, and for - # the same reason: a registry credential is not an agent's to hold). Until then the - # dispatcher starts, fails to pull, and says so — which is the visible failure, not a - # silent one. + # impossible. CI builds, verifies and — on main — publishes agent-runner + # (.gitea/workflows/images.yaml), and Renovate replaces this value with the digest the + # REGISTRY returned for the image's `main` tag, in a pull request that also moves the + # Deployment's image (same marker, same branch). Nobody pushes this image from a workstation. + # Until the first publish the dispatcher starts, fails to pull, and says so — which is the + # visible failure, not a silent one. + # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000" # --- the model ------------------------------------------------------------- diff --git a/platform/pm-agent/deployment.yaml b/platform/pm-agent/deployment.yaml index 96c8812..fdd4adb 100644 --- a/platform/pm-agent/deployment.yaml +++ b/platform/pm-agent/deployment.yaml @@ -66,9 +66,12 @@ spec: - name: dispatch # THE SAME IMAGE THE WORKERS RUN, at the same digest. They share the committed Job # template and the env-name contract baked into it, so a mismatched pair would - # disagree about the shape the admission policy enforces. Replaced together, in one - # pull request, with the digest `infra/agent/build-image.sh` printed — the all-zero + # disagree about the shape the admission policy enforces. Replaced together, in ONE + # Renovate pull request: both lines carry the same marker, so Renovate treats them as + # one dependency and bumps them in the same branch. The digest comes from CI's publish + # of agent-runner's `main` tag (.gitea/workflows/images.yaml) — the all-zero # placeholder below is impossible on purpose (see configmap.yaml PM_WORKER_IMAGE). + # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000 imagePullPolicy: IfNotPresent command: diff --git a/renovate.json5 b/renovate.json5 index f6decdf..48e5ee4 100644 --- a/renovate.json5 +++ b/renovate.json5 @@ -22,9 +22,28 @@ // podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \ // ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json5 // -// NOTE on what was NOT wrong: `enabledManagers` containing bare `regex` was suspected of needing to -// be `custom.regex`. The validator accepts `regex`, so that suspicion was unfounded and nothing -// here changes it. +// CORRECTION (2026-10-29): THE VALIDATOR ABOVE WAS NOT THE RENOVATE THE WORKFLOW RUNS. The bare +// `ghcr.io/renovatebot/renovate` tag is `latest` (44.167.0 when this was measured), while +// renovatebot/github-action@v41.0.9 defaults `renovate-version` to `39` (its action.yml), i.e. +// 39.264.0. Validated against THAT, this file failed with five errors — every +// `managerFilePatterns` key ("Custom Manager contains disallowed fields", "Invalid configuration +// option: flux.managerFilePatterns", ...), because 39.x only knows `fileMatch`; the rename landed +// in 40.2.0. So even after the workflow found this file, every run would have stopped at +// CONFIG_VALIDATION. renovate.yaml now pins `renovate-version` to the exact release this file is +// validated against. VALIDATE WITH THAT SAME TAG: +// podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \ +// ghcr.io/renovatebot/renovate: \ +// renovate-config-validator /tmp/renovate.json5 +// (Not `--strict`: that also fails on a pending migration of the pre-existing top-level +// `"docker": {"enabled": true}` block at the bottom, which is a warning, not an error.) +// +// `regex` vs `custom.regex` in enabledManagers, RESOLVED FROM SOURCE (Renovate 39.264.0 and +// later): both work. lib/config/migrations/custom/enabled-managers-migration.ts rewrites +// `regex` -> `custom.regex`, and that migration runs on every real repository run +// (lib/workers/repository/init/merge.ts -> migrateAndValidate), not only in the validator; +// lib/modules/manager/index.ts getEnabledManagersList accepts either spelling. `custom.regex` is +// the canonical name (introduced in 37.51.1, renovatebot/renovate#24079), so it is used below and +// nothing depends on the migration any more. { "$schema": "https://docs.renovatebot.com/renovate-schema.json", // Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17 // RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags. @@ -51,7 +70,7 @@ "labels": ["dependencies", "automated-update"], "ignoreUnstable": true, "respectLatest": true, - "enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"], + "enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "custom.regex", "github-actions"], "flux": { "managerFilePatterns": [ "/(^|/)platform/.+\\.ya?ml$/", @@ -93,6 +112,42 @@ "#\\s*renovate:\\s*datasource=(?\\S+)\\s+depName=(?\\S+)(?:\\s+versioning=(?\\S+))?\\s*\\n.*?(?v?[0-9]+\\.[0-9]+\\.[0-9]+)" ], "versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}" + }, + // ADDED 2026-10-29 — THIS REPO'S OWN IMAGES (toolbox, agent-runner, conformance-runner). + // CI publishes them (.gitea/workflows/images.yaml) under an immutable `sha-` tag and a + // moving `main` tag, and never writes to git. Every reference to them is DIGEST-ONLY + // (`@sha256:`, no tag), because the admission policies + // (platform/kyverno/admission/pm-agent-job-shape.yaml, conformance-trigger-job-shape.yaml) + // admit exactly that form. A digest alone gives Renovate nothing to follow, so the tag to + // follow rides in a marker comment on the line ABOVE the reference: + // + // # renovate-digest: image=// tag=main + // sha256:<64 hex> + // + // `main` is not a version, so Renovate skips the release lookup and proposes a `digest` + // update whenever the tag's digest moves (lib/workers/repository/process/lookup/index.ts: + // an unversioned currentValue WITH a currentDigest yields updateType `digest`). The + // `sha256:` prefix is INSIDE currentDigest on purpose: Renovate replaces that capture + // verbatim with the new digest, which carries the prefix. + // + // DELIBERATELY NOT TRACKED: infra/agent/worker-job-template.yaml. It is baked INTO + // agent-runner, so bumping it would rebuild agent-runner and loop forever; pm-dispatch + // overwrites that field from PM_WORKER_IMAGE at runtime anyway. + // REGISTRY AUTH: the Gitea registry answers 401 to anonymous reads, and the platform token is + // scoped to hostType `gitea`, which a docker lookup never matches + // (lib/util/host-rules.ts find). renovate.yaml supplies a hostType `docker` rule. + { + "customType": "regex", + "description": "This repo's own CI-published images, pinned by digest, following the tag named in a `# renovate-digest:` marker on the previous line.", + "managerFilePatterns": [ + "/(^|/)versions\\.env$/", + "/(^|/)platform/pm-agent/.+\\.ya?ml$/" + ], + "matchStrings": [ + "#\\s*renovate-digest:\\s*image=(?\\S+)\\s+tag=(?\\S+)\\s*\\n[^\\n]*?(?sha256:[a-f0-9]{64})" + ], + "datasourceTemplate": "docker", + "versioningTemplate": "docker" } ], "packageRules": [ @@ -151,6 +206,26 @@ "needs-review", "blast-radius" ] + }, + { + "description": "ADDED 2026-10-29. This repo's own images are followed ONLY through the `# renovate-digest:` markers above. The `kubernetes` manager also sees platform/pm-agent/deployment.yaml and infra/agent/worker-job-template.yaml, where the image has a digest and no tag; left on, it would look those up as `latest` (a tag CI never pushes) and, for the template, re-open the self-referential loop the custom manager deliberately avoids.", + "matchManagers": ["kubernetes"], + "matchPackageNames": [ + "gitea.home.keithjackson.dev/home-network/toolbox", + "gitea.home.keithjackson.dev/home-network/agent-runner", + "gitea.home.keithjackson.dev/home-network/conformance-runner" + ], + "enabled": false + }, + { + "description": "ADDED 2026-10-29. Digest bumps for this repo's own images: the image was built and verified by CI before it was published, so these auto-merge on a green gate like any digest (rule 1). Labelled so they are easy to find. NOTE: the conformance-runner bump edits conformance/versions.env, a sensitive path (infra/gitea/approval-policy.json), so it will wait for an administrator's /approve-override regardless of this rule.", + "matchManagers": ["custom.regex"], + "matchPackageNames": [ + "gitea.home.keithjackson.dev/home-network/toolbox", + "gitea.home.keithjackson.dev/home-network/agent-runner", + "gitea.home.keithjackson.dev/home-network/conformance-runner" + ], + "addLabels": ["own-image"] } ], // WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A @@ -168,10 +243,20 @@ // shape, not software to keep current. `conformance/**` was already ignored for the same // reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise // have opened. + // conformance/** NARROWED 2026-10-29 to its subdirectories: ignorePaths applies to custom + // managers too (lib/workers/repository/extract/file-match.ts getMatchingFiles -> + // filterIgnoredFiles), so the blanket glob also hid conformance/versions.env, which now carries + // the conformance-runner digest Renovate must maintain. Everything else under conformance/ + // stays ignored, for the original reason (test fixtures with deliberately fixed shapes). "ignorePaths": [ "**/*.example", "**/*.sops.yaml", - "conformance/**", + "conformance/config/**", + "conformance/incluster/**", + "conformance/lib/**", + "conformance/requirements/**", + "conformance/runs/**", + "conformance/tests/**", "platform/image-automation/per-service/**", "platform/kyverno/tests/**" ], diff --git a/runbooks/pm-agent.md b/runbooks/pm-agent.md index 7f30319..a99410e 100644 --- a/runbooks/pm-agent.md +++ b/runbooks/pm-agent.md @@ -67,10 +67,10 @@ Work through these in order; the first three are by far the most likely. | Symptom in the log | Cause | Fix | |---|---|---| | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` | -| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first | +| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been published; the committed digest is the deliberate placeholder | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first | | `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` | | `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours | -| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` | +| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | restore the digest form; the digest comes from CI's publish (`.gitea/workflows/images.yaml` job summary) and Renovate's bump PR, never from a hand push | | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob | | `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) | | `at the worker cap` / `worker limit reached` | the per-cycle cap, draining a backlog one issue at a time | expected; it will continue next cycle | From 8f35234fcc8a2613195f5be95886c2fbad021643 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Thu, 29 Oct 2026 21:44:38 -0500 Subject: [PATCH 2/4] fix(images): name the measured runner blocker; the job container gets no Docker daemon PR #81's first images run failed in 6s: act_runner 0.2.11 forces docker_host '-' for a tcp:// DOCKER_HOST, and gitea/act drops every bind when valid_volumes is empty. Giving jobs the privileged DinD daemon is node-root for any workflow author, so it is left to the operator; options are in the handoff entry. The driver's failure now names the cause. Co-Authored-By: Claude Opus 5.5 --- .gitea/workflows/images.yaml | 9 ++++ .gitea/workflows/scripts/images.sh | 9 +++- decisions/2026-10-29-image-pipeline.md | 67 ++++++++++++++++++++++++-- 3 files changed, 80 insertions(+), 5 deletions(-) diff --git a/.gitea/workflows/images.yaml b/.gitea/workflows/images.yaml index 3d027bc..6170fe4 100644 --- a/.gitea/workflows/images.yaml +++ b/.gitea/workflows/images.yaml @@ -41,6 +41,15 @@ # not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER` # is optional and only labels the login. # +# ┌─ BLOCKER, MEASURED ON PR #81 (run 325): THIS RUNNER CANNOT BUILD IMAGES YET ───────────────┐ +# │ The job container gets no Docker daemon. act_runner 0.2.11 forces container.docker_host to │ +# │ "-" whenever DOCKER_HOST is tcp:// (internal/app/cmd/daemon.go), and with valid_volumes: [] │ +# │ gitea/act strips every bind including the socket (pkg/container/docker_run.go │ +# │ sanitizeConfig). So this job fails at its first check, loudly, until the operator chooses │ +# │ how workflow code may reach a daemon — see decisions/2026-10-29-image-pipeline.md. It is │ +# │ not a required check, so it does not block merges; it is red, which is the honest state. │ +# └──────────────────────────────────────────────────────────────────────────────────────────────┘ +# # THE RUNNER (apps/wave-1/gitea-actions.yaml): act_runner 0.2.11, one job at a time, Docker # executor on a privileged `docker:27-dind` sidecar whose image store is an emptyDir on the node # (no sizeLimit) and whose memory limit is 2Gi — shared by the job container AND every build diff --git a/.gitea/workflows/scripts/images.sh b/.gitea/workflows/scripts/images.sh index 2d83fb4..a36a950 100755 --- a/.gitea/workflows/scripts/images.sh +++ b/.gitea/workflows/scripts/images.sh @@ -200,8 +200,13 @@ push_one() { # -> sets DIGEST[image] to the registry-returned digest log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}" command -v "${ENGINE}" >/dev/null || die "${ENGINE} is not on PATH in the job container" "${ENGINE}" info >/dev/null 2>&1 \ - || die "the ${ENGINE} daemon is not reachable from the job container — the runner did not - hand it a Docker socket (act_runner container.docker_host). Nothing can be built." + || die "the ${ENGINE} daemon is not reachable from the job container, so nothing can be built. + MEASURED ON THIS RUNNER (PR #81, 2026-10-29): act_runner 0.2.11 never gives a job a + daemon. Its DOCKER_HOST is tcp:// (the DinD sidecar), and internal/app/cmd/daemon.go then + forces container.docker_host to '-' (\"can't be mounted\"); separately, with + valid_volumes: [] gitea/act's sanitizeConfig drops EVERY bind, the socket included. + Fixing it means handing workflow code the privileged DinD daemon — an operator security + decision, set out in decisions/2026-10-29-image-pipeline.md. Not a fault in this script." "${ENGINE}" version 2>/dev/null | sed -n '1,12p' || true "${ENGINE}" system df || true diff --git a/decisions/2026-10-29-image-pipeline.md b/decisions/2026-10-29-image-pipeline.md index 7fa2979..32e5a68 100644 --- a/decisions/2026-10-29-image-pipeline.md +++ b/decisions/2026-10-29-image-pipeline.md @@ -83,6 +83,56 @@ both `build-image.sh` scripts said "publishing is an operator action". `infra/devcontainer/README.md`, and a new "How images are built" section in `.gitea/workflows/README.md`. +### BLOCKER — the runner cannot build images, and fixing that is a security decision + +**Measured on this PR's own CI run (images run 325): the job failed in 6 s** with +`failed to connect to the docker API at unix:///var/run/docker.sock ... no such file`. The job +container is given **no Docker daemon at all**, for two independent reasons, both read from the +exact versions deployed: + +- act_runner **v0.2.11** `internal/app/cmd/daemon.go`: when `DOCKER_HOST` has a scheme other than + `unix`/`npipe` — here `tcp://localhost:2376`, the DinD sidecar — it sets + `cfg.Container.DockerHost = "-"`, which means "never mount into job containers". +- gitea/act **v0.261.2** (act_runner's `go.mod` replace) `pkg/container/docker_run.go` + `sanitizeConfig`: with `valid_volumes` empty (the runner has no config file, so the default + `[]`), `hostConfig.Binds = []string{}`. **Every** bind is dropped, the daemon socket included, + and so is any `volumes:` a workflow asks for. + +I assumed the opposite while designing (from act's `getDockerDaemonSocketMountPath`, which never +gets the chance to run). That was wrong, and the PR run is what showed it. + +**Why I did not fix it in this PR.** Any fix gives workflow code the DinD daemon. That daemon is +`privileged: true`, so `docker run --privileged -v /:/host` from a job is **root on the node**. +Today a malicious workflow — for example one pushed by a prompt-injected pm-agent worker, which +holds a repo write token — runs in an unprivileged container and holds no cluster credential. +After the fix it does not. That is a change to the security boundary, and it is the operator's +call. It also cannot be tested from a PR anyway: the runner only changes after merge + Flux. + +**Options, for the operator:** + +1. **Socket to every job (smallest change):** in `apps/wave-1/gitea-actions.yaml`, + - mount a shared `emptyDir` at `/var/run` in both the `dind` sidecar and the `runner`, so + dockerd's own `unix:///var/run/docker.sock` is visible at the same path in both; + - set the runner's `DOCKER_HOST=unix:///var/run/docker.sock`, and drop `DOCKER_TLS_VERIFY` + and `DOCKER_CERT_PATH`, because TLS over a unix socket breaks the client; + - add an act_runner config file (`CONFIG_FILE`) with `container.valid_volumes: + ["/var/run/docker.sock"]` and `runner.file: /data/.runner`, so the existing registration + is kept. + + Consequence: any workflow on any branch can reach node root. +2. **Option 1 on a SECOND runner** with its own label (e.g. `image-builder`) and its own DinD, + with `images.yaml` `runs-on` that label. The gate runner stays socket-less, and image builds + stop competing with the gate for the single slot and its 7 GB node disk. **It is not a + security boundary**: Gitea cannot restrict which workflow may target a label, so any branch + could still `runs-on: image-builder`. +3. **A builder that is not a privileged daemon**, e.g. a rootless BuildKit sidecar exposed to + jobs. That bounds a job to BuildKit's rootless sandbox instead of node root. It needs the + verification steps rewritten, because they *run* the image (as uid 65532, `--read-only`), + which BuildKit alone does not do. This is a larger piece of work. + +Nothing in this PR has to change for options 1 or 2 except `runs-on` for option 2. The driver +already produced verified images with a stock docker 27 CLI (see *What I verified*). + ### Decisions and pushback - **The push credential is a PAT secret, `REGISTRY_PUSH_TOKEN`; the Actions token cannot push.** @@ -156,6 +206,14 @@ both `build-image.sh` scripts said "publishing is an operator action". - agent-runner: `PM_SELFTEST=1 pm-dispatch` OK, and the stdin-fed `probe-validation.sh` gave "20 refusal(s), 0 failure(s)"; - conformance-runner: all seven `conformance-clone` refusals. +- **The docker-CLI code path**, i.e. what CI runs. I ran the same driver inside + `docker:27-cli` (`ENGINE=docker`, `DOCKER_BUILDKIT=0`) against podman's Docker-compatible API + socket. Exit code 0; all three images built and verified (selftest OK; probe OK, 20 refusals). + This covers the docker-specific calls (`image inspect` formats, `run -i` with stdin, + `--tmpfs`, `--read-only`, `--user`). It does **not** cover BuildKit or a real DinD daemon. +- **This PR's CI, run 325 (`images`): FAILED in 6 s**, because there is no Docker daemon in the + job container (see BLOCKER). Gate run 324 failed only on `approval-policy` (expected, + sensitive paths). Every other stage passed there, including `gitea-settings`. - **Other driver paths, run locally:** - agent-runner alone builds the toolbox as its base; - an empty diff builds nothing; @@ -198,13 +256,16 @@ both `build-image.sh` scripts said "publishing is an operator action". finalising a blob: copying the upload into package storage on Longhorn. A ~250 MB compressed layer should finish well inside 15 s, but I could not measure it without pushing. If the first publish shows 504s on blob PUTs, set `timeouts.request` on the gitea HTTPRoute. -- **DinD fit is estimated, not measured** (until the PR run reports): +- **DinD fit is estimated, not measured** (no build has run there, see the BLOCKER): - memory: 2Gi is shared by the job container and every build container; - disk: the toolbox, agent-runner, conformance-runner, Fedora base and build cache should total roughly 3 GB on top of the 1.5 GB act image, against 7 GB free on the node, with `dind-storage` having no sizeLimit. -- **That act_runner hands the job container a Docker socket is inferred** from act_runner's - DOCKER_HOST handling, not measured. The driver fails loudly if it does not. +- **No image has been built in this runner's DinD**, because it cannot be done until the BLOCKER + is resolved. Two things are therefore unmeasured until then: + - DinD memory (2Gi) and disk fit; + - BuildKit (`buildx` v0.20.1 is on the daemon) as opposed to the legacy builder I tested + with. - **Renovate has still never completed a run here.** Three things block it: - #80, the `configurationFile` name; - `RENOVATE_TOKEN` gaining `read:package`; From f65c12b3953f8adf6a14d43dc7096f48ee9311ba Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Thu, 29 Oct 2026 22:21:03 -0500 Subject: [PATCH 3/4] feat(images): build on rootless BuildKit over mTLS; verification moves into Containerfile verify stages Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched. - platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined, read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP :1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it). No Kyverno exception: every applied policy matches app-namespace=true only. - Each Containerfile names its shipped stage and ends in a `verify` stage holding every former `docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now. - images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes. Fails loudly when the builder secrets are absent or buildkitd is unreachable. - infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets. Co-Authored-By: Claude Opus 5.5 --- .gitea/workflows/README.md | 24 +- .gitea/workflows/images.yaml | 54 +- .gitea/workflows/scripts/images.sh | 315 +++++++----- clusters/base/buildkit.yaml | 21 + clusters/base/kustomization.yaml | 1 + conformance/incluster/Containerfile | 88 +++- conformance/incluster/README.md | 10 +- conformance/incluster/build-image.sh | 114 ++--- .../alert-rules-unit-tested/check.sh | 4 +- decisions/2026-10-29-image-pipeline.md | 474 ++++++++++-------- docs/plans/pm-agent-issue-to-pr.md | 4 +- infra/agent/Containerfile | 82 ++- infra/agent/build-image.sh | 130 ++--- infra/agent/probe-validation.sh | 10 +- infra/devcontainer/Containerfile | 50 +- infra/devcontainer/README.md | 8 +- infra/devcontainer/build-image.sh | 98 ++-- infra/gitea/buildkit-client-secret.sh | 104 ++++ platform/buildkit/README.md | 74 +++ platform/buildkit/certificates.yaml | 100 ++++ platform/buildkit/configmap.yaml | 24 + platform/buildkit/deployment.yaml | 142 ++++++ platform/buildkit/kustomization.yaml | 13 + platform/buildkit/namespace.yaml | 26 + platform/buildkit/networkpolicy.yaml | 88 ++++ platform/buildkit/service.yaml | 19 + platform/buildkit/versions.env | 15 + platform/pm-agent/README.md | 6 +- 28 files changed, 1490 insertions(+), 608 deletions(-) create mode 100644 clusters/base/buildkit.yaml create mode 100755 infra/gitea/buildkit-client-secret.sh create mode 100644 platform/buildkit/README.md create mode 100644 platform/buildkit/certificates.yaml create mode 100644 platform/buildkit/configmap.yaml create mode 100644 platform/buildkit/deployment.yaml create mode 100644 platform/buildkit/kustomization.yaml create mode 100644 platform/buildkit/namespace.yaml create mode 100644 platform/buildkit/networkpolicy.yaml create mode 100644 platform/buildkit/service.yaml create mode 100644 platform/buildkit/versions.env diff --git a/.gitea/workflows/README.md b/.gitea/workflows/README.md index c42412c..ace6abc 100644 --- a/.gitea/workflows/README.md +++ b/.gitea/workflows/README.md @@ -120,10 +120,20 @@ CI. That is a real coverage gap, not a phase. This repo's three images — `toolbox` (`infra/devcontainer/`), `agent-runner` (`infra/agent/`, built `FROM` the toolbox) and `conformance-runner` (`conformance/incluster/`) — are built, verified and published by [`images.yaml`](images.yaml), never from a workstation -(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh); each -image is built by **its own** `build-image.sh`, the same script a developer runs locally, so the -in-image verification (non-root uid, read-only rootfs, tool presence, `PM_SELFTEST=1 -pm-dispatch`, the `conformance-clone` refusals) exists once. +(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh). + +**The builder is rootless BuildKit** ([`platform/buildkit/`](../../platform/buildkit/README.md)), +reached over mTLS with a pinned `buildx` — no job here touches a Docker daemon, privileged or +otherwise (operator decision on PR #81). The client certificate reaches jobs as the Actions +secrets `BUILDKIT_CA`, `BUILDKIT_CLIENT_CERT`, `BUILDKIT_CLIENT_KEY`. + +**Verification is a build target.** Each Containerfile ends in a `verify` stage that runs every +in-image assertion at build time as the non-root uid with `RUN --network=none` (tool presence and +pinned versions, `PM_SELFTEST=1 pm-dispatch`, the pm-worker probe, the `conformance-clone` +refusals, no writable path outside the workspace). The pipeline builds `-verify` before it +builds, let alone pushes, the image. Build args come from each image's own +`build-image.sh --print-build-args`, which is also the local podman path — so the recipe and the +proof exist once. | Trigger | What happens | |---|---| @@ -143,8 +153,10 @@ account with package write on the org). The automatic Actions token cannot push [`infra/gitea/registry-retention.sh`](../../infra/gitea/registry-retention.sh), an operator action, keeps every digest git pins plus the newest tags. -Cost on this runner: the job shares the single act_runner slot with the gate, and the DinD store -is a node `emptyDir`; the job prunes it on exit. +Cost: the job shares the single act_runner slot with the gate, but the build itself runs in the +`buildkitd` pod, not in DinD, so the runner's disk and memory are untouched. Until +`platform/buildkit` is deployed and its client certificate copied +(`infra/gitea/buildkit-client-secret.sh`), the job fails loudly at its builder preflight. ## Files diff --git a/.gitea/workflows/images.yaml b/.gitea/workflows/images.yaml index 6170fe4..bbe700c 100644 --- a/.gitea/workflows/images.yaml +++ b/.gitea/workflows/images.yaml @@ -41,21 +41,22 @@ # not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER` # is optional and only labels the login. # -# ┌─ BLOCKER, MEASURED ON PR #81 (run 325): THIS RUNNER CANNOT BUILD IMAGES YET ───────────────┐ -# │ The job container gets no Docker daemon. act_runner 0.2.11 forces container.docker_host to │ -# │ "-" whenever DOCKER_HOST is tcp:// (internal/app/cmd/daemon.go), and with valid_volumes: [] │ -# │ gitea/act strips every bind including the socket (pkg/container/docker_run.go │ -# │ sanitizeConfig). So this job fails at its first check, loudly, until the operator chooses │ -# │ how workflow code may reach a daemon — see decisions/2026-10-29-image-pipeline.md. It is │ -# │ not a required check, so it does not block merges; it is red, which is the honest state. │ -# └──────────────────────────────────────────────────────────────────────────────────────────────┘ +# THE BUILDER — rootless BuildKit (operator decision on PR #81, 2026-10-29). +# The act runner's job containers get NO Docker daemon (act_runner 0.2.11 forces docker_host "-" +# for a tcp:// DinD, and gitea/act strips every bind when valid_volumes is empty — measured on +# this PR's first run), and handing them the privileged DinD daemon would be root on the node for +# anyone who can push a branch. So images are built by platform/buildkit/: a ROOTLESS buildkitd +# (not privileged; uid 1000 in a user namespace), reached over mTLS at +# buildkitd.buildkit.svc.cluster.local:1234 with a pinned `buildx` (remote driver). Its client +# certificate reaches jobs as three Actions secrets — BUILDKIT_CA, BUILDKIT_CLIENT_CERT, +# BUILDKIT_CLIENT_KEY — copied from cert-manager by infra/gitea/buildkit-client-secret.sh. +# Pull requests get the client certificate (they must build) but never a registry credential. +# +# UNTIL platform/buildkit IS DEPLOYED (Flux applies it only after merge) THIS JOB FAILS, LOUDLY, +# at its builder preflight. A skipped Gitea job reports `success`, so it is red on purpose. +# +# VERIFICATION is each Containerfile's `verify` stage, built before anything is pushed. # -# THE RUNNER (apps/wave-1/gitea-actions.yaml): act_runner 0.2.11, one job at a time, Docker -# executor on a privileged `docker:27-dind` sidecar whose image store is an emptyDir on the node -# (no sizeLimit) and whose memory limit is 2Gi — shared by the job container AND every build -# container. The toolbox is ~1.4 GB uncompressed and agent-runner adds ~0.3 GB on top of it, so the -# cleanup step below is not optional: without it each run would leave ~3 GB on a node that had -# 7 GB free when this was written, and the next eviction would take the runner with it. name: images on: @@ -71,6 +72,7 @@ on: - conformance/incluster/build-image.sh - .gitea/workflows/images.yaml - .gitea/workflows/scripts/images.sh + - platform/buildkit/versions.env push: branches: [main] paths: @@ -83,6 +85,7 @@ on: - conformance/incluster/build-image.sh - .gitea/workflows/images.yaml - .gitea/workflows/scripts/images.sh + - platform/buildkit/versions.env workflow_dispatch: inputs: images: @@ -115,6 +118,11 @@ jobs: MODE: verify IMAGES: auto PR_BASE_REF: ${{ github.base_ref }} + # The BUILDER's client certificate (platform/buildkit). Not a registry credential: + # a pull request can build, never push. + BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }} + BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }} + BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }} run: | set -euo pipefail BASE_SHA="$(git merge-base "origin/${PR_BASE_REF}" HEAD)" @@ -127,6 +135,9 @@ jobs: env: MODE: verify IMAGES: ${{ inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images }} + BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }} + BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }} + BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }} run: ./.gitea/workflows/scripts/images.sh # ── main: publish ─────────────────────────────────────────────────────────────────────── @@ -168,16 +179,7 @@ jobs: BASE_SHA: ${{ github.event.before }} REGISTRY_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }} REGISTRY_USER: ${{ vars.REGISTRY_PUSH_USER }} + BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }} + BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }} + BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }} run: ./.gitea/workflows/scripts/images.sh - - # ── always: give the DinD store back ──────────────────────────────────────────────────── - # The runner's image store is a node emptyDir with no sizeLimit (see header). Everything this - # job built or pulled is removed; `prune -a` cannot remove the job's own image, because the - # job container is still running on it. - - name: Clean the DinD image store - if: always() - run: | - docker logout gitea.home.keithjackson.dev >/dev/null 2>&1 || true - docker image prune -af >/dev/null || true - docker builder prune -af >/dev/null || true - docker system df || true diff --git a/.gitea/workflows/scripts/images.sh b/.gitea/workflows/scripts/images.sh index a36a950..3243d83 100755 --- a/.gitea/workflows/scripts/images.sh +++ b/.gitea/workflows/scripts/images.sh @@ -4,35 +4,41 @@ # # Builds, verifies and (on main) PUBLISHES the three images this repo ships: # -# toolbox infra/devcontainer/ the pinned dev/gate toolchain -# agent-runner infra/agent/ FROM toolbox; the PM dispatcher + worker +# toolbox infra/devcontainer/ the pinned dev/gate toolchain +# agent-runner infra/agent/ FROM toolbox; the PM dispatcher + worker # conformance-runner conformance/incluster/ the in-cluster L2/L3 Chainsaw runner # -# Each image is built by ITS OWN build-image.sh — the same script a developer runs locally — -# so the in-image verification (non-root uid, read-only rootfs, tool presence, pm-dispatch -# selftest, conformance-clone refusals, ...) is written once and runs in both places. This -# file only decides WHAT to build, in which order, FROM which base, and where it goes. +# THE BUILDER (operator decision on PR #81): a ROOTLESS BuildKit daemon, platform/buildkit/, +# reached over mTLS. No CI job may reach a privileged daemon, so there is no `docker build` and no +# `docker run` here: this script drives buildkitd with a pinned `buildx` (remote driver) and +# nothing else. The client certificate arrives as Actions secrets (BUILDKIT_CA, +# BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY; infra/gitea/buildkit-client-secret.sh puts them +# there). If the daemon is unreachable this FAILS LOUDLY — it never skips. +# +# VERIFICATION IS A BUILD TARGET. Each Containerfile ends in a `verify` stage that runs every +# in-image assertion at build time (as the non-root uid, with RUN --network=none); this script +# builds `-verify` and only then the image. Build args come from each image's own +# `build-image.sh --print-build-args`, the same list the local podman path uses. # # MODES (env MODE): -# verify build + verify, never push, holds NO credential. Pull requests. -# publish build + verify + push to ${REGISTRY}/${NAMESPACE}/ with tags +# verify build the verify targets, never push, hold no registry credential. Pull requests. +# publish verify, then build + push to ${REGISTRY}/${NAMESPACE}/ with tags # `sha-<12 hex>` (immutable: never re-pushed) and `main` (moving; Renovate follows # it). The digest recorded is the one the REGISTRY returns, read back after the push. # # WHAT GETS BUILT (env IMAGES): `auto` (default) diffs BASE_SHA..HEAD_SHA against each image's # inputs; otherwise a space-separated list. A change to the pipeline itself (images.yaml, this -# file) builds everything in verify mode, and nothing extra in publish mode: re-publishing -# unchanged inputs would only mint new digests and churn Renovate PRs. +# file, platform/buildkit/versions.env) builds everything in verify mode, and nothing extra in +# publish mode: re-publishing unchanged inputs would only mint new digests and churn Renovate PRs. # -# THE agent-runner BASE — the one ordering rule, and the reason this is not three jobs: -# * toolbox built in this run -> agent-runner builds FROM it (in publish mode, FROM the -# digest the registry just returned for it). -# * verify mode, toolbox not changed -> the toolbox is built anyway as a base. A pull request -# holds no registry credential, so it cannot pull the published one. -# * publish mode, toolbox not changed -> AGENT_BASE_IMAGE from infra/agent/versions.env, -# i.e. the published toolbox digest that Renovate keeps current. If that is still the -# all-zero placeholder the toolbox has never been published, so it is built and published -# first (this is how the very first run bootstraps). +# THE agent-runner BASE: +# * verify mode -> FROM the toolbox built in this run, linked with a bake `target:` context — +# a pull request holds no registry credential and that toolbox exists in no registry. If the +# toolbox's inputs did not change it is built anyway, as the base. +# * publish mode, toolbox published in this run -> FROM the digest the registry just returned. +# * publish mode otherwise -> AGENT_BASE_IMAGE from infra/agent/versions.env (the published +# toolbox digest Renovate keeps current). If that is still the all-zero placeholder the +# toolbox has never been published, so it is built and published first (the bootstrap). # # LOOP-FREEDOM. Renovate turns each publish into a digest-bump PR. Those bumps land in # platform/pm-agent/ (no image input), conformance/versions.env (no image input) and @@ -46,9 +52,14 @@ set -euo pipefail REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" cd "${REPO_ROOT}" +# shellcheck disable=SC1091 +BK_HOST_OVERRIDE="${BUILDKIT_HOST:-}"; BK_NAME_OVERRIDE="${BUILDKIT_SERVERNAME:-}" +. ./platform/buildkit/versions.env +BUILDKIT_HOST="${BK_HOST_OVERRIDE:-${BUILDKIT_HOST}}" +BUILDKIT_SERVERNAME="${BK_NAME_OVERRIDE:-${BUILDKIT_SERVERNAME}}" + MODE="${MODE:-verify}" IMAGES="${IMAGES:-auto}" -ENGINE="${ENGINE:-docker}" REGISTRY="${REGISTRY:-gitea.home.keithjackson.dev}" NAMESPACE="${NAMESPACE:-home-network}" HEAD_SHA="${HEAD_SHA:-$(git rev-parse HEAD)}" @@ -63,13 +74,15 @@ MOVING_TAG="main" log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; } info() { printf ' %s\n' "$*"; } die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; printf -- '- **ERROR**: %s\n' "$*" >>"${SUMMARY}"; exit 1; } +yn() { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; } case "${MODE}" in verify|publish) ;; *) die "MODE must be verify or publish, got '${MODE}'" ;; esac +for t in git jq curl sha256sum; do command -v "$t" >/dev/null || die "the job image lacks '$t'"; done -# --- inputs per image: a change under any of these rebuilds that image --------------------- -# conformance-runner takes its chainsaw/kubectl/fedora pins from infra/devcontainer/versions.env. -# conformance/versions.env is NOT an input (it only RECORDS the published digest), and must not -# become one: Renovate writes it after every publish, and an input there would loop. +# --- per-image facts ------------------------------------------------------------------------ +# Inputs: a change under any of these rebuilds that image. conformance-runner takes its pins from +# infra/devcontainer/versions.env. conformance/versions.env is NOT an input (it only RECORDS the +# published digest), and must not become one: Renovate writes it after every publish -> a loop. inputs() { case "$1" in toolbox) echo "infra/devcontainer/Containerfile infra/devcontainer/versions.env infra/devcontainer/build-image.sh" ;; @@ -77,16 +90,17 @@ inputs() { conformance-runner) echo "conformance/incluster/Containerfile conformance/incluster/conformance-clone conformance/incluster/build-image.sh infra/devcontainer/versions.env" ;; esac } -PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh" +context_of() { case "$1" in toolbox) echo infra/devcontainer ;; agent-runner) echo infra/agent ;; conformance-runner) echo conformance/incluster ;; esac; } +PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh platform/buildkit/versions.env" declare -A WANT=() BUILT=() DIGEST=() SECS=() NOTE=() changed_any() { # -> 0 if BASE_SHA..HEAD_SHA touches any - # shellcheck disable=SC2086 [ -n "$(git diff --name-only "${BASE_SHA}" "${HEAD_SHA}" -- "$@")" ] } select_images() { + local i if [ "${IMAGES}" != "auto" ]; then for i in ${IMAGES}; do case " ${ALL_IMAGES[*]} " in *" ${i} "*) WANT[$i]=1; NOTE[$i]="requested" ;; *) die "unknown image '${i}'" ;; esac @@ -99,7 +113,6 @@ select_images() { return fi info "diff base ${BASE_SHA} .. ${HEAD_SHA}" - local i for i in "${ALL_IMAGES[@]}"; do # shellcheck disable=SC2046 if changed_any $(inputs "$i"); then WANT[$i]=1; NOTE[$i]="inputs changed"; fi @@ -110,6 +123,114 @@ select_images() { fi } +# --- the builder: pinned buildx + mTLS to rootless buildkitd --------------------------------- +WORK="$(mktemp -d)" +trap 'rm -rf "${WORK}"' EXIT # client key + registry auth live only here, only this long +export BUILDX_CONFIG="${WORK}/buildx" DOCKER_CONFIG="${WORK}/docker" +# Single-manifest images: the admission policies, Renovate and registry-retention.sh all reason +# about one manifest per tag. (Provenance attestations would turn every push into an index.) +export BUILDX_NO_DEFAULT_ATTESTATIONS=1 +mkdir -p "${BUILDX_CONFIG}" "${DOCKER_CONFIG}" "${WORK}/certs" +echo '{}' > "${DOCKER_CONFIG}/config.json" + +setup_builder() { + log "Builder: rootless buildkitd at ${BUILDKIT_HOST} (mTLS)" + BUILDX="${BUILDX_BIN:-}" + if [ -z "${BUILDX}" ]; then + BUILDX="${WORK}/buildx-bin" + curl -fsSL --retry 3 -o "${BUILDX}" \ + "https://github.com/docker/buildx/releases/download/v${BUILDX_VERSION}/buildx-v${BUILDX_VERSION}.linux-amd64" \ + || die "could not download buildx v${BUILDX_VERSION}" + echo "${BUILDX_SHA256} ${BUILDX}" | sha256sum -c --quiet - \ + || die "buildx v${BUILDX_VERSION} does not match the sha256 pinned in platform/buildkit/versions.env — refusing to run it" + chmod +x "${BUILDX}" + fi + info "client: $("${BUILDX}" version)" + + local missing=() + [ -n "${BUILDKIT_CA:-}" ] || missing+=(BUILDKIT_CA) + [ -n "${BUILDKIT_CLIENT_CERT:-}" ] || missing+=(BUILDKIT_CLIENT_CERT) + [ -n "${BUILDKIT_CLIENT_KEY:-}" ] || missing+=(BUILDKIT_CLIENT_KEY) + [ "${#missing[@]}" -eq 0 ] || die "the builder's client certificate is missing: ${missing[*]} not set. + These Actions secrets are copied from the cert-manager Secret buildkit/buildkit-client-ci by + infra/gitea/buildkit-client-secret.sh (an operator action, after platform/buildkit is + deployed). Without them nothing can be built — this fails rather than skipping ON PURPOSE." + umask 077 + printf '%s\n' "${BUILDKIT_CA}" > "${WORK}/certs/ca.pem" + printf '%s\n' "${BUILDKIT_CLIENT_CERT}" > "${WORK}/certs/cert.pem" + printf '%s\n' "${BUILDKIT_CLIENT_KEY}" > "${WORK}/certs/key.pem" + umask 022 + + "${BUILDX}" create --name ci-remote --driver remote \ + --driver-opt "cacert=${WORK}/certs/ca.pem,cert=${WORK}/certs/cert.pem,key=${WORK}/certs/key.pem,servername=${BUILDKIT_SERVERNAME}" \ + "${BUILDKIT_HOST}" >/dev/null + if ! out="$(timeout 90 "${BUILDX}" inspect --bootstrap --builder ci-remote 2>&1)"; then + printf '%s\n' "${out}" >&2 + die "buildkitd is UNREACHABLE at ${BUILDKIT_HOST} — nothing can be built, and this job fails + rather than skipping. In order of likelihood: + 1. platform/buildkit is not deployed yet. Flux applies it only after it merges to main + (Kustomization 'buildkit'); a pull request that ADDS it cannot reach it. + 2. the BUILDKIT_* secrets are stale (the client cert was renewed) or from another CA: + re-run infra/gitea/buildkit-client-secret.sh. + 3. the NetworkPolicy (platform/buildkit/networkpolicy.yaml) does not admit this runner." + fi + printf '%s\n' "${out}" | grep -E '^(Name|Driver|Status|BuildKit version|Platforms):' | sed 's/^/ /' || true +} + +# KEY=VALUE lines from an image's build-image.sh --print-build-args -> a JSON object. +build_args_json() { + local ctx; ctx="$(context_of "$1")" + "./${ctx}/build-image.sh" --print-build-args | jq -Rn '[inputs | select(length>0) | capture("^(?[^=]+)=(?.*)$")] | map({(.k): .v}) | add' +} + +# bake_run +bake_run() { + local f="$1"; shift + "${BUILDX}" bake --builder ci-remote --progress plain -f "$f" \ + --metadata-file "${WORK}/meta.json" "$@" +} + +# Write a bake file with a verify target and a ship target for one image. +# $1 image, $2 agent base mode: "" | "link" | +write_bake() { + local img="$1" base="${2:-}" ctx args f + ctx="$(context_of "$img")" + args="$(build_args_json "$img")" + f="${WORK}/bake-${img}.json" + local contexts='{}' + if [ "${img}" = agent-runner ]; then + if [ "${base}" = link ]; then + # FROM ${AGENT_BASE_IMAGE} resolves to the named context below, i.e. the toolbox target + # of this same bake file — built here, never pushed anywhere. + args="$(jq -c '. + {AGENT_BASE_IMAGE: "toolbox-in-this-run"}' <<<"${args}")" + contexts='{"toolbox-in-this-run": "target:toolbox"}' + else + args="$(jq -c --arg b "${base}" '. + {AGENT_BASE_IMAGE: $b}' <<<"${args}")" + fi + fi + jq -n --arg img "$img" --arg ctx "$ctx" --argjson args "$args" --argjson contexts "$contexts" \ + --arg tbargs "$( [ "${base}" = link ] && build_args_json toolbox || echo '{}')" \ + --arg rev "${HEAD_SHA}" --arg repo "${REGISTRY}/${NAMESPACE}/${img}" \ + --arg sha "${SHA_TAG}" --arg moving "${MOVING_TAG}" --arg mode "${MODE}" ' + {target: ( + { + ($img + "-verify"): {context: $ctx, dockerfile: "Containerfile", target: "verify", + args: $args, contexts: $contexts, output: ["type=cacheonly"]}, + ($img): ({context: $ctx, dockerfile: "Containerfile", target: $img, args: $args, + contexts: $contexts, + labels: {"org.opencontainers.image.revision": $rev}} + + (if $mode == "publish" + then {tags: [($repo + ":" + $sha), ($repo + ":" + $moving)], output: ["type=image,push=true"]} + else {output: ["type=cacheonly"]} end)) + } + + (if ($contexts | has("toolbox-in-this-run")) + then {toolbox: {context: "infra/devcontainer", dockerfile: "Containerfile", target: "toolbox", + args: ($tbargs | fromjson), output: ["type=cacheonly"]}} + else {} end) + )}' > "$f" + echo "$f" +} + # --- registry helpers (publish mode only) ---------------------------------------------------- REG_TOKEN="" registry_token() { @@ -120,8 +241,8 @@ registry_token() { [ -n "${REG_TOKEN}" ] || die "registry returned no bearer token" } # Prints the manifest digest the REGISTRY serves for :; nothing if it is absent (404). -# Any other answer (401, 5xx, no digest header) is fatal: "cannot tell" must never be read as -# "not published", or the immutable-tag guard would be bypassed by an auth hiccup. +# Any other answer is fatal: "cannot tell" must never be read as "not published", or the +# immutable-tag guard would be bypassed by an auth hiccup. registry_digest() { registry_token local hdrs code digest @@ -142,87 +263,50 @@ registry_digest() { esac } -local_ref() { echo "${REGISTRY}/${NAMESPACE}/$1:ci-${HEAD_SHA:0:12}"; } - -build_one() { # [agent base] - local img="$1" base="${2:-}" t0 tag - tag="ci-${HEAD_SHA:0:12}" +# build_one [agent base] — verify, then (publish) build + push + read back. +build_one() { + local img="$1" base="${2:-}" t0 f pushed got existing t0=$(date +%s) - log "BUILD + VERIFY ${img}" - case "${img}" in - toolbox) - ENGINE="${ENGINE}" IMAGE="$(local_ref toolbox)" BUILD_REVISION="${HEAD_SHA}" \ - ./infra/devcontainer/build-image.sh ;; - agent-runner) - ENGINE="${ENGINE}" IMAGE_TAG="${tag}" AGENT_BASE_IMAGE="${base}" BUILD_REVISION="${HEAD_SHA}" \ - ./infra/agent/build-image.sh ;; - conformance-runner) - ENGINE="${ENGINE}" IMAGE_TAG="${tag}" BUILD_REVISION="${HEAD_SHA}" \ - ./conformance/incluster/build-image.sh ;; - esac + f="$(write_bake "$img" "$base")" + log "VERIFY ${img} (Containerfile target: verify, on rootless BuildKit)" + bake_run "$f" "${img}-verify" BUILT[$img]=1 - SECS[$img]=$(( $(date +%s) - t0 )) - info "${img}: built and verified in ${SECS[$img]}s" -} - -push_one() { # -> sets DIGEST[image] to the registry-returned digest - local img="$1" repo src existing pushed got t0 - repo="${REGISTRY}/${NAMESPACE}/${img}" - src="$(local_ref "${img}")" - t0=$(date +%s) - log "PUSH ${img} -> ${repo}:{${SHA_TAG},${MOVING_TAG}}" - existing="$(registry_digest "${img}" "${SHA_TAG}")" - if [ -n "${existing}" ]; then + if [ "${MODE}" = publish ]; then + existing="$(registry_digest "${img}" "${SHA_TAG}")" # Immutable tag: never overwrite. A rebuild is not byte-reproducible, so re-pushing would # silently change what `sha-` means after something may already have pinned it. - die "${repo}:${SHA_TAG} already exists (${existing}). Tags here are immutable; this run must not - replace it. (publish skips images that already exist at this commit BEFORE building them, - so reaching this line means the tag appeared mid-run.)" + [ -z "${existing}" ] || die "${REGISTRY}/${NAMESPACE}/${img}:${SHA_TAG} already exists (${existing}); tags are immutable" + log "BUILD + PUSH ${img} -> ${REGISTRY}/${NAMESPACE}/${img}:{${SHA_TAG},${MOVING_TAG}}" + bake_run "$f" "${img}" + pushed="$(jq -r --arg t "$img" '.[$t]["containerimage.digest"] // empty' "${WORK}/meta.json")" + # The AUTHORITATIVE digest is what the registry serves, read back — not what the client + # reported. They should agree; if they do not, nothing downstream may trust either. + got="$(registry_digest "${img}" "${SHA_TAG}")" + [ -n "${got}" ] || die "pushed ${img}:${SHA_TAG} but the registry does not serve it" + [ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] || die "${img}: client reported ${pushed} but the registry serves ${got}" + [ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \ + || die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})" + DIGEST[$img]="${got}" + info "${img}: registry digest ${got}" fi - "${ENGINE}" tag "${src}" "${repo}:${SHA_TAG}" - "${ENGINE}" tag "${src}" "${repo}:${MOVING_TAG}" - pushed="$("${ENGINE}" push "${repo}:${SHA_TAG}" | tee /dev/stderr | grep -oE 'digest: sha256:[0-9a-f]{64}' | tail -1 | cut -d' ' -f2)" - "${ENGINE}" push "${repo}:${MOVING_TAG}" >/dev/null - # The AUTHORITATIVE digest is what the registry serves, read back — not what the client - # printed. They should agree; if they do not, nothing downstream may trust either. - got="$(registry_digest "${img}" "${SHA_TAG}")" - [ -n "${got}" ] || die "pushed ${repo}:${SHA_TAG} but the registry does not serve it" - [ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] \ - || die "${img}: client reported ${pushed} but the registry serves ${got}" - [ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \ - || die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})" - DIGEST[$img]="${got}" - SECS[$img]=$(( ${SECS[$img]:-0} + $(date +%s) - t0 )) - info "${img}: registry digest ${got}" + SECS[$img]=$(( $(date +%s) - t0 )) + info "${img}: done in ${SECS[$img]}s" } # --- main --------------------------------------------------------------------------------- log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}" -command -v "${ENGINE}" >/dev/null || die "${ENGINE} is not on PATH in the job container" -"${ENGINE}" info >/dev/null 2>&1 \ - || die "the ${ENGINE} daemon is not reachable from the job container, so nothing can be built. - MEASURED ON THIS RUNNER (PR #81, 2026-10-29): act_runner 0.2.11 never gives a job a - daemon. Its DOCKER_HOST is tcp:// (the DinD sidecar), and internal/app/cmd/daemon.go then - forces container.docker_host to '-' (\"can't be mounted\"); separately, with - valid_volumes: [] gitea/act's sanitizeConfig drops EVERY bind, the socket included. - Fixing it means handing workflow code the privileged DinD daemon — an operator security - decision, set out in decisions/2026-10-29-image-pipeline.md. Not a fault in this script." -"${ENGINE}" version 2>/dev/null | sed -n '1,12p' || true -"${ENGINE}" system df || true +select_images if [ "${MODE}" = publish ]; then : "${REGISTRY_TOKEN:?REGISTRY_TOKEN is required in publish mode}" REGISTRY_USER="${REGISTRY_USER:-ci-image-publisher}" - printf '%s' "${REGISTRY_TOKEN}" | "${ENGINE}" login "${REGISTRY}" -u "${REGISTRY_USER}" --password-stdin >/dev/null \ - || die "docker login ${REGISTRY} failed — check REGISTRY_PUSH_TOKEN (scope write:package, owner has package write on the org)" -fi - -select_images - -# Publish idempotence: an image already published at THIS commit is not rebuilt. This is what -# makes a re-run after a half-finished publish safe — and it is the only way a rerun can -# finish, because the immutable sha tag must never be pushed twice. -if [ "${MODE}" = publish ]; then + # buildx (and through it buildkitd, which does the push) reads registry auth from this file. + jq -n --arg r "${REGISTRY}" --arg a "$(printf '%s:%s' "${REGISTRY_USER}" "${REGISTRY_TOKEN}" | base64 -w0)" \ + '{auths: {($r): {auth: $a}}}' > "${DOCKER_CONFIG}/config.json" + chmod 600 "${DOCKER_CONFIG}/config.json" + registry_token + # Publish idempotence: an image already published at THIS commit is not rebuilt. That is what + # makes a re-run after a half-finished publish safe — the immutable tag is never pushed twice. for i in "${ALL_IMAGES[@]}"; do [ -n "${WANT[$i]:-}" ] || continue d="$(registry_digest "${i}" "${SHA_TAG}")" @@ -236,11 +320,11 @@ fi # Decide the agent-runner base (see header). AGENT_BASE="" if [ -n "${WANT[agent-runner]:-}" ]; then - # shellcheck disable=SC1091 PINNED_BASE="$(. ./infra/agent/versions.env; echo "${AGENT_BASE_IMAGE}")" - if [ "${MODE}" = verify ] && [ -z "${WANT[toolbox]:-}" ]; then - WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)" - elif [ "${MODE}" = publish ] && [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then + if [ "${MODE}" = verify ]; then + AGENT_BASE=link + [ -n "${WANT[toolbox]:-}" ] || { WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)"; } + elif [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then case "${PINNED_BASE}" in *@"${ZERO_DIGEST}") WANT[toolbox]=1; NOTE[toolbox]="never published; required as agent-runner's base" ;; *@sha256:*) AGENT_BASE="${PINNED_BASE}" ;; @@ -249,37 +333,30 @@ if [ -n "${WANT[agent-runner]:-}" ]; then fi fi -yn() { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; } for i in "${ALL_IMAGES[@]}"; do info "plan: ${i}: $(yn "${WANT[$i]:-}" BUILD skip) (${NOTE[$i]:-inputs unchanged})"; done - -if [ -n "${WANT[toolbox]:-}" ]; then - build_one toolbox - if [ "${MODE}" = publish ]; then push_one toolbox; fi +if [ -z "${WANT[toolbox]:-}${WANT[agent-runner]:-}${WANT[conformance-runner]:-}" ]; then + info "nothing to build" +else + setup_builder fi + +if [ -n "${WANT[toolbox]:-}" ]; then build_one toolbox; fi if [ -n "${WANT[agent-runner]:-}" ]; then if [ -z "${AGENT_BASE}" ]; then - if [ "${MODE}" = publish ]; then - AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}" - else - AGENT_BASE="$(local_ref toolbox)" - fi + AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}" fi - info "agent-runner base: ${AGENT_BASE}" + info "agent-runner base: $( [ "${AGENT_BASE}" = link ] && echo "the toolbox built in this run (bake target: context)" || echo "${AGENT_BASE}")" build_one agent-runner "${AGENT_BASE}" - if [ "${MODE}" = publish ]; then push_one agent-runner; fi -fi -if [ -n "${WANT[conformance-runner]:-}" ]; then - build_one conformance-runner - if [ "${MODE}" = publish ]; then push_one conformance-runner; fi fi +if [ -n "${WANT[conformance-runner]:-}" ]; then build_one conformance-runner; fi # --- summary ------------------------------------------------------------------------------ { echo "## Images (${MODE})" echo - echo "commit \`${HEAD_SHA}\`" + echo "commit \`${HEAD_SHA}\` — builder: rootless BuildKit \`${BUILDKIT_HOST}\` (mTLS)" echo - echo "| image | built + verified | seconds | published digest (registry-returned) | note |" + echo "| image | verify target built | seconds | published digest (registry-returned) | note |" echo "|---|---|---|---|---|" for i in "${ALL_IMAGES[@]}"; do d="${DIGEST[$i]:-}" diff --git a/clusters/base/buildkit.yaml b/clusters/base/buildkit.yaml new file mode 100644 index 0000000..dedb3e8 --- /dev/null +++ b/clusters/base/buildkit.yaml @@ -0,0 +1,21 @@ +--- +# Flux Kustomization -> ./platform/buildkit — rootless BuildKit, the image builder CI uses +# (.gitea/workflows/images.yaml). Its own Kustomization so a stuck Certificate or a failing +# builder can never stall any other tier (Flux health-checks Certificates under wait:true). +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: buildkit + namespace: flux-system +spec: + interval: 10m + retryInterval: 2m + timeout: 5m + prune: true + wait: true + dependsOn: + - name: platform # cert-manager CRDs (Issuer/Certificate) come from its HelmRelease + sourceRef: + kind: GitRepository + name: flux-system + path: ./platform/buildkit diff --git a/clusters/base/kustomization.yaml b/clusters/base/kustomization.yaml index 5f816bf..d56e149 100644 --- a/clusters/base/kustomization.yaml +++ b/clusters/base/kustomization.yaml @@ -98,4 +98,5 @@ resources: # deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon. # See clusters/base/kured.yaml and platform/kured/helmrelease.yaml. - kured.yaml # Flux Kustomization -> ./platform/kured (dependsOn platform) + - buildkit.yaml # Flux Kustomization -> ./platform/buildkit (dependsOn platform) - cluster-vars.yaml # overlay knob surface (base defaults) diff --git a/conformance/incluster/Containerfile b/conformance/incluster/Containerfile index 35419dc..e864869 100644 --- a/conformance/incluster/Containerfile +++ b/conformance/incluster/Containerfile @@ -98,7 +98,10 @@ RUN set -eux; \ # --- runtime ----------------------------------------------------------------- ARG FEDORA_VERSION=44 -FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION} +# Stage `conformance-runner` is the image that ships; stage `verify` (end of file) is a TEST stage +# built FROM it that only runs assertions and is never pushed. build-image.sh and the CI +# pipeline build `--target verify` to prove it and `--target conformance-runner` for the image. +FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION} AS conformance-runner ARG CHAINSAW_VERSION= ARG KUBECTL_VERSION= @@ -198,3 +201,86 @@ WORKDIR /workspace # prepend something the policy cannot see. `bash` is the interactive fallback for a human # debugging the image locally. CMD ["/bin/bash"] + +# ============================================================================= +# verify — a TEST stage, never shipped. Every assertion conformance/incluster/build-image.sh used +# to make with `run --user 65532 --read-only --tmpfs /workspace` now runs here, at build time, +# because CI builds on rootless BuildKit where no container runtime exists. +# +# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE: +# * uid 65532 ............ exact (USER inherited from the conformance-runner stage). +# * no network ........... exact: RUN --network=none, and the first check proves it. +# * --read-only rootfs ... NOT reproducible (a RUN step always has a writable layer). +# Approximated by permission: everything uid 65532 can write must lie +# under /workspace, /tmp or /var/tmp. In the Job only /workspace (and +# /results) are writable — the image points HOME and TMPDIR there — and +# readOnlyRootFilesystem is enforced by +# platform/kyverno/admission/conformance-trigger-job-shape.yaml. +# ============================================================================= +FROM conformance-runner AS verify +RUN --network=none /bin/bash -euo pipefail <<'VERIFY' +echo "--- no network (the build-time stand-in for the Job's isolation) ---" +if getent hosts gitea.home.keithjackson.dev >/dev/null 2>&1 || timeout 3 bash -c '/dev/null; then + echo "UNEXPECTED: network reachable in the verify stage"; exit 1 +fi +echo "confirmed: no network" +echo "--- id ---"; id +echo "--- HOME ---"; echo "HOME=${HOME}" +echo "--- bash ---"; bash --version | head -1 +echo "--- python3 ---"; python3 --version +echo "--- git ---"; git --version +echo "--- kubectl ---"; kubectl version --client +echo "--- chainsaw ---"; chainsaw version +echo "--- baked versions ---"; cat /etc/conformance-runner-versions +echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---" +stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone +if [ -w /usr/local/bin/conformance-clone ]; then + echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1 +fi +echo "confirmed: not writable by uid $(id -u)" +# Each of these MUST fail. Assert the non-zero exit as well as the message: a check that only +# greps stderr would pass a build where the guard had stopped rejecting. +expect_fail() { + local label="$1"; shift + local out rc=0 + out="$(env "$@" 2>&1)" || rc=$? + echo "--- ${label} ---" + printf "%s\n" "${out}" | head -2 + if [ "${rc}" -eq 0 ]; then + echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1 + fi + echo "(rejected, exit ${rc})" +} +expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \ + conformance-clone --layer l2 +expect_fail "conformance-clone rejects a missing SHA" \ + conformance-clone +expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \ + CONFORMANCE_GIT_SHA=main conformance-clone +expect_fail "conformance-clone rejects an UNRENDERED placeholder" \ + CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone +expect_fail "conformance-clone rejects a HOME off the workspace volume" \ + CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \ + HOME=/etc conformance-clone +expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \ + CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \ + CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone +expect_fail "conformance-clone refuses a clone target off the workspace volume" \ + CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \ + CONFORMANCE_SRC=/etc/src conformance-clone +echo "--- the root filesystem is not writable by the runner (expected: a failure) ---" +if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi +echo "confirmed: / is not writable" +echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---" +stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \ + | grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)" +if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf ' %s\n' ${stray} | head -20; exit 1; fi +echo "confirmed: no writable path outside /workspace, /tmp, /var/tmp" +echo "--- HOME is writable (the workspace emptyDir) ---" +touch "${HOME}/.probe" || { echo "UNEXPECTED: ${HOME} not writable"; exit 1; } +echo "confirmed: ${HOME} is writable" +echo "--- uid is non-zero ---" +[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; } +echo "confirmed: running as uid $(id -u), not root" +echo "CONFORMANCE-RUNNER VERIFIED" +VERIFY diff --git a/conformance/incluster/README.md b/conformance/incluster/README.md index 3d306dc..215154b 100644 --- a/conformance/incluster/README.md +++ b/conformance/incluster/README.md @@ -61,10 +61,12 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02). > digest-pinned to the conformance-runner image, `command` exactly > `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and > `env`/`envFrom` were the same shape of hole and are closed in the same policy. -- `build-image.sh` — builds the above and **verifies the toolchain by running it** inside the - image as uid 65532 with `--read-only`. It **never pushes**, and neither does anyone from a - workstation: CI (`.gitea/workflows/images.yaml`) runs this same script, and on a push to - `main` publishes the verified image under `sha-` and `main`. Renovate then moves +- `build-image.sh` — builds the above and **verifies the toolchain by running it**: the + Containerfile's `verify` stage executes every tool and every `conformance-clone` refusal as + uid 65532 with no network, at build time. It **never pushes**, and neither does anyone from a + workstation: CI (`.gitea/workflows/images.yaml`) builds the same Containerfile and `verify` + stage on rootless BuildKit (`platform/buildkit/`), and on a push to `main` publishes the + verified image under `sha-` and `main`. Renovate then moves `CONFORMANCE_RUNNER_IMAGE_DIGEST` in `conformance/versions.env` onto the registry-returned digest in a pull request (that file is a sensitive path, so the bump needs an administrator's `/approve-override`). diff --git a/conformance/incluster/build-image.sh b/conformance/incluster/build-image.sh index c836484..c5b1063 100755 --- a/conformance/incluster/build-image.sh +++ b/conformance/incluster/build-image.sh @@ -3,7 +3,8 @@ # conformance/incluster/build-image.sh — build the pinned `conformance-runner` # image and prove it works. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1. # -# ./build-image.sh # build + verify + print the image id +# ./build-image.sh # build the verify target, then the image +# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes) # ./build-image.sh --no-verify # build only (skip the in-image tool proof) # ./build-image.sh --verify-clone # ALSO really clone this repo with conformance-clone. # # Opt-in because it needs the Gitea host reachable; the @@ -13,13 +14,12 @@ # IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this) # # THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either. -# Publishing is done by CI: .gitea/workflows/images.yaml calls this script, and on a push to -# `main` it pushes what this script built AND verified to the Gitea registry, records the -# registry-returned digest in the job summary, and Renovate then opens the pull request that -# moves conformance/versions.env onto that digest. So this script's job is to build and to -# prove the toolchain works INSIDE the image as the non-root uid the policy demands — the same -# proof locally and in CI. It runs under ENGINE=docker in CI (Docker-in-Docker), so nothing -# below may be podman-only. +# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on +# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and +# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest +# in the job summary; Renovate then opens the pull request that moves conformance/versions.env +# onto that digest. The proof that the image works is the Containerfile's `verify` stage, which +# this script (podman locally) and CI both build. # # VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the # single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same @@ -39,9 +39,11 @@ TAG_OVERRIDE="${IMAGE_TAG:-}" VERIFY=1 VERIFY_CLONE=0 +PRINT_ARGS=0 for arg in "$@"; do case "$arg" in --no-verify) VERIFY=0 ;; + --print-build-args) PRINT_ARGS=1 ;; --verify-clone) VERIFY_CLONE=1 ;; -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;; *) echo "unknown argument: $arg" >&2; exit 2 ;; @@ -52,8 +54,6 @@ log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; } info() { printf ' %s\n' "$*"; } die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; } -command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)" - # --- pins: sourced, never invented ------------------------------------------- TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env" [ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here" @@ -63,6 +63,13 @@ source "${TOOLCHAIN_ENV}" : "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}" : "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}" +# THE build args, in ONE place; `--print-build-args` hands them to the CI pipeline, which builds +# this Containerfile on rootless BuildKit. +BUILD_ARGS=("CHAINSAW_VERSION=${CHAINSAW_VERSION}" "KUBECTL_VERSION=${KUBECTL_VERSION}" "FEDORA_VERSION=${FEDORA_VERSION}") +if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi + +command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)" + # --- image name: fixed by the admission policy ------------------------------- # platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY # gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path @@ -84,86 +91,27 @@ info " FEDORA ${FEDORA_VERSION}" LABEL_ARGS=() [ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}") -"${ENGINE}" build "${LABEL_ARGS[@]}" \ - --file "${HERE}/Containerfile" \ - --tag "${IMAGE_REF}" \ - --build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \ - --build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \ - --build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \ - "${HERE}" +ARG_FLAGS=() +for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done # --- verify: the point of the whole exercise --------------------------------- -# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the -# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root -# filesystem — the shape the admission policy forces. A pass here is the only evidence -# this image is usable; a build that merely succeeded is not. +# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So the proof is that +# the tools EXECUTE, as the NON-ROOT uid, with no network, that conformance-clone is not writable +# by that uid and refuses every hostile input, and that nothing outside /workspace is writable. +# That proof is the Containerfile's `verify` stage, built here and in CI alike (CI has no +# container runtime — rootless BuildKit only). See the stage's header for what a build step can +# and cannot reproduce of the Job's shape. A build that merely succeeded is not evidence. if [ "${VERIFY}" -eq 1 ]; then - log "Verifying the toolchain inside the image (non-root, read-only rootfs)" - "${ENGINE}" run --rm \ - --user 65532:65532 \ - --read-only \ - --tmpfs /workspace:rw,mode=1777 \ - --entrypoint "" \ - "${IMAGE_REF}" \ - /bin/bash -euo pipefail -c ' - echo "--- id ---"; id - echo "--- HOME ---"; echo "HOME=${HOME}" - echo "--- bash ---"; bash --version | head -1 - echo "--- python3 ---"; python3 --version - echo "--- git ---"; git --version - echo "--- kubectl ---"; kubectl version --client - echo "--- chainsaw ---"; chainsaw version - echo "--- baked versions ---"; cat /etc/conformance-runner-versions - echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---" - stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone - if [ -w /usr/local/bin/conformance-clone ]; then - echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1 - fi - echo "confirmed: not writable by uid $(id -u)" - # Each of these MUST fail. Assert the non-zero exit as well as the message: a check - # that only greps stderr would pass a build where the guard had stopped rejecting. - # (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the - # expected failure — without it the verify dies at the first negative case.) - expect_fail() { - local label="$1"; shift - local out rc=0 - out="$(env "$@" 2>&1)" || rc=$? - echo "--- ${label} ---" - printf "%s\n" "${out}" | head -2 - if [ "${rc}" -eq 0 ]; then - echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1 - fi - echo "(rejected, exit ${rc})" - } - expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \ - conformance-clone --layer l2 - expect_fail "conformance-clone rejects a missing SHA" \ - conformance-clone - expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \ - CONFORMANCE_GIT_SHA=main conformance-clone - expect_fail "conformance-clone rejects an UNRENDERED placeholder" \ - CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone - expect_fail "conformance-clone rejects a HOME off the workspace volume" \ - CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \ - HOME=/etc conformance-clone - expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \ - CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \ - CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone - expect_fail "conformance-clone refuses a clone target off the workspace volume" \ - CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \ - CONFORMANCE_SRC=/etc/src conformance-clone - echo "--- rootfs is read-only (expected: a failure) ---" - if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi - echo "confirmed: / is read-only" - echo "--- HOME is writable (the workspace emptyDir) ---" - touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable" - echo "--- uid is non-zero ---" - [ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root" - ' + log "Verifying the toolchain (Containerfile target: verify)" + "${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}" log "Toolchain verified" fi +"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target conformance-runner \ + --file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}" + # --- verify the clone helper against the REAL repository (opt-in: needs network) ---- +# LOCAL ONLY: this one `run`s the image, which needs a container engine, so CI does not do it. # The negative cases above prove conformance-clone refuses bad input. This proves the # positive case: it lands the EXACT commit asked for and can prove it did. Without this, the # rev-parse assertion is only an unexercised line of shell. diff --git a/conformance/tests/l0-static/alert-rules-unit-tested/check.sh b/conformance/tests/l0-static/alert-rules-unit-tested/check.sh index ae7c593..47b6e29 100755 --- a/conformance/tests/l0-static/alert-rules-unit-tested/check.sh +++ b/conformance/tests/l0-static/alert-rules-unit-tested/check.sh @@ -250,8 +250,8 @@ check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text, # The build recipe moved from dev.sh into build-image.sh on 2026-10-29, so that `dev.sh build` # and the CI image pipeline (.gitea/workflows/images.yaml) share ONE recipe. Check the recipe. BUILDSH = ROOT / "infra/devcontainer/build-image.sh" -check(BUILDSH.is_file() and "--build-arg PROMTOOL_VERSION=" in BUILDSH.read_text(), - "infra/devcontainer/build-image.sh does not pass --build-arg PROMTOOL_VERSION — the ARG " +check(BUILDSH.is_file() and '"PROMTOOL_VERSION=$PROMTOOL_VERSION"' in BUILDSH.read_text(), + "infra/devcontainer/build-image.sh does not pass PROMTOOL_VERSION in its BUILD_ARGS — the ARG " "would be empty and the build would fetch a bogus URL") INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh" diff --git a/decisions/2026-10-29-image-pipeline.md b/decisions/2026-10-29-image-pipeline.md index 32e5a68..40c3909 100644 --- a/decisions/2026-10-29-image-pipeline.md +++ b/decisions/2026-10-29-image-pipeline.md @@ -1,7 +1,7 @@ --- date: 2026-10-29 branch: feat/image-pipeline -title: "Images are built, verified and published by CI; Renovate carries the digests into git" +title: "Images are built, verified and published by CI on rootless BuildKit; Renovate carries the digests into git" --- ## What changed @@ -11,127 +11,177 @@ the codebase, never to a one-off workstation build/push. Before this branch the package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and both `build-image.sh` scripts said "publishing is an operator action". -- **`.gitea/workflows/images.yaml`** + **`.gitea/workflows/scripts/images.sh`** (the logic). - - Pull request: build and verify only. No push, and the step references no secret at all. - - Push to `main`, or `workflow_dispatch` on `main`: build, verify, then push to - `gitea.home.keithjackson.dev/home-network/{toolbox,agent-runner,conformance-runner}`. - Tags are `sha-<12 hex>` (immutable: the driver refuses to overwrite one) and `main` (moving). - The digest is read back from the registry with a HEAD on the manifest, compared with what the - client printed, and written to the job summary. - - Only images whose inputs changed are built. A re-run on the same commit skips images that are - already published. - - agent-runner builds FROM the toolbox built in the same run (on main, FROM its just-pushed - digest). Otherwise: - - in verify mode the toolbox is built anyway as its base, because a PR holds no credential; - - in publish mode it uses `AGENT_BASE_IMAGE` from `infra/agent/versions.env`; - - if that is still the zero placeholder, the toolbox is built and published first. This is - how the first run on main bootstraps. - - An always-run step prunes the DinD image store. -- **One build recipe per image, reused by CI.** - - New `infra/devcontainer/build-image.sh`. `dev.sh build` now delegates to it. It has an - in-image verification step that the toolbox never had: every tool is on PATH, and - promtool, kubectl, chainsaw, kyverno and sops report their pins. - - `infra/agent/build-image.sh` and `conformance/incluster/build-image.sh` were made - engine-neutral, because CI runs `ENGINE=docker` against DinD: - - `podman image exists` became `image inspect`, with a pull fallback; - - `{{.Digest}}` and `--digestfile` are gone. No local digest is printed, since only the - registry's digest is meaningful; - - `probe-validation.sh` is fed over **stdin** instead of a bind mount. Under DinD a bind - source is resolved on the daemon's filesystem, which would silently mount an empty - directory; - - `probe-validation.sh` itself gained `${BASH_SOURCE[0]:-$0}` so it survives `bash -s` - under `set -u`. - - `AGENT_BASE_IMAGE` and `IMAGE_TAG` can be overridden from the environment, and - `BUILD_REVISION` adds an `org.opencontainers.image.revision` label. - - The "publishing is an operator action" text was replaced throughout. -- **Digest references.** Each now carries a `# renovate-digest: image= tag=main` marker - on the line above it: - - `platform/pm-agent/deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`), which share - one dependency, so they bump in one PR; - - `infra/agent/versions.env` `AGENT_BASE_IMAGE`, which was `home-k3s-dev:local` and is now - the published toolbox by digest (zero placeholder until the first publish); - - `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to the - zero placeholder because Renovate can only replace a digest it can match. +### The builder: rootless BuildKit — DECISION MADE (operator, on PR #81) - The admission policies are **untouched**; the `@sha256:` form is preserved. +**The problem it answers, measured on this PR's first CI run (images run 325, failed in 6 s):** a +CI job on this runner gets **no Docker daemon at all**, for two reasons, both in the deployed +versions' source: +- act_runner v0.2.11 `internal/app/cmd/daemon.go` sets `container.docker_host` to `-` ("never + mount into jobs") whenever `DOCKER_HOST` is `tcp://`, which it is here (the DinD sidecar); +- gitea/act v0.261.2 `pkg/container/docker_run.go` `sanitizeConfig` drops **every** bind, + socket included, when `valid_volumes` is empty, which it is (the runner has no config file). + +Giving jobs the DinD daemon would have made every branch push root on the node, because that +daemon is `privileged: true`. + +**The decision:** no job may reach a privileged daemon, and images are built by a **rootless +BuildKit** daemon over mTLS. The runner (`apps/wave-1/gitea-actions.yaml`, DOCKER_HOST, +valid_volumes) is **untouched**. + +- **`platform/buildkit/`** — a new hand-managed component in its own namespace `buildkit`. The + runner manifest is renderer-generated (`catalog/render.py` EXTRA_UNITS), and the builder has no + business sitting next to a privileged container. Flux reconciles it as the new Kustomization + `buildkit` (`clusters/base/buildkit.yaml`, `dependsOn: platform`). + - **Deployment:** `moby/buildkit:v0.34.1-rootless@sha256:f68752a5…0c4f89`, pinned by index + digest. The securityContext, every line measured (see *What I verified*): + - `privileged: false`; + - uid/gid 1000, `runAsNonRoot`; + - `capabilities: drop ALL, add SETUID, SETGID`; + - `allowPrivilegeEscalation: true`; + - `seccompProfile: Unconfined` and `appArmorProfile: Unconfined`; + - `readOnlyRootFilesystem: true`, with five emptyDirs. + + Other settings: + - `--oci-worker-no-process-sandbox`; + - no ServiceAccount token; + - memory limit 4Gi, which covers build steps too; + - cache emptyDir with `sizeLimit: 6Gi`, and GC capped at 4 GB in `buildkitd.toml` + (ConfigMap). + - **mTLS via cert-manager** (`certificates.yaml`): a self-signed CA used only here, the server + cert `buildkitd-tls`, and the client cert `buildkit-client-ci`. buildkitd verifies every + client against that CA. + - **Service:** ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`. + - **NetworkPolicies:** + - default-deny; + - ingress on 1234 **only from the act-runner pod** (`gitea-actions`, + `app.kubernetes.io/name=act-runner`); + - egress: DNS, world :443/:80 for downloads, and the `ingress` entity (the Gateway) for the + registry push. + - `versions.env`: the CI-side client, `buildx` v0.38.1 with the sha256 from its + `checksums.txt`, plus the address and certificate name CI dials. +- **How DinD job containers appear to Cilium (measured, read-only).** + `iptables -t nat -S POSTROUTING` in the dind container shows + `-A POSTROUTING -s 172.18.0.0/16 ! -o br-… -j MASQUERADE` (and the same for docker0). So a job's + connection leaves with the **act-runner pod's IP, and therefore that pod's Cilium identity**. + That makes "only the act-runner pod" expressible and correct, but it cannot tell one job from + another, or a job from act_runner itself. Per-job authentication is the client certificate. +- **Admission: no Kyverno exception needed.** + - Every ClusterPolicy actually applied (`platform-config/kustomization.yaml`; live: + `conformance-trigger-job-shape`, `label-cdi-worker-pods`, `pm-agent-job-shape`, + `require-auth-mode-label`, `require-default-deny-baseline`, `require-egress-class-label`, + `restrict-host-namespaces`, `secret-provenance`, `validate-default-deny-shape`) matches only + namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`, or only its own + namespace in the job-shape policies' case. `buildkit` carries no such label. + - There are no namespaced Kyverno Policies and no cluster-wide PSA default (both measured). + - The namespace is deliberately left without a PSA label, because PSS `baseline` forbids + `Unconfined` seccomp and AppArmor. + - `validate-default-deny-shape` does apply to the namespace's default-deny NetworkPolicy, and + its shape (both directions) satisfies it. +- **Unprivileged user namespaces on the nodes (measured, read-only, via Ansible `sysctl`).** On + all three Debian 13.7 nodes, kernel `7.1.8+deb13-amd64`: + - `kernel.unprivileged_userns_clone = 1`; + - `user.max_user_namespaces = 127206`; + - no `kernel.apparmor_restrict_unprivileged_userns` (an Ubuntu-only key); + - AppArmor enabled (`Y`). + + So this is a satisfied prerequisite, not an open one. +- **`infra/gitea/buildkit-client-secret.sh`** (operator action): copies `ca.crt`, `tls.crt` and + `tls.key` from the Secret `buildkit/buildkit-client-ci` into the Actions secrets `BUILDKIT_CA`, + `BUILDKIT_CLIENT_CERT` and `BUILDKIT_CLIENT_KEY`. Before writing anything it checks that: + - the cert verifies against the CA; + - the cert is a client-auth cert; + - the key matches the cert. + + It never prints a value. `--check` reports whether the stored copy is older than the current + certificate. + +### Verification moved into the Containerfiles + +`docker run` cannot exist on BuildKit, so each Containerfile now names its shipped stage after the +image (`toolbox`, `agent-runner`, `conformance-runner`) and ends in a **`verify` test stage**, +`FROM AS verify`. That stage runs every previous assertion at build time, as the image's +non-root uid (65532 for agent-runner and conformance-runner), in `RUN --network=none`. It is never +pushed. +- **Every existing assertion is preserved**, and the toolbox gained one it never had: every tool + on PATH, and five tools report their pinned versions. +- `probe-validation.sh` is `COPY`ed into the verify stage only, so it never ships. +- **Bug fixed on the way:** in the old `docker run` block, `cmd | grep -q … && echo ok` and + `[ … ] && echo` could never fail the script under `set -e` + `pipefail`. The "entrypoints refuse + arguments", "not root", "no API key baked in" and "HOME writable" checks asserted nothing. They + are real assertions now. +- **What a build step cannot reproduce, and how it is approximated** (also written in each stage's + header): + - **uid:** exact. + - **`--network none`:** exact, and each stage *proves* it. Its first check fails if DNS or a + TCP connection works. A negative control without `--network=none` fails that check. + - **`--read-only` rootfs:** not reproducible, because a RUN step always has a writable layer. + It is approximated by permission: + - `touch /should-fail` must fail; + - a `find / -xdev -writable ! -type l` inventory, run as the uid, must lie entirely under + `/workspace`, `/tmp` or `/var/tmp`. `! -type l` was needed: Fedora's `/usr/tmp` symlink + to `/var/tmp` tripped it. + + The real control remains the pods' `readOnlyRootFilesystem`, which the admission policies + enforce. + - **`--tmpfs /workspace`:** approximated by the image's own `/workspace`, owned by 65532, which + is exactly what the pod's emptyDir replaces. + + The opt-in `--verify-clone` still `run`s the image, so it remains local-only. +- **`build-image.sh` (all three):** + - verify is `build --target verify` and the image is `build --target `; + - a new `--print-build-args` prints the pins as `KEY=VALUE`, which is what CI consumes. The + arg list therefore exists once; + - the local podman path still works. +- The L0 check `alert-rules-unit-tested` now looks for `"PROMTOOL_VERSION=$PROMTOOL_VERSION"` in + the toolbox's build-arg list. + +### The pipeline (`.gitea/workflows/images.yaml` + `scripts/images.sh`) + +- **The builder.** The driver downloads the pinned `buildx` and refuses a checksum mismatch. It + writes the client cert from the `BUILDKIT_*` secrets to a 0600 temp dir that is removed on exit, + creates a `remote` builder, and runs `buildx inspect --bootstrap`. +- **It fails loudly, never skips.** It stops with a clear message if: + - the secrets are absent; + - buildkitd is unreachable. The message lists the causes in order: not deployed yet (Flux + applies it only after merge), stale certificate, NetworkPolicy. +- **Pull request:** `buildx bake -verify` for changed images. Nothing is pushed and no + registry credential is referenced. The `BUILDKIT_*` builder secrets *are* given to PR jobs, + because they must build. + - agent-runner is built FROM the toolbox built **in the same run** through a bake `target:` + context. That toolbox exists in no registry, so the toolbox is built as its base even when + its own inputs did not change. +- **Push to main / dispatch on main:** for each image, the verify target first, and only then the + image itself with `push=true`. Tags are `sha-<12 hex>` (immutable) and `main` (moving). + - The digest is read back from the registry and compared with the client's metadata. + - agent-runner builds FROM the toolbox digest just pushed, or else from the pinned + `AGENT_BASE_IMAGE` (the zero placeholder bootstraps). + - `BUILDX_NO_DEFAULT_ATTESTATIONS=1` keeps one manifest per tag, which is what admission, + Renovate and the retention script assume. +- **Unchanged from the first iteration:** change detection, publish idempotence, + `REGISTRY_PUSH_TOKEN` and its preflight, the Renovate markers, and `registry-retention.sh`. The + DinD prune step is gone, because nothing is built in DinD now. + +### Unchanged from the first iteration (still true) + +- **Digest references** carry `# renovate-digest: image= tag=main` markers: + - `platform/pm-agent/deployment.yaml` and `configmap.yaml`; + - `infra/agent/versions.env` `AGENT_BASE_IMAGE`, now the published toolbox by digest, zero + until the first publish; + - `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to zero. + + The admission policies are untouched. - **`renovate.json5`**: - - a new `custom.regex` manager for the markers; + - the `custom.regex` marker manager; - `enabledManagers` `regex` → `custom.regex`; - - a packageRule that disables the `kubernetes` manager for these three repos. Otherwise it - would also see the digest-only lines and look them up as `latest`; - - an `own-image` label; - - `ignorePaths` narrowed from `conformance/**` to its subdirectories. `ignorePaths` applies - to custom managers too, so the blanket glob hid `conformance/versions.env`. -- **`renovate.yaml`**: `renovate-version: 44.167.0` pinned, plus a `RENOVATE_HOST_RULES` docker - rule for the Gitea registry, using the same `RENOVATE_TOKEN`, which now also needs - `read:package`. See below for why the version pin is a real fix and not tidiness. -- **`infra/gitea/registry-retention.sh`**: dry run by default, `--apply` to delete. This is an - operator action. Per image it keeps: - - `main` and `latest`; - - the 10 newest `sha-` tags; - - every tag whose digest is pinned anywhere in `origin/main`; - - every digest-named version. -- **`infra/gitea/conformance-trigger-kubeconfig.sh`**: the "runner image is digest-pinned" - precondition grepped the literal line `CONFORMANCE_RUNNER_IMAGE="${…_REPO}@${…_DIGEST}"` - for 64 hex digits, so it **could never pass**. It now sources the file, evaluates the value, - and rejects the zero placeholder. -- **`conformance/tests/l0-static/alert-rules-unit-tested/check.sh`**: its "the toolbox build - passes `--build-arg PROMTOOL_VERSION`" assertion now reads `infra/devcontainer/build-image.sh` - instead of `dev.sh`. The gate caught this on the first run, because the recipe had moved. -- **Docs**: `platform/pm-agent/README.md`, `runbooks/pm-agent.md` (two rows), - `docs/plans/pm-agent-issue-to-pr.md` (follow-up 1), `conformance/incluster/README.md`, - `infra/devcontainer/README.md`, and a new "How images are built" section in - `.gitea/workflows/README.md`. + - the `kubernetes` manager disabled for these three repos; + - `ignorePaths` narrowed so `conformance/versions.env` is visible. -### BLOCKER — the runner cannot build images, and fixing that is a security decision - -**Measured on this PR's own CI run (images run 325): the job failed in 6 s** with -`failed to connect to the docker API at unix:///var/run/docker.sock ... no such file`. The job -container is given **no Docker daemon at all**, for two independent reasons, both read from the -exact versions deployed: - -- act_runner **v0.2.11** `internal/app/cmd/daemon.go`: when `DOCKER_HOST` has a scheme other than - `unix`/`npipe` — here `tcp://localhost:2376`, the DinD sidecar — it sets - `cfg.Container.DockerHost = "-"`, which means "never mount into job containers". -- gitea/act **v0.261.2** (act_runner's `go.mod` replace) `pkg/container/docker_run.go` - `sanitizeConfig`: with `valid_volumes` empty (the runner has no config file, so the default - `[]`), `hostConfig.Binds = []string{}`. **Every** bind is dropped, the daemon socket included, - and so is any `volumes:` a workflow asks for. - -I assumed the opposite while designing (from act's `getDockerDaemonSocketMountPath`, which never -gets the chance to run). That was wrong, and the PR run is what showed it. - -**Why I did not fix it in this PR.** Any fix gives workflow code the DinD daemon. That daemon is -`privileged: true`, so `docker run --privileged -v /:/host` from a job is **root on the node**. -Today a malicious workflow — for example one pushed by a prompt-injected pm-agent worker, which -holds a repo write token — runs in an unprivileged container and holds no cluster credential. -After the fix it does not. That is a change to the security boundary, and it is the operator's -call. It also cannot be tested from a PR anyway: the runner only changes after merge + Flux. - -**Options, for the operator:** - -1. **Socket to every job (smallest change):** in `apps/wave-1/gitea-actions.yaml`, - - mount a shared `emptyDir` at `/var/run` in both the `dind` sidecar and the `runner`, so - dockerd's own `unix:///var/run/docker.sock` is visible at the same path in both; - - set the runner's `DOCKER_HOST=unix:///var/run/docker.sock`, and drop `DOCKER_TLS_VERIFY` - and `DOCKER_CERT_PATH`, because TLS over a unix socket breaks the client; - - add an act_runner config file (`CONFIG_FILE`) with `container.valid_volumes: - ["/var/run/docker.sock"]` and `runner.file: /data/.runner`, so the existing registration - is kept. - - Consequence: any workflow on any branch can reach node root. -2. **Option 1 on a SECOND runner** with its own label (e.g. `image-builder`) and its own DinD, - with `images.yaml` `runs-on` that label. The gate runner stays socket-less, and image builds - stop competing with the gate for the single slot and its 7 GB node disk. **It is not a - security boundary**: Gitea cannot restrict which workflow may target a label, so any branch - could still `runs-on: image-builder`. -3. **A builder that is not a privileged daemon**, e.g. a rootless BuildKit sidecar exposed to - jobs. That bounds a job to BuildKit's rootless sandbox instead of node root. It needs the - verification steps rewritten, because they *run* the image (as uid 65532, `--read-only`), - which BuildKit alone does not do. This is a larger piece of work. - -Nothing in this PR has to change for options 1 or 2 except `runs-on` for option 2. The driver -already produced verified images with a stock docker 27 CLI (see *What I verified*). + `renovate.yaml` pins `renovate-version: 44.167.0` and adds a docker `RENOVATE_HOST_RULES` entry + using `RENOVATE_TOKEN`, which now needs `read:package`. +- **`infra/gitea/registry-retention.sh`** (dry run by default; operator action); the + `conformance-trigger-kubeconfig.sh` precondition fix; and the docs (pm-agent README, runbook, + plan, conformance README, devcontainer README, workflows README). The docs are updated again + for BuildKit, and there is a new `platform/buildkit/README.md`. ### Decisions and pushback @@ -147,7 +197,8 @@ already produced verified images with a stock docker 27 CLI (see *What I verifie - `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3. The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight - pattern. + pattern. With BuildKit the credential reaches the daemon through the buildx session (a + 0600 `config.json` in the job's temp dir), never through the cluster. - **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate: - `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` → `custom.regex`; @@ -195,89 +246,112 @@ already produced verified images with a stock docker 27 CLI (see *What I verifie ## What I verified -- **Local gate, pinned toolbox, from this worktree, committed HEAD.** Command: - `ENGINE= DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`. - The shim adds the `.git` mount. Result: see the PR body for the rc and stage list. They are - recorded there rather than here because this entry is part of the tree being gated. -- **Local podman builds through the CI driver**, run as - `MODE=verify ENGINE=podman BASE_SHA=$(git merge-base origin/main HEAD) .gitea/workflows/scripts/images.sh`. - It returned rc 0 and built and verified all three images: - - toolbox: every tool on PATH, and the five pins matched; - - agent-runner: `PM_SELFTEST=1 pm-dispatch` OK, and the stdin-fed `probe-validation.sh` - gave "20 refusal(s), 0 failure(s)"; - - conformance-runner: all seven `conformance-clone` refusals. -- **The docker-CLI code path**, i.e. what CI runs. I ran the same driver inside - `docker:27-cli` (`ENGINE=docker`, `DOCKER_BUILDKIT=0`) against podman's Docker-compatible API - socket. Exit code 0; all three images built and verified (selftest OK; probe OK, 20 refusals). - This covers the docker-specific calls (`image inspect` formats, `run -i` with stdin, - `--tmpfs`, `--read-only`, `--user`). It does **not** cover BuildKit or a real DinD daemon. -- **This PR's CI, run 325 (`images`): FAILED in 6 s**, because there is no Docker daemon in the - job container (see BLOCKER). Gate run 324 failed only on `approval-policy` (expected, - sensitive paths). Every other stage passed there, including `gitea-settings`. -- **Other driver paths, run locally:** - - agent-runner alone builds the toolbox as its base; - - an empty diff builds nothing; - - publish mode without a token dies before doing anything; - - `infra/agent/build-image.sh` with the zero placeholder dies with build-locally - instructions. -- **Renovate, with the pinned 44.167.0 image, locally:** - - `renovate-config-validator` exits 0; - - a `--platform=local` `extract` dry run finds all four marker references (custom regex manager, - `currentValue: main`, `currentDigest: sha256:000…`); - - a `lookup` dry run shows the `kubernetes`-manager duplicates as `skipReason: disabled` and the - marker deps taking the "unsupported/unversioned value main → getDigest(main)" path. That - lookup currently 404s because nothing is published yet. -- **Renovate 39.264.0 rejects main's current `renovate.json5`.** Validator output reproduced - above. -- **`registry-retention.sh` dry run against the live Gitea (read-only):** zero versions per image, - nothing to delete. The token exchange and the package listing both answered. -- **Live, read-only measurements** (kubectl and the Gitea API): - - the runner pod is on k3s-prod-1, which had **7 GB** nodefs available; - - DinD holds only the 1.54 GB act image; buildx is v0.20.1; the daemon is 27.5.1 on overlay2 - with cgroup v2; - - Gitea `/data` is 9.7G, 1% used; - - the Gateway's Envoy (Cilium) has `streamIdleTimeout: 300s`, `maxStreamDuration: 0s`, and - **no route `timeout`** on the gitea route; - - Gitea's `LIMIT_SIZE_CONTAINER` is unset (default −1 = unlimited, `modules/setting/packages.go`). +**Proved locally, against the exact pinned images:** + +- **Rootless buildkitd, configured as the manifest configures it.** + - Setup: the pinned `moby/buildkit:v0.34.1-rootless` digest, under rootless podman, with + `--cap-drop ALL --cap-add SETUID --cap-add SETGID`, seccomp unconfined and `--read-only`; + tmpfs at the same five paths; the ConfigMap's `buildkitd.toml`; mTLS on :1234. + - The image's `/etc/subuid` was overridden with `user:1:999` and `user:1001:64535`, because + rootless podman maps only 65,537 uids. rootlesskit's real `newuidmap` user-namespace path + still ran. + - It started, `buildctl debug workers` listed the OCI worker, and `debug workers -v` showed the + GC config applied (max 4.29 GB, reserved 1.07 GB). +- **The securityContext, item by item.** Each requirement was measured by taking it away: + - **default seccomp:** daemon starts, every RUN step fails → `Unconfined` is required; + - **no `SETUID`/`SETGID`:** rootlesskit dies with + `newuidmap … operation not permitted` → both capabilities are required; + - **`--read-only` rootfs:** works once the five emptyDirs exist. Its only failure was the + probe client's `~/.docker`, which is now an emptyDir; + - **minimal caps + Unconfined seccomp:** works, and a RUN step executes. +- **The full CI path, in the actual CI job image.** Setup: `ghcr.io/catthehacker/ubuntu:act-22.04` + running `.gitea/workflows/scripts/images.sh` in verify mode. It downloaded and sha256-checked + buildx v0.38.1, connected over mTLS, and bake-built all three `verify` targets from an empty + cache. rc 0, 1m32s total: + - `TOOLBOX VERIFIED` (42 s); + - `AGENT-RUNNER VERIFIED` (26 s): selftest OK, probe "20 refusal(s), 0 failure(s)", and built + FROM the in-run toolbox through the `target:` context, with the log showing + `[agent-runner-verify toolbox 1/6]` and `base=toolbox-in-this-run`; + - `CONFORMANCE-RUNNER VERIFIED` (21 s): all seven refusals; + - every stage printed `confirmed: no network`. +- **mTLS rejects what it should.** Tested with a real BuildKit API call (`buildctl debug workers`): + - no client cert → `remote error: tls: certificate required`; + - a client cert from another CA → rejected the same way; + - the valid cert → accepted. + + Through the driver: + - another CA's client cert → rc 1; + - the wrong CA for the server → rc 1; + - an unreachable host → rc 1 in about 20 s, with the explanatory message; + - missing secrets → rc 1, naming the three secrets. +- **The local podman path**, `build-image.sh` for all three (rc 0). + - Negative control: a verify stage expecting a different promtool version printed + `DRIFT promtool want 9.9.9` and failed the build. + - Negative control: the `--network=none` check fails when the flag is removed. +- **`buildkit-client-secret.sh`'s sanity checks**, run offline on local certs. They tell apart: + - a good client cert; + - a server cert (not client-auth); + - a mismatched key; + - a cert from another CA. + +**Measured live, read-only:** +- node user namespaces, Debian 13.7 and AppArmor (via an Ansible `sysctl`/`cat`); +- DinD's MASQUERADE rules; +- the live ClusterPolicies, namespaced Policies, PSA labels, and cert-manager v1.21.3 with its + ClusterIssuers; +- the act-runner pod labels. + +**Renovate (unchanged since the first iteration):** +- the validator with 44.167.0: exit 0; +- `extract` finds all four markers; +- the `kubernetes` duplicates are `skipReason: disabled`; +- 39.264.0 rejects `main`'s config. + +**Local gate, CI results:** see the PR body. They are recorded there because this entry is part of +the tree being gated. ## What I did NOT verify -- **Nothing has been pushed to the registry, by me or by CI.** The publish path has never run: - - the registry HEAD/digest read-back; +- **platform/buildkit has never run in the cluster.** It can't until it merges: agents may not + `kubectl apply`, and Flux reconciles only `main`. So this PR's own `images` job **cannot build + and is expected to fail at the builder preflight**, which is the designed loud failure. The + following are therefore all unproven in-cluster: + - kubelet admitting the pod; + - containerd honouring `appArmorProfile: Unconfined`; + - the five emptyDirs; + - cert-manager issuing the three certs; + - the CiliumNetworkPolicies. +- **AppArmor `Unconfined` is reasoned, not measured.** My test host runs SELinux, so the AppArmor + requirement comes from upstream's example plus the nodes having AppArmor enabled. If the pod + were to work under `RuntimeDefault` AppArmor, that line could be tightened; I did not test it. +- **The local userns mapping is narrower than the cluster's.** Locally it was 65,535 ids across + two ranges; in the cluster it is the image's own `100000:65536` against real host subuids. That + is the normal case, but it is the cluster's first test. +- **The NetworkPolicies are untested.** In particular: + - that `toEntities: [ingress]` admits the push to the Gateway VIP. The comment explains the + reasoning, but it is unmeasured; + - that world :443/:80 covers every download the builds make. A dnf mirror on another port + would fail a build visibly. +- **No push, anywhere.** The whole publish path is unexercised: + - `REGISTRY_PUSH_TOKEN` reaching buildkitd through the buildx session; + - `push=true`; + - the metadata digest compared with the registry read-back; + - agent-runner FROM `toolbox@sha256:`; - the immutable-tag guard; - - "already published at this commit" idempotence; - - agent-runner `FROM @sha256:`; - - `docker login` with a PAT. + - idempotence. - The first merge to main is the first test, and it needs `REGISTRY_PUSH_TOKEN` to exist first. -- **Large-layer push through the Gateway is UNVERIFIED.** Envoy has no request-body cap and no - stream duration cap here. But with no route `timeout` set, Envoy's **default 15 s upstream - timeout** applies from the end of the request to the end of the response. That covers Gitea - finalising a blob: copying the upload into package storage on Longhorn. A ~250 MB compressed - layer should finish well inside 15 s, but I could not measure it without pushing. If the first - publish shows 504s on blob PUTs, set `timeouts.request` on the gitea HTTPRoute. -- **DinD fit is estimated, not measured** (no build has run there, see the BLOCKER): - - memory: 2Gi is shared by the job container and every build container; - - disk: the toolbox, agent-runner, conformance-runner, Fedora base and build cache should - total roughly 3 GB on top of the 1.5 GB act image, against 7 GB free on the node, with - `dind-storage` having no sizeLimit. -- **No image has been built in this runner's DinD**, because it cannot be done until the BLOCKER - is resolved. Two things are therefore unmeasured until then: - - DinD memory (2Gi) and disk fit; - - BuildKit (`buildx` v0.20.1 is on the daemon) as opposed to the legacy builder I tested - with. -- **Renovate has still never completed a run here.** Three things block it: - - #80, the `configurationFile` name; - - `RENOVATE_TOKEN` gaining `read:package`; - - this branch's version pin. - - `renovatebot/github-action@v41.0.9` running a 44.x image is untested. The action only passes - the tag to `docker run`. -- **Pods cannot pull these images yet regardless.** The kubelet pull credential is PR #61 - (`pm-agent-registry` dockerconfigjson, `read:package`), still open. Until it merges, a correct - digest still ends in ImagePullBackOff (401), as before. -- **`registry-retention.sh --apply` has never run**, and its keep logic has only run against an - empty registry. -- **`git diff` change detection on Gitea `push` events relies on `github.event.before`.** A - force-push or a first push falls back to building everything, which is intended but not - observed. + So are the 15 s Envoy route-timeout risk on blob finalisation, and the Gitea PVC (10Gi, shared + with repositories) absorbing ~2 GB per toolbox publish until retention runs. +- **Node disk under real builds.** The cache emptyDir (6Gi limit, 4 GB GC cap) sits on nodes that + had 7–9 GB free. It is untested under concurrent pressure. +- **Certificate rotation is designed, not exercised.** The server cert renews 30 days early but + buildkitd only reads it at start; the client copy needs the script re-run. +- **`--oci-worker-no-process-sandbox` is an accepted risk, not a mitigated one.** A build step can + ptrace the daemon and read its TLS key and cache. Anyone with the client cert, which means every + PR job, can run build steps. +- **Renovate has still never completed a run.** It is blocked on #80, on `RENOVATE_TOKEN` gaining + `read:package`, and on this branch. The `github-action@v41.0.9` + 44.x image pairing is + untested. +- **Pods cannot pull these images until PR #61** (the kubelet pull secret) merges. +- **`registry-retention.sh --apply` has never run**; its keep logic has only run against an empty + registry. diff --git a/docs/plans/pm-agent-issue-to-pr.md b/docs/plans/pm-agent-issue-to-pr.md index 9f18152..7d994fc 100644 --- a/docs/plans/pm-agent-issue-to-pr.md +++ b/docs/plans/pm-agent-issue-to-pr.md @@ -245,8 +245,8 @@ about runtime behaviour in this document is a design claim, not a measurement. the credential decision now being local Gitea accounts with PATs, that means a `dockerconfigjson` for an account whose token carries `read:package`. Only then does the digest matter. **Superseded 2026-10-29:** the build-and-push is no longer an operator step. - `.gitea/workflows/images.yaml` builds and verifies the image with - `infra/agent/build-image.sh`, publishes it on a push to `main`, and Renovate opens the one + `.gitea/workflows/images.yaml` builds and verifies the image on rootless BuildKit + (`platform/buildkit/`), publishes it on a push to `main`, and Renovate opens the one PR that replaces both placeholder digests (`decisions/2026-10-29-image-pipeline.md`). The same two blockers sit in front of `conformance-runner`, which shares the registry path and has never been pulled. diff --git a/infra/agent/Containerfile b/infra/agent/Containerfile index 14da4d8..4b85b48 100644 --- a/infra/agent/Containerfile +++ b/infra/agent/Containerfile @@ -50,7 +50,10 @@ # default is deliberately EMPTY so a build that forgets it fails loudly instead of # resolving some other image named `agent-base`. ARG AGENT_BASE_IMAGE= -FROM ${AGENT_BASE_IMAGE} +# Stage `agent-runner` is the image that ships; stage `verify` (end of file) is a TEST stage built +# FROM it that only runs assertions and is never pushed. build-image.sh and the CI pipeline +# build `--target verify` to prove the image and `--target agent-runner` for the image itself. +FROM ${AGENT_BASE_IMAGE} AS agent-runner # Re-declared AFTER the FROM: an ARG from before the first FROM is not in scope in the # build stage, so the provenance layer below would otherwise record an empty base. @@ -208,3 +211,80 @@ WORKDIR /workspace # paths above; an ENTRYPOINT here would either be ignored or prepend something the policy # cannot see. `bash` is the interactive fallback for a human debugging the image locally. CMD ["/bin/bash"] + +# ============================================================================= +# verify — a TEST stage, never shipped. Every assertion infra/agent/build-image.sh used to make +# with `run --user 65532 --read-only --tmpfs /workspace --tmpfs /tmp --network none` now runs +# here, at build time, because CI builds on rootless BuildKit where no container runtime exists. +# +# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE, stated so nobody over-reads a green verify: +# * uid 65532 ............ exact: USER is inherited from the agent-runner stage. +# * --network none ....... exact: RUN --network=none, and the first check proves it. +# * --read-only rootfs ... NOT reproducible: a RUN step always has a writable layer. It is +# APPROXIMATED by permission: as uid 65532, every path the process can +# write is inventoried and must lie under /workspace, /tmp or /var/tmp +# (the pod mounts emptyDirs on the first two; /var/tmp is read-only +# there, and TMPDIR points into /workspace so nothing should use it). +# The pod's readOnlyRootFilesystem is enforced by admission +# (platform/kyverno/admission/pm-agent-job-shape.yaml), not by this. +# * --tmpfs /workspace ... approximated by the image's own /workspace (owned 65532), which is +# what the pod's emptyDir replaces. +# The probe script is COPYed into THIS stage only, so it never ships in the image. +# ============================================================================= +FROM agent-runner AS verify +COPY probe-validation.sh /opt/verify/probe-validation.sh +RUN --network=none /bin/bash -euo pipefail <<'VERIFY' +echo "--- no network (the build-time stand-in for run --network none) ---" +if getent hosts registry.npmjs.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then + echo "UNEXPECTED: network reachable in the verify stage"; exit 1 +fi +echo " confirmed: no network" +echo "--- id ---"; id +echo "--- HOME ---"; echo "HOME=${HOME} CLAUDE_CONFIG_DIR=${CLAUDE_CONFIG_DIR}" +echo "--- baked versions ---"; cat /etc/agent-runner-versions +echo "--- the gate toolchain the worker needs (CLAUDE.md §3) ---" +for t in git python3 bash yamllint check-jsonschema kyverno promtool chainsaw flux sops age kubectl jq diff; do + if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t"; + else printf " MISSING %s\n" "$t"; missing=1; fi +done +[ -z "${missing:-}" ] || { echo "the gate cannot run in this image"; exit 1; } +echo "--- python can import the renderer dependency ---" +python3 -c "import yaml; print('PyYAML', yaml.__version__)" +echo "--- claude ---"; claude --version +echo "--- the two entrypoints refuse arguments ---" +# Each must REFUSE. (The old docker-run version wrote `cmd | grep -q ... && echo`, which under +# `set -e` + pipefail could never fail the script; these are real assertions.) +for e in pm-worker pm-dispatch; do + out="$(/usr/local/bin/$e --anything 2>&1 || true)" + grep -q "takes no arguments" <<<"$out" || { echo "UNEXPECTED: $e accepted an argument: $out"; exit 1; } + echo " $e: rejected" +done +echo "--- the dispatcher selftest (branch vocabulary, result envelopes, Job template, persona) ---" +PM_SELFTEST=1 /usr/local/bin/pm-dispatch +echo "--- the entrypoints are root-owned and NOT writable by the runtime uid ---" +for f in /usr/local/bin/pm-worker /usr/local/bin/pm-dispatch /etc/pm-agent/worker-job-template.yaml; do + stat -c " %n %U:%G %a" "$f" + if [ -w "$f" ]; then echo "UNEXPECTED: uid $(id -u) can write $f"; exit 1; fi +done +echo " confirmed: not writable by uid $(id -u)" +echo "--- the root filesystem is not writable by the runtime uid (expected: a failure) ---" +if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi +echo " confirmed: / is not writable" +echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---" +stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \ + | grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)" +if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf ' %s\n' ${stray} | head -20; exit 1; fi +echo " confirmed: no writable path outside /workspace, /tmp, /var/tmp" +echo "--- the two writable paths really are writable ---" +touch "${HOME}/.probe" /tmp/.probe || { echo "UNEXPECTED: ${HOME} or /tmp not writable"; exit 1; } +echo " confirmed: ${HOME} and /tmp are writable" +echo "--- uid is non-zero ---" +[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; } +echo " confirmed: uid $(id -u), not root" +echo "--- no Anthropic API key is baked into the image (it would outrank the subscription token) ---" +[ -z "${ANTHROPIC_API_KEY:-}${ANTHROPIC_AUTH_TOKEN:-}" ] || { echo "UNEXPECTED: an Anthropic key is baked in"; exit 1; } +echo " confirmed: neither is set" +echo "--- pm-worker's input validation, with the interpreter and script that run in the cluster ---" +PM_WORKER_PATH=/usr/local/bin/pm-worker /bin/bash /opt/verify/probe-validation.sh +echo "AGENT-RUNNER VERIFIED" +VERIFY diff --git a/infra/agent/build-image.sh b/infra/agent/build-image.sh index e854f4e..cba017b 100755 --- a/infra/agent/build-image.sh +++ b/infra/agent/build-image.sh @@ -2,7 +2,8 @@ ############################################################################### # infra/agent/build-image.sh — build the pinned `agent-runner` image and prove it works. # -# ./build-image.sh # build + verify + print the image id +# ./build-image.sh # build the verify target, then the image +# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes) # ./build-image.sh --no-verify # build only (skip the in-image proof) # ENGINE=docker ./build-image.sh # build with docker instead of podman # AGENT_BASE_IMAGE=home-k3s-dev:local ./build-image.sh @@ -11,12 +12,12 @@ # IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this) # # THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either. -# Publishing is done by CI: .gitea/workflows/images.yaml calls this script, and on a push to -# `main` it pushes what this script built AND verified to the Gitea registry, records the -# registry-returned digest in the job summary, and Renovate then opens the pull request that -# moves platform/pm-agent/ onto that digest. So this script's job is to build, and to prove -# the toolchain and both entrypoints work INSIDE the image as the non-root uid with the -# read-only root filesystem the admission policy forces — the same proof locally and in CI. +# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on +# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and +# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest +# in the job summary. Renovate then opens the pull request that moves platform/pm-agent/ onto +# that digest. This script is the LOCAL path (podman by default); its proof — the +# Containerfile's `verify` stage — is the same one CI runs. # # VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the pins this image adds # (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds the @@ -26,11 +27,10 @@ # kept current by Renovate. An AGENT_BASE_IMAGE in the environment overrides it — CI passes # the toolbox it built in the same run, and a developer can pass `home-k3s-dev:local`. # -# ENGINE PORTABILITY. CI runs this with ENGINE=docker against a Docker-in-Docker daemon, so -# nothing below may be podman-only (`image exists`, `--digestfile`, `{{.Digest}}`), and -# nothing may BIND-MOUNT a file from this checkout into a container: under DinD the daemon -# resolves bind sources on ITS filesystem, where this checkout does not exist, and silently -# mounts an empty directory instead. Files go in over stdin. +# NO `run` STEP. Verification used to `run` the image as uid 65532 with a read-only rootfs. +# CI has no container runtime (rootless BuildKit only), so the assertions moved into the +# Containerfile's `verify` stage and this script builds that target. `--print-build-args` +# prints the pins as KEY=VALUE for the CI pipeline; AGENT_BASE_IMAGE is the caller's choice. ############################################################################### set -euo pipefail @@ -39,9 +39,11 @@ REPO_ROOT="$(cd "${HERE}/../.." && pwd)" ENGINE="${ENGINE:-podman}" VERIFY=1 +PRINT_ARGS=0 for arg in "$@"; do case "$arg" in --no-verify) VERIFY=0 ;; + --print-build-args) PRINT_ARGS=1 ;; -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;; *) echo "unknown argument: $arg" >&2; exit 2 ;; esac @@ -51,8 +53,6 @@ log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; } info() { printf ' %s\n' "$*"; } die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; } -command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)" - # --- pins: sourced, never invented ------------------------------------------- AGENT_ENV="${HERE}/versions.env" [ -f "${AGENT_ENV}" ] || die "missing ${AGENT_ENV} — the pins live there, not here" @@ -68,6 +68,17 @@ source "${AGENT_ENV}" : "${AGENT_MODEL:?AGENT_MODEL not set in ${AGENT_ENV}}" : "${AGENT_IMAGE_REPO:?AGENT_IMAGE_REPO not set in ${AGENT_ENV}}" +# THE build args, in ONE place; `--print-build-args` hands them to CI (which builds on rootless +# BuildKit and supplies its own base reference). AGENT_BASE_IMAGE is deliberately NOT in this +# list: the caller decides the base. +BUILD_ARGS=( + "NODEJS_MAJOR=${NODEJS_MAJOR}" "CLAUDE_CODE_VERSION=${CLAUDE_CODE_VERSION}" + "CLAUDE_CODE_MIN_VERSION=${CLAUDE_CODE_MIN_VERSION}" "AGENT_MODEL=${AGENT_MODEL}" +) +if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi + +command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)" + # --- the base must exist, locally or in the registry --------------------------- # Deliberately NOT built on demand. The toolbox is its own reviewed artefact with its own # pins, and silently building it here would mean this script could change the toolchain a @@ -103,89 +114,26 @@ info " model ${AGENT_MODEL}" # Provenance: CI passes the commit it is building, so a pulled image names its source tree. LABEL_ARGS=() [ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}") - -"${ENGINE}" build "${LABEL_ARGS[@]}" \ - --file "${HERE}/Containerfile" \ - --tag "${IMAGE_REF}" \ - --build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}" \ - --build-arg "NODEJS_MAJOR=${NODEJS_MAJOR}" \ - --build-arg "CLAUDE_CODE_VERSION=${CLAUDE_CODE_VERSION}" \ - --build-arg "CLAUDE_CODE_MIN_VERSION=${CLAUDE_CODE_MIN_VERSION}" \ - --build-arg "AGENT_MODEL=${AGENT_MODEL}" \ - "${HERE}" +ARG_FLAGS=(--build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}") +for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done # --- verify: the point of the whole exercise --------------------------------- -# A build that merely succeeded is not evidence this image is usable. Prove the tools -# EXECUTE, as the NON-ROOT uid, with a READ-ONLY root filesystem and only the two volumes -# the admission policy admits — the exact shape a worker Job runs in. Anything that works -# here but not there is a 3am discovery. +# A build that merely succeeded is not evidence this image is usable. The proof is the +# Containerfile's `verify` stage: as the NON-ROOT uid, with no network, the toolchain executes, +# both entrypoints refuse arguments, the dispatcher selftest passes, the entrypoints are not +# writable, nothing outside /workspace and /tmp is writable, and pm-worker's input guards +# (probe-validation.sh) refuse every hostile input. It is a build target rather than a +# `run` so the SAME proof runs here and in CI, where no container runtime exists — see the +# stage's header for what a build step can and cannot reproduce of the pod's shape. if [ "${VERIFY}" -eq 1 ]; then - log "Verifying inside the image (uid 65532, read-only rootfs, workspace+tmp only)" - "${ENGINE}" run --rm \ - --user 65532:65532 \ - --read-only \ - --tmpfs /workspace:rw,mode=1777 \ - --tmpfs /tmp:rw,mode=1777 \ - --network none \ - --entrypoint "" \ - "${IMAGE_REF}" \ - /bin/bash -euo pipefail -c ' - echo "--- id ---"; id - echo "--- HOME ---"; echo "HOME=${HOME} CLAUDE_CONFIG_DIR=${CLAUDE_CONFIG_DIR}" - echo "--- baked versions ---"; cat /etc/agent-runner-versions - echo "--- the gate toolchain the worker needs (CLAUDE.md §3) ---" - for t in git python3 bash yamllint check-jsonschema kyverno promtool chainsaw flux sops age kubectl jq diff; do - if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t"; - else printf " MISSING %s\n" "$t"; missing=1; fi - done - [ -z "${missing:-}" ] || { echo "the gate cannot run in this image"; exit 1; } - echo "--- python can import the renderer dependency ---" - python3 -c "import yaml; print(\"PyYAML\", yaml.__version__)" - echo "--- claude ---"; claude --version - echo "--- the two entrypoints refuse arguments ---" - /usr/local/bin/pm-worker --anything 2>&1 | grep -q "takes no arguments" && echo " pm-worker: rejected" - /usr/local/bin/pm-dispatch --anything 2>&1 | grep -q "takes no arguments" && echo " pm-dispatch: rejected" - echo "--- the dispatcher selftest (branch vocabulary, result envelopes, Job template, persona) ---" - PM_SELFTEST=1 /usr/local/bin/pm-dispatch - echo "--- the entrypoints are root-owned and NOT writable by the runtime uid ---" - for f in /usr/local/bin/pm-worker /usr/local/bin/pm-dispatch /etc/pm-agent/worker-job-template.yaml; do - stat -c " %n %U:%G %a" "$f" - if [ -w "$f" ]; then echo "UNEXPECTED: uid $(id -u) can write $f"; exit 1; fi - done - echo " confirmed: not writable by uid $(id -u)" - echo "--- rootfs is read-only (expected: a failure) ---" - if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi - echo " confirmed: / is read-only" - echo "--- the two writable paths really are writable ---" - touch "${HOME}/.probe" /tmp/.probe && echo " confirmed: ${HOME} and /tmp are writable" - echo "--- uid is non-zero ---" - [ "$(id -u)" -ne 0 ] && echo " confirmed: uid $(id -u), not root" - echo "--- no Anthropic API key is baked into the image (it would outrank the subscription token) ---" - [ -z "${ANTHROPIC_API_KEY:-}${ANTHROPIC_AUTH_TOKEN:-}" ] && echo " confirmed: neither is set" - ' - # The worker's input guards, exercised in the image, with the interpreter and the script - # that will actually run in the cluster. The probe is mounted read-only rather than baked - # in: it is a build-time check, not something a worker Job has any business carrying. - log "Probing pm-worker's input validation inside the image" - # - # The probe goes in over STDIN, not as a bind mount: under Docker-in-Docker (CI) a bind - # source is resolved on the daemon's filesystem, where this file does not exist, and the - # container would get an empty directory. PM_WORKER_PATH points it at the baked worker, - # since the probe's own default (a sibling of the probe file) has no meaning on stdin. - "${ENGINE}" run --rm -i \ - --user 65532:65532 \ - --read-only \ - --tmpfs /workspace:rw,mode=1777 \ - --tmpfs /tmp:rw,mode=1777 \ - --network none \ - -e PM_WORKER_PATH=/usr/local/bin/pm-worker \ - --entrypoint "" \ - "${IMAGE_REF}" \ - /bin/bash -s < "${HERE}/probe-validation.sh" - + log "Verifying (Containerfile target: verify)" + "${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}" log "Image verified" fi +"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target agent-runner \ + --file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}" + # --- result ------------------------------------------------------------------ # No digest is printed: a manifest digest does not exist until a registry has the manifest, # and the only digest anything may pin is the one the REGISTRY returns. CI captures that at diff --git a/infra/agent/probe-validation.sh b/infra/agent/probe-validation.sh index d888700..57f44ec 100755 --- a/infra/agent/probe-validation.sh +++ b/infra/agent/probe-validation.sh @@ -16,14 +16,14 @@ # scrubbed environment and fake credentials, and if a guard ever stopped firing, the case # would fail here rather than by really cloning something. # -# Run offline, no cluster, no Gitea, no model. infra/agent/build-image.sh runs it inside the -# built image as well, where the interpreter and the script are the ones that will actually -# run in the cluster. +# Run offline, no cluster, no Gitea, no model. The Containerfile's `verify` stage runs it inside +# the built image as well (as uid 65532, no network), where the interpreter and the script are +# the ones that will actually run in the cluster — locally via build-image.sh and in CI. ############################################################################### set -uo pipefail -# `:-$0` because build-image.sh feeds this script over STDIN (`bash -s`), where BASH_SOURCE is -# empty and `set -u` would abort on it; PM_WORKER_PATH is always set in that case. +# `:-$0` keeps this safe if it is ever fed over stdin (`bash -s`), where BASH_SOURCE is empty +# and `set -u` would abort on it. HERE="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)" WORKER="${PM_WORKER_PATH:-${HERE}/pm-worker}" [ -r "$WORKER" ] || { echo "cannot read ${WORKER}" >&2; exit 2; } diff --git a/infra/devcontainer/Containerfile b/infra/devcontainer/Containerfile index a46720a..469d199 100644 --- a/infra/devcontainer/Containerfile +++ b/infra/devcontainer/Containerfile @@ -8,7 +8,10 @@ # packages are retained as harmless disk-image utilities. Fedora base is pinned for # reproducibility. ARG FEDORA_VERSION=44 -FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} +# Stage `toolbox` is the image that ships. Stage `verify` (end of file) is a TEST stage: it is +# built FROM the finished image and only runs assertions, and it is never pushed. Build with +# `--target toolbox` for the image and `--target verify` to prove it (build-image.sh does both). +FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} AS toolbox ARG OPENTOFU_VERSION ARG AGE_VERSION @@ -95,3 +98,48 @@ RUN { echo "opentofu=${OPENTOFU_VERSION}"; echo "age=${AGE_VERSION}"; echo "sops WORKDIR /work CMD ["/bin/bash"] + +# ============================================================================= +# verify — a TEST stage, never shipped. It runs every assertion against the finished image AT +# BUILD TIME, so CI needs no container runtime to prove the image: the image pipeline builds +# this target on rootless BuildKit (platform/buildkit/), where `docker run` does not exist. +# `RUN --network=none` reproduces the old `run --network none` exactly, and the first check +# below proves it (it fails if any network is reachable). +# ============================================================================= +FROM toolbox AS verify +ARG PROMTOOL_VERSION +ARG KUBECTL_VERSION +ARG CHAINSAW_VERSION +ARG KYVERNO_VERSION +ARG SOPS_VERSION +RUN --network=none /bin/bash -euo pipefail <<'VERIFY' +echo "--- no network (the build-time stand-in for run --network none) ---" +if getent hosts registry.fedoraproject.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then + echo "UNEXPECTED: network reachable in the verify stage"; exit 1 +fi +echo " confirmed: no network" +echo "--- baked versions ---"; cat /etc/dev-toolchain-versions +echo "--- every tool is on PATH ---" +for t in tofu age age-keygen sops flux kubectl helm kyverno kyverno-json chainsaw yq \ + yamllint check-jsonschema promtool ansible-playbook git make jq diff python3 \ + wg qrencode; do + if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t"; + else printf " MISSING %s\n" "$t"; missing=1; fi +done +[ -z "${missing:-}" ] || { echo "toolbox is missing tools"; exit 1; } +echo "--- version-sensitive tools report their pins ---" +: "${PROMTOOL_VERSION:?}" "${KUBECTL_VERSION:?}" "${CHAINSAW_VERSION:?}" "${KYVERNO_VERSION:?}" "${SOPS_VERSION:?}" +check() { # name pin actual-output + if printf "%s" "$3" | grep -qF "$2"; then printf " ok %-10s %s\n" "$1" "$2"; + else printf " DRIFT %-10s want %s, got: %s\n" "$1" "$2" "$(printf "%s" "$3" | head -1)"; drift=1; fi +} +check promtool "${PROMTOOL_VERSION}" "$(promtool --version 2>&1)" +check kubectl "${KUBECTL_VERSION}" "$(kubectl version --client 2>&1)" +check chainsaw "${CHAINSAW_VERSION}" "$(chainsaw version 2>&1)" +check kyverno "${KYVERNO_VERSION}" "$(kyverno version 2>&1)" +check sops "${SOPS_VERSION}" "$(sops --version --disable-version-check 2>&1 || sops --version 2>&1)" +[ -z "${drift:-}" ] || { echo "a tool does not report its pinned version"; exit 1; } +echo "--- python can import what the renderer and the gate need ---" +python3 -c "import yaml; print(' PyYAML', yaml.__version__)" +echo "TOOLBOX VERIFIED" +VERIFY diff --git a/infra/devcontainer/README.md b/infra/devcontainer/README.md index 3cfbaa3..6ff9d40 100644 --- a/infra/devcontainer/README.md +++ b/infra/devcontainer/README.md @@ -36,9 +36,11 @@ cd infra/devcontainer ### The published toolbox -`./dev.sh build` delegates to [`build-image.sh`](build-image.sh), which is the **one** build -recipe: CI runs the same script (with its in-image verification — every tool executes, and the -version-sensitive ones report their `versions.env` pins) and, on a push to `main` that changes +`./dev.sh build` delegates to [`build-image.sh`](build-image.sh). The recipe is the +`Containerfile` plus that script's build-arg list, and the proof is the Containerfile's `verify` +stage (every tool on `PATH`, the version-sensitive ones report their `versions.env` pins, no +network). CI builds the same Containerfile on rootless BuildKit with the args from +`build-image.sh --print-build-args`, and, on a push to `main` that changes `Containerfile`, `versions.env` or `build-image.sh`, **publishes** the result as `gitea.home.keithjackson.dev/home-network/toolbox` (tags `sha-` and `main`) — see [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). `agent-runner` is diff --git a/infra/devcontainer/build-image.sh b/infra/devcontainer/build-image.sh index 9467a8e..07c9cc4 100755 --- a/infra/devcontainer/build-image.sh +++ b/infra/devcontainer/build-image.sh @@ -6,12 +6,14 @@ # ./build-image.sh --no-verify # build only # IMAGE=foo:bar ./build-image.sh # build under another local name # ENGINE=docker ./build-image.sh # build with docker instead of podman +# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes) # -# ONE BUILD DEFINITION, TWO CALLERS. `dev.sh build` calls this for local development, and the -# CI image pipeline (.gitea/workflows/images.yaml -> .gitea/workflows/scripts/images.sh) calls -# it to build the image it then PUBLISHES as gitea.home.keithjackson.dev/home-network/toolbox. -# Keeping the build args in exactly one place is what makes "the toolbox CI publishes" and "the -# toolbox you gate with locally" the same recipe. +# ONE BUILD DEFINITION, TWO CALLERS. `dev.sh build` calls this for local development. The CI +# image pipeline (.gitea/workflows/images.yaml -> .gitea/workflows/scripts/images.sh) builds the +# SAME Containerfile on rootless BuildKit (platform/buildkit/) — no container engine exists in a +# CI job — taking its build args from `--print-build-args`, and PUBLISHES the result as +# gitea.home.keithjackson.dev/home-network/toolbox. The Containerfile (including its `verify` +# test stage) and the arg list here are the one recipe both use. # # THIS SCRIPT NEVER PUSHES. Publishing is CI's job (images.yaml, on push to main), not a # workstation's: an image that exists only because someone ran `podman push` from a laptop has @@ -26,9 +28,11 @@ ENGINE="${ENGINE:-podman}" IMAGE="${IMAGE:-home-k3s-dev:local}" VERIFY=1 +PRINT_ARGS=0 for arg in "$@"; do case "$arg" in --no-verify) VERIFY=0 ;; + --print-build-args) PRINT_ARGS=1 ;; -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;; *) echo "unknown argument: $arg" >&2; exit 2 ;; esac @@ -37,72 +41,44 @@ done log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; } die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; } -command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)" - # shellcheck disable=SC1091 source "${HERE}/versions.env" +# THE build args, in ONE place. `--print-build-args` hands exactly this list to the CI pipeline +# (.gitea/workflows/scripts/images.sh), which builds on rootless BuildKit rather than calling +# "$ENGINE build" — so CI and a local build cannot drift apart on a pin. +BUILD_ARGS=( + "FEDORA_VERSION=$FEDORA_VERSION" "OPENTOFU_VERSION=$OPENTOFU_VERSION" "AGE_VERSION=$AGE_VERSION" + "SOPS_VERSION=$SOPS_VERSION" "FLUX_VERSION=$FLUX_VERSION" "KUBECTL_VERSION=$KUBECTL_VERSION" + "HELM_VERSION=$HELM_VERSION" "KYVERNO_VERSION=$KYVERNO_VERSION" + "KYVERNO_JSON_VERSION=$KYVERNO_JSON_VERSION" "CHAINSAW_VERSION=$CHAINSAW_VERSION" + "YAMLLINT_VERSION=$YAMLLINT_VERSION" "YQ_VERSION=$YQ_VERSION" + "CHECK_JSONSCHEMA_VERSION=$CHECK_JSONSCHEMA_VERSION" "PROMTOOL_VERSION=$PROMTOOL_VERSION" +) +if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi + +command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)" + +ARG_FLAGS=() +for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done # Provenance: CI passes the commit it is building, so a pulled image names its source tree. LABEL_ARGS=() [ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}") -log "Building ${IMAGE}" -"$ENGINE" build "${LABEL_ARGS[@]}" \ - --build-arg FEDORA_VERSION="$FEDORA_VERSION" \ - --build-arg OPENTOFU_VERSION="$OPENTOFU_VERSION" \ - --build-arg AGE_VERSION="$AGE_VERSION" \ - --build-arg SOPS_VERSION="$SOPS_VERSION" \ - --build-arg FLUX_VERSION="$FLUX_VERSION" \ - --build-arg KUBECTL_VERSION="$KUBECTL_VERSION" \ - --build-arg HELM_VERSION="$HELM_VERSION" \ - --build-arg KYVERNO_VERSION="$KYVERNO_VERSION" \ - --build-arg KYVERNO_JSON_VERSION="$KYVERNO_JSON_VERSION" \ - --build-arg CHAINSAW_VERSION="$CHAINSAW_VERSION" \ - --build-arg YAMLLINT_VERSION="$YAMLLINT_VERSION" \ - --build-arg YQ_VERSION="$YQ_VERSION" \ - --build-arg CHECK_JSONSCHEMA_VERSION="$CHECK_JSONSCHEMA_VERSION" \ - --build-arg PROMTOOL_VERSION="$PROMTOOL_VERSION" \ - -t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}" - -# --- verify ------------------------------------------------------------------ -# A build that succeeded is not evidence the toolbox can run the gate. Every tool the gate and -# the runbooks call must EXECUTE, and the version-sensitive ones must report the version -# versions.env pins — promtool above all, because ci/versions.env asserts a promtool/Prometheus -# lockstep and a drifted engine evaluates alert-rule tests differently (see versions.env). +# --- verify -------------------------------------------------------------------- +# The assertions live in the Containerfile's `verify` stage (every tool on PATH, the +# version-sensitive ones report their versions.env pins — promtool above all, because +# ci/versions.env asserts a promtool/Prometheus lockstep — and no network), so the SAME proof +# runs here under podman and in CI under rootless BuildKit. A failing assertion fails the build. if [ "${VERIFY}" -eq 1 ]; then - log "Verifying the toolchain inside ${IMAGE}" - "$ENGINE" run --rm --network none --entrypoint "" \ - -e PROMTOOL_VERSION="${PROMTOOL_VERSION}" \ - -e KUBECTL_VERSION="${KUBECTL_VERSION}" \ - -e CHAINSAW_VERSION="${CHAINSAW_VERSION}" \ - -e KYVERNO_VERSION="${KYVERNO_VERSION}" \ - -e SOPS_VERSION="${SOPS_VERSION}" \ - "$IMAGE" /bin/bash -euo pipefail -c ' - echo "--- baked versions ---"; cat /etc/dev-toolchain-versions - echo "--- every tool is on PATH ---" - for t in tofu age age-keygen sops flux kubectl helm kyverno kyverno-json chainsaw yq \ - yamllint check-jsonschema promtool ansible-playbook git make jq diff python3 \ - wg qrencode; do - if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t"; - else printf " MISSING %s\n" "$t"; missing=1; fi - done - [ -z "${missing:-}" ] || { echo "toolbox is missing tools"; exit 1; } - echo "--- version-sensitive tools report their pins ---" - check() { # name pin actual-output - if printf "%s" "$3" | grep -qF "$2"; then printf " ok %-10s %s\n" "$1" "$2"; - else printf " DRIFT %-10s want %s, got: %s\n" "$1" "$2" "$(printf "%s" "$3" | head -1)"; drift=1; fi - } - check promtool "${PROMTOOL_VERSION}" "$(promtool --version 2>&1)" - check kubectl "${KUBECTL_VERSION}" "$(kubectl version --client 2>&1)" - check chainsaw "${CHAINSAW_VERSION}" "$(chainsaw version 2>&1)" - check kyverno "${KYVERNO_VERSION}" "$(kyverno version 2>&1)" - check sops "${SOPS_VERSION}" "$(sops --version --disable-version-check 2>&1 || sops --version 2>&1)" - [ -z "${drift:-}" ] || { echo "a tool does not report its pinned version"; exit 1; } - echo "--- python can import what the renderer and the gate need ---" - python3 -c "import yaml; print(\" PyYAML\", yaml.__version__)" - ' + log "Verifying (Containerfile target: verify)" + "$ENGINE" build "${ARG_FLAGS[@]}" --target verify -f "${HERE}/Containerfile" "${HERE}" log "Toolbox verified" fi +log "Building ${IMAGE}" +"$ENGINE" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target toolbox \ + -t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}" + log "Built ${IMAGE}" printf ' image id %s\n' "$("$ENGINE" image inspect --format '{{.Id}}' "$IMAGE")" diff --git a/infra/gitea/buildkit-client-secret.sh b/infra/gitea/buildkit-client-secret.sh new file mode 100755 index 0000000..2b9f8fb --- /dev/null +++ b/infra/gitea/buildkit-client-secret.sh @@ -0,0 +1,104 @@ +#!/usr/bin/env bash +# Copy the image builder's CLIENT certificate from the cluster into the three repository Actions +# secrets the image pipeline reads: BUILDKIT_CA, BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY. +# +# ./infra/gitea/buildkit-client-secret.sh # copy and verify +# ./infra/gitea/buildkit-client-secret.sh --check # compare only; changes nothing +# +# RUNNING IT IS AN OPERATOR ACTION. It writes repository secrets. +# +# WHERE THE CERTIFICATE COMES FROM. platform/buildkit/certificates.yaml has cert-manager issue +# `buildkit-client-ci` from a CA that exists only for the builder; buildkitd admits a connection +# only with a certificate signed by that CA. CI jobs hold NO cluster credential (R-SEC-02), so they +# cannot read the Secret themselves: this script is the one bridge, run by a human who has both +# a kube context and the Gitea credential. +# +# WHEN TO RUN IT +# 1. once, after Flux has applied platform/buildkit (Kustomization `buildkit` Ready, and +# `kubectl -n buildkit get certificate buildkit-client-ci` READY=True); +# 2. after every renewal of that certificate (duration 1y, renewed 30 days early). Until then +# the image pipeline fails LOUDLY at its builder preflight — it never skips. +# `--check` tells you whether the copy is stale. +# +# NOTHING SECRET IS PRINTED. The key goes from `kubectl get secret` into the Gitea secrets API +# through shell variables and nowhere else. The script prints only names, sizes and the +# certificate's subject and expiry (public data). +# +# NEEDS: kubectl with a context for the cluster (default k3s-prod), jq, curl, openssl, base64, +# and the Gitea credential `git` already has stored for the host — the same requirements as +# infra/gitea/conformance-trigger-kubeconfig.sh, run from the same place. +set -euo pipefail +HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl +NS=buildkit; SECRET=buildkit-client-ci; KCTX="${KCTX:-k3s-prod}" +CHECK=0 +for a in "$@"; do + case "$a" in + --check) CHECK=1 ;; + -h|--help) sed -n '2,/^set -euo/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;; + *) echo "unknown argument: $a" >&2; exit 2 ;; + esac +done +die() { echo "ERROR: $*" >&2; exit 1; } +for t in kubectl jq curl openssl base64 git; do command -v "$t" >/dev/null || die "missing required tool: $t"; done + +echo "==> reading Secret ${NS}/${SECRET} (context ${KCTX}; values not shown)" +js="$(kubectl --context "$KCTX" -n "$NS" get secret "$SECRET" -o json)" \ + || die "cannot read ${NS}/${SECRET} — is platform/buildkit deployed and the Certificate Ready?" +CA="$(jq -r '.data["ca.crt"] // empty' <<<"$js" | base64 -d)" +CRT="$(jq -r '.data["tls.crt"] // empty' <<<"$js" | base64 -d)" +KEY="$(jq -r '.data["tls.key"] // empty' <<<"$js" | base64 -d)" +unset js +[ -n "$CA" ] && [ -n "$CRT" ] && [ -n "$KEY" ] || die "${NS}/${SECRET} lacks ca.crt/tls.crt/tls.key" + +# Sanity before anything is written: the leaf is a CLIENT cert issued by this CA, and the key +# belongs to it. A wrong pairing would otherwise surface only as a TLS failure in CI. +openssl verify -CAfile <(printf '%s\n' "$CA") <(printf '%s\n' "$CRT") >/dev/null 2>&1 \ + || die "tls.crt does not verify against ca.crt" +openssl x509 -noout -ext extendedKeyUsage <<<"$CRT" 2>/dev/null | grep -q "TLS Web Client Authentication" \ + || die "tls.crt is not a client-auth certificate" +[ "$(openssl x509 -noout -pubkey <<<"$CRT" | openssl sha256)" = "$(openssl pkey -pubout <<<"$KEY" 2>/dev/null | openssl sha256)" ] \ + || die "tls.key does not match tls.crt" +echo " $(openssl x509 -noout -subject <<<"$CRT") / $(openssl x509 -noout -enddate <<<"$CRT")" + +creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)" +U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"; unset creds +[ -n "$U" ] && [ -n "$P" ] || die "no stored credential for $HOST" +echo "==> authenticating to Gitea as '${U}' (secret not shown)" +api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; } +API="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets" + +if [ "$CHECK" -eq 1 ]; then + # Gitea never returns a secret's value, only its name and creation time. "Stale" therefore + # means: the secret was created before the certificate's current notBefore. + names="$(api "$API" | jq -c '[.[]? | {name, created_at}]')" + nb="$(date -u -d "$(openssl x509 -noout -startdate <<<"$CRT" | cut -d= -f2)" +%s)" + rc=0 + for n in BUILDKIT_CA BUILDKIT_CLIENT_CERT BUILDKIT_CLIENT_KEY; do + c="$(jq -r --arg n "$n" '.[] | select(.name==$n) | .created_at' <<<"$names")" + if [ -z "$c" ]; then echo " MISSING $n"; rc=1 + elif [ "$(date -u -d "$c" +%s)" -lt "$nb" ]; then echo " STALE $n (stored $c, certificate re-issued since)"; rc=1 + else echo " current $n (stored $c)"; fi + done + exit "$rc" +fi + +put() { # name value + local code + code="$(api -o /dev/null -w '%{http_code}' -X PUT -H 'Content-Type: application/json' \ + -d "$(jq -n --arg d "$2" '{data:$d}')" "${API}/$1")" + [ "$code" = 201 ] || [ "$code" = 204 ] || die "PUT $1 failed: HTTP ${code}" + echo " stored $1 (${#2} bytes, value not shown)" +} +echo "==> storing repository Actions secrets" +put BUILDKIT_CA "$CA" +put BUILDKIT_CLIENT_CERT "$CRT" +put BUILDKIT_CLIENT_KEY "$KEY" +unset CA CRT KEY + +# VERIFY BY READING BACK (names only — Gitea never returns values). curl -sS exits 0 on an HTTP +# error, which is how set-ci-token.sh once claimed a store that never happened. +names="$(api "$API" | jq -r '[.[]?.name] | join(" ")')" +for n in BUILDKIT_CA BUILDKIT_CLIENT_CERT BUILDKIT_CLIENT_KEY; do + grep -qw "$n" <<<"$names" || die "stored $n but it is NOT listed (${names}) — refusing to claim success" +done +echo "==> stored and VERIFIED present. Re-run the images workflow (workflow_dispatch) to use them." diff --git a/platform/buildkit/README.md b/platform/buildkit/README.md new file mode 100644 index 0000000..ed49634 --- /dev/null +++ b/platform/buildkit/README.md @@ -0,0 +1,74 @@ +# platform/buildkit — rootless BuildKit, the CI image builder + +The builder behind [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). It +builds `toolbox`, `agent-runner` and `conformance-runner`, and on `main` pushes them to the +Gitea registry. Flux reconciles it as the Kustomization `buildkit` +([`clusters/base/buildkit.yaml`](../../clusters/base/buildkit.yaml), `dependsOn: platform`). + +## Why it exists (operator decision on PR #81, 2026-10-29) + +CI jobs run as containers on the act runner's privileged DinD sidecar, and **cannot reach any +Docker daemon**. act_runner 0.2.11 forces `docker_host: "-"` for a `tcp://` DinD, and gitea/act +strips every bind mount when `valid_volumes` is empty. Measured on PR #81's first run. Giving jobs +that daemon would make every branch push root on the node. So the rule is: **no job may reach a +privileged daemon.** Builds go to this rootless daemon over mTLS instead. + +## Shape + +| Piece | What it is | +|---|---| +| `deployment.yaml` | `moby/buildkit:v0.34.1-rootless`, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The `securityContext` is commented line by line with how each item was measured | +| `certificates.yaml` | cert-manager: a self-signed CA used only here, plus the server cert `buildkitd-tls` and the CI client cert `buildkit-client-ci` | +| `service.yaml` | ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`, mTLS only. Never on the Gateway | +| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and the Gateway (`ingress` entity) for the registry push | +| `configmap.yaml` | `buildkitd.toml`: cache GC capped at 4 GB, because the cache is an `emptyDir` on node disk | +| `versions.env` | The CI side: pinned `buildx` (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource | + +## Admission + +**No Kyverno exception is needed.** +- Every ClusterPolicy that is actually applied (the list in + [`platform-config/kustomization.yaml`](../../platform-config/kustomization.yaml)) matches only + namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`. This namespace, like + `gitea-actions`, does not carry that label. +- The job-shape policies (`pm-agent-job-shape`, `conformance-trigger-job-shape`) match only their + own namespaces. +- There is no cluster-wide Pod Security default. This namespace is deliberately left without a + `pod-security` label, because PSS `baseline` forbids the `Unconfined` seccomp and AppArmor + profiles that rootless BuildKit needs. + +## Prerequisites (operator, in order) + +1. **Unprivileged user namespaces on the nodes.** Already true. Measured 2026-10-29 on all three + Debian 13.7 nodes (kernel 7.1.8): `kernel.unprivileged_userns_clone = 1`, + `user.max_user_namespaces = 127206`, and no Ubuntu-style AppArmor userns restriction. +2. **Merge.** Flux applies this directory. Check that the Kustomization `buildkit` is Ready and + that all three Certificates in namespace `buildkit` are READY. +3. **Copy the client certificate to CI.** Run + [`infra/gitea/buildkit-client-secret.sh`](../../infra/gitea/buildkit-client-secret.sh). + Re-run it yearly, after renewal; `--check` reports whether the copy is stale. +4. **Restart buildkitd after the server certificate renews.** It reads the certificate at start, + and cert-manager renews it 30 days before expiry, so a restart inside that window is needed. + +Until step 3 is done, the `images` workflow fails at its builder preflight with a message naming +the cause. It never skips. + +## What a build step can do + +The daemon runs with `--oci-worker-no-process-sandbox`, so build steps share its PID namespace. +buildkitd warns about this itself at startup. + +- **A build step CAN** signal or ptrace the daemon and read what it can read, including this + pod's TLS key and the build cache. +- **A build step CANNOT** reach the node. Everything runs as uid 1000 in a user namespace, + unprivileged, with no host mounts. +- **Who can run one:** anyone holding the client certificate, i.e. every CI job, including pull + requests. + +## Local equivalent + +The pinned image runs under rootless podman with the same restrictions: +`--cap-drop ALL --cap-add SETUID --cap-add SETGID --security-opt seccomp=unconfined --read-only`, +plus tmpfs at the same five paths. That is how the `securityContext` was measured. Rootless podman +maps only 65,537 uids, so the image's `/etc/subuid` (`user:100000:65536`) must be overridden with +two ranges that fit: `user:1:999` and `user:1001:64535`. diff --git a/platform/buildkit/certificates.yaml b/platform/buildkit/certificates.yaml new file mode 100644 index 0000000..ac0a280 --- /dev/null +++ b/platform/buildkit/certificates.yaml @@ -0,0 +1,100 @@ +--- +# mTLS for buildkitd, issued by cert-manager from a CA that exists ONLY for this purpose. +# +# buildkit-selfsigned (Issuer) --signs--> buildkit-ca (CA Certificate, 10y) +# buildkit-ca (Issuer) --signs--> buildkitd-tls (server, the daemon presents it) +# --signs--> buildkit-client-ci (client, CI presents it) +# +# buildkitd verifies EVERY client certificate against this CA (--tlscacert), so holding a client +# certificate signed by it is what "may build" means. Nothing else in the cluster trusts this CA, +# and it trusts nothing else. +# +# THE CLIENT CERTIFICATE REACHES CI AS ACTIONS SECRETS, not through the cluster: CI jobs hold no +# cluster credential (R-SEC-02). infra/gitea/buildkit-client-secret.sh copies ca.crt/tls.crt/ +# tls.key from the Secret below into the repository secrets BUILDKIT_CA, BUILDKIT_CLIENT_CERT and +# BUILDKIT_CLIENT_KEY without printing them. Re-run it after every renewal (see renewBefore). +# +# ROTATION. cert-manager renews both leaves `renewBefore` their expiry and rewrites the Secrets. +# * server: buildkitd reads its certificate at START, so the pod must restart inside the 30-day +# renewal window (a node reboot by kured, a Deployment rollout, or a manual restart). +# * client: the Actions secrets are a COPY; re-run the script. Until then CI fails loudly at +# its TLS preflight (.gitea/workflows/scripts/images.sh), never silently. +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: buildkit-selfsigned + namespace: buildkit +spec: + selfSigned: {} +--- +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: buildkit-ca + namespace: buildkit +spec: + isCA: true + commonName: buildkit-ca + secretName: buildkit-ca + duration: 87600h # 10 years: rotating the CA means re-issuing everything and re-copying + privateKey: + algorithm: ECDSA + size: 256 + issuerRef: + name: buildkit-selfsigned + kind: Issuer +--- +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: buildkit-ca + namespace: buildkit +spec: + ca: + secretName: buildkit-ca +--- +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: buildkitd-tls + namespace: buildkit +spec: + secretName: buildkitd-tls + commonName: buildkitd.buildkit.svc.cluster.local + dnsNames: + - buildkitd.buildkit.svc.cluster.local # what CI dials (platform/buildkit/versions.env) + - buildkitd.buildkit.svc + - buildkitd + duration: 8760h + renewBefore: 720h + usages: + - server auth + - digital signature + privateKey: + algorithm: ECDSA + size: 256 + rotationPolicy: Always + issuerRef: + name: buildkit-ca + kind: Issuer +--- +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: buildkit-client-ci + namespace: buildkit +spec: + secretName: buildkit-client-ci + commonName: gitea-actions-image-pipeline + duration: 8760h + renewBefore: 720h + usages: + - client auth + - digital signature + privateKey: + algorithm: ECDSA + size: 256 + rotationPolicy: Always + issuerRef: + name: buildkit-ca + kind: Issuer diff --git a/platform/buildkit/configmap.yaml b/platform/buildkit/configmap.yaml new file mode 100644 index 0000000..1c537a9 --- /dev/null +++ b/platform/buildkit/configmap.yaml @@ -0,0 +1,24 @@ +--- +# buildkitd.toml. Validated against the pinned image (v0.34.1-rootless): `buildctl debug workers +# -v` reports "Maximum used space: 4.29GB / Reserved space: 1.07GB" for the default policy. +# +# WHY 4GB. The cache lives in an emptyDir on the NODE's disk (deployment.yaml), and the nodes had +# 7-9 GB free when this was written. The largest image (toolbox) is ~1.4 GB uncompressed; the +# three images share their Fedora base. The emptyDir's sizeLimit sits above this cap, so GC +# trims the cache well before the kubelet would evict the pod for exceeding it. +apiVersion: v1 +kind: ConfigMap +metadata: + name: buildkitd-config + namespace: buildkit + labels: + app.kubernetes.io/name: buildkitd +data: + buildkitd.toml: | + [worker.oci] + enabled = true + gc = true + maxUsedSpace = "4GB" + reservedSpace = "1GB" + [worker.containerd] + enabled = false diff --git a/platform/buildkit/deployment.yaml b/platform/buildkit/deployment.yaml new file mode 100644 index 0000000..67d826c --- /dev/null +++ b/platform/buildkit/deployment.yaml @@ -0,0 +1,142 @@ +--- +# buildkitd, ROOTLESS. The builder behind .gitea/workflows/images.yaml; see README.md. +# +# THE SECURITY CONTEXT IS THE BOUNDARY, AND EVERY LINE OF IT WAS MEASURED (2026-10-29, the pinned +# image under podman with the same restrictions; decisions/2026-10-29-image-pipeline.md): +# * privileged: false — never. This is the whole point of the operator's decision on PR #81: +# the alternative was handing CI the privileged DinD daemon, i.e. root on the node. +# * runAsUser/Group 1000 — the image's `user`; rootlesskit maps it to uid 0 INSIDE a user +# namespace only. The nodes allow that (measured: kernel.unprivileged_userns_clone = 1, +# user.max_user_namespaces = 127206 on all three Debian 13 nodes). +# * capabilities: drop ALL, add SETUID + SETGID — the image's newuidmap/newgidmap are NOT +# setuid; they carry FILE capabilities, which need SETUID/SETGID in the bounding set. +# Measured: with drop ALL and nothing added, rootlesskit fails "newuidmap ... operation not +# permitted" and the daemon never starts. +# * allowPrivilegeEscalation: true — required for the same reason: no_new_privs (what `false` +# sets) stops a file capability from ever taking effect. It does NOT grant privileges the +# bounding set above does not contain. +# * seccompProfile: Unconfined — measured: with the default seccomp profile the daemon starts +# but every RUN step fails (the runtime cannot create the build step's namespaces/mounts). +# * appArmorProfile: Unconfined — upstream's documented requirement (buildkit +# examples/kubernetes, *.rootless.yaml); the nodes run AppArmor (enabled=Y) and containerd's +# default profile denies mount. NOT measured here: the test host runs SELinux, not AppArmor. +# * readOnlyRootFilesystem: true — measured to work with the five emptyDirs below. +# +# --oci-worker-no-process-sandbox: build steps share the daemon's PID namespace instead of getting +# their own (creating one needs an unmasked /proc, i.e. more privilege). Consequence, stated by +# buildkitd itself at startup: a build step can signal or ptrace the daemon and read what the +# daemon can read — including this pod's TLS key and the build cache. So a build step is trusted +# at the level of "may impersonate this builder", which is the level anyone holding the client +# certificate already has. It can NOT reach the node: everything still runs as uid 1000 in a user +# namespace, unprivileged, with no host mounts. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: buildkitd + namespace: buildkit + labels: + app.kubernetes.io/name: buildkitd +spec: + replicas: 1 + strategy: + type: Recreate # one builder, one cache; never two daemons fighting over a node's disk + selector: + matchLabels: + app.kubernetes.io/name: buildkitd + template: + metadata: + labels: + app.kubernetes.io/name: buildkitd + spec: + automountServiceAccountToken: false # the builder needs no Kubernetes API at all + enableServiceLinks: false + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 # makes the 0440 TLS Secret below readable by uid 1000, and nobody else + containers: + - name: buildkitd + # Pinned by digest (the multi-arch index). The kubernetes manager in renovate.json5 + # tracks platform/**, so Renovate proposes bumps; the CI client (buildx, pinned in + # versions.env) talks to any recent daemon. + image: docker.io/moby/buildkit:v0.34.1-rootless@sha256:f68752a5a94acfcdad2d07867b7075601a421638e17fc5f28efc1c814d0c4f89 + args: + - --addr + - unix:///run/user/1000/buildkit/buildkitd.sock # local only: the probes below + - --addr + - tcp://0.0.0.0:1234 # CI, mTLS only + - --tlscacert + - /certs/ca.crt # client certs are VERIFIED against this CA: no cert, no build + - --tlscert + - /certs/tls.crt + - --tlskey + - /certs/tls.key + - --config + - /etc/buildkit/buildkitd.toml + - --oci-worker-no-process-sandbox + ports: + - name: buildkit + containerPort: 1234 + protocol: TCP + readinessProbe: + exec: + command: ["buildctl", "debug", "workers"] + initialDelaySeconds: 5 + periodSeconds: 30 + livenessProbe: + exec: + command: ["buildctl", "debug", "workers"] + initialDelaySeconds: 30 + periodSeconds: 60 + timeoutSeconds: 10 + securityContext: + privileged: false + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + allowPrivilegeEscalation: true + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + add: ["SETUID", "SETGID"] + seccompProfile: + type: Unconfined + appArmorProfile: + type: Unconfined + resources: + # Build steps (dnf, npm, pip) run INSIDE this container's cgroup, so the limit covers + # the daemon and the build together. + requests: + cpu: 250m + memory: 512Mi + ephemeral-storage: 2Gi + limits: + memory: 4Gi + volumeMounts: + - { name: certs, mountPath: /certs, readOnly: true } + - { name: config, mountPath: /etc/buildkit, readOnly: true } + - { name: cache, mountPath: /home/user/.local/share/buildkit } + - { name: run, mountPath: /run/user } + - { name: tmpdir, mountPath: /home/user/.local/tmp } # the image's TMPDIR + - { name: tmp, mountPath: /tmp } + - { name: clicfg, mountPath: /home/user/.docker } # buildctl (the probes) writes here + volumes: + - name: certs + secret: + secretName: buildkitd-tls # cert-manager: ca.crt + tls.crt + tls.key + defaultMode: 0440 + - name: config + configMap: + name: buildkitd-config + - name: cache + emptyDir: + sizeLimit: 6Gi # GC caps use at 4GB (configmap.yaml); this is the backstop + - name: run + emptyDir: {} + - name: tmpdir + emptyDir: {} + - name: tmp + emptyDir: {} + - name: clicfg + emptyDir: {} diff --git a/platform/buildkit/kustomization.yaml b/platform/buildkit/kustomization.yaml new file mode 100644 index 0000000..064bf0f --- /dev/null +++ b/platform/buildkit/kustomization.yaml @@ -0,0 +1,13 @@ +--- +# platform/buildkit — rootless BuildKit, the CI image builder. Reconciled by the Flux +# Kustomization `buildkit` (clusters/base/buildkit.yaml). See README.md. +# versions.env is NOT a resource: it pins the CI-side client (buildx) and the address CI dials. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - namespace.yaml + - certificates.yaml + - configmap.yaml + - deployment.yaml + - service.yaml + - networkpolicy.yaml diff --git a/platform/buildkit/namespace.yaml b/platform/buildkit/namespace.yaml new file mode 100644 index 0000000..64ae767 --- /dev/null +++ b/platform/buildkit/namespace.yaml @@ -0,0 +1,26 @@ +--- +# buildkit — the image builder for CI (.gitea/workflows/images.yaml). Operator decision on PR #81 +# (2026-10-29): images are built by a ROOTLESS BuildKit daemon reached over mTLS, and NO CI job +# may reach a privileged daemon. See README.md for the whole design. +# +# A NAMESPACE OF ITS OWN, deliberately NOT `gitea-actions`: +# * The act-runner namespace exists to hold a PRIVILEGED DinD sidecar. Putting the builder +# beside it would put a TLS server key next to a privileged container for no benefit. +# * Its NetworkPolicies, Secrets and quota then belong to one component, and the ingress rule +# ("only the act-runner pod may connect") can be written from the outside. +# +# NO `app-namespace` LABEL, like gitea-actions: this is CI plumbing, not a user-facing app. That +# is also why NO Kyverno exception is needed — every applied ClusterPolicy +# (platform-config/kustomization.yaml) matches only namespaces labelled +# homelab.keithjackson.dev/app-namespace: "true", and none of the job-shape policies match here. +# +# NO pod-security.kubernetes.io LABEL, on purpose. Rootless BuildKit needs seccomp AND AppArmor +# `Unconfined` (measured — see deployment.yaml), which PSS `baseline` forbids; there is no +# cluster-wide PSA default here, so an unlabelled namespace admits it. It is NOT privileged and +# the deployment's securityContext is the boundary — read it before relaxing anything. +apiVersion: v1 +kind: Namespace +metadata: + name: buildkit + labels: + homelab.keithjackson.dev/ci: buildkit diff --git a/platform/buildkit/networkpolicy.yaml b/platform/buildkit/networkpolicy.yaml new file mode 100644 index 0000000..1ec331d --- /dev/null +++ b/platform/buildkit/networkpolicy.yaml @@ -0,0 +1,88 @@ +--- +# Default-deny for the namespace, then two narrow allows. +# +# WHO MAY CONNECT — AND WHY THE ACT-RUNNER POD IS THE RIGHT SOURCE (measured 2026-10-29). +# CI jobs do not run as pods. act_runner starts each job as a container on its DinD sidecar, on a +# docker bridge network INSIDE the act-runner pod's network namespace, and dockerd masquerades it: +# kubectl -n gitea-actions exec deploy/act-runner -c dind -- iptables -t nat -S POSTROUTING +# -A POSTROUTING -s 172.18.0.0/16 ! -o br-c0f13d041ad6 -j MASQUERADE +# -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE +# So a job's connection leaves with the act-runner POD's IP and therefore Cilium's identity for +# that pod (namespace gitea-actions, app.kubernetes.io/name=act-runner). That is exactly what the +# ingress rule names. It cannot distinguish one job from another, or a job from act_runner +# itself — per-job authentication is the mTLS client certificate, not this policy. What the +# policy adds is that NOTHING ELSE in the cluster can even open the port. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-deny + namespace: buildkit + labels: + homelab.keithjackson.dev/baseline: default-deny +spec: + podSelector: {} + policyTypes: + - Ingress + - Egress + ingress: [] + egress: + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 +--- +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: buildkitd-allow-ci-ingress + namespace: buildkit + labels: + app.kubernetes.io/name: buildkitd +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: buildkitd + ingress: + - fromEndpoints: + - matchLabels: + io.kubernetes.pod.namespace: gitea-actions + app.kubernetes.io/name: act-runner + toPorts: + - ports: + - { port: "1234", protocol: TCP } +--- +# WHAT A BUILD NEEDS TO REACH. Build steps download (dnf mirrors, GitHub releases, dl.k8s.io, +# npm, PyPI, Fedora/Docker registries) and the result is pushed to the Gitea registry. +# * world 443/80: the downloads. 80 because Fedora's mirror list includes plain-http mirrors +# (dnf verifies package GPG signatures, not transport). +# * ingress (the Gateway's Envoy, reserved:ingress): the push. gitea.home.keithjackson.dev +# resolves in-cluster to the Gateway VIP (split-horizon CoreDNS), and Cilium hands that +# traffic to the Gateway, whose identity is `ingress` — the same identity gitea's own +# ingress allow admits. +# Nothing else in the cluster: not the API server, not other pods. A build step is code from a +# pull request. +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: buildkitd-allow-egress + namespace: buildkit + labels: + app.kubernetes.io/name: buildkitd +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: buildkitd + egress: + - toEntities: + - world + toPorts: + - ports: + - { port: "443", protocol: TCP } + - { port: "80", protocol: TCP } + - toEntities: + - ingress diff --git a/platform/buildkit/service.yaml b/platform/buildkit/service.yaml new file mode 100644 index 0000000..705f993 --- /dev/null +++ b/platform/buildkit/service.yaml @@ -0,0 +1,19 @@ +--- +# The address CI dials: tcp://buildkitd.buildkit.svc.cluster.local:1234 (platform/buildkit/ +# versions.env), mTLS only. ClusterIP — never exposed on the Gateway. +apiVersion: v1 +kind: Service +metadata: + name: buildkitd + namespace: buildkit + labels: + app.kubernetes.io/name: buildkitd +spec: + type: ClusterIP + selector: + app.kubernetes.io/name: buildkitd + ports: + - name: buildkit + port: 1234 + targetPort: buildkit + protocol: TCP diff --git a/platform/buildkit/versions.env b/platform/buildkit/versions.env new file mode 100644 index 0000000..38b9261 --- /dev/null +++ b/platform/buildkit/versions.env @@ -0,0 +1,15 @@ +# platform/buildkit/versions.env — the CI side of the builder. Sourced by +# .gitea/workflows/scripts/images.sh. NOT a Kubernetes resource (kustomization.yaml omits it). +# +# The CLIENT. CI drives the daemon with `buildx` (remote driver) rather than bare `buildctl` +# because the image pipeline needs `bake`'s `target:` contexts: on a pull request agent-runner is +# built FROM the toolbox built in the same run, which never exists in any registry. +# Pinned with the sha256 from the release's own checksums.txt +# (https://github.com/docker/buildx/releases/download/v0.38.1/checksums.txt); the pipeline +# refuses a download that does not match. Bump BOTH lines together. +BUILDX_VERSION=0.38.1 +BUILDX_SHA256=e25d5cd675092bb5d1d03632fdc46cdf3fede7b514dc5f023a105e00d3ce5ad4 + +# The DAEMON's address and the name its certificate carries (platform/buildkit/certificates.yaml). +BUILDKIT_HOST=tcp://buildkitd.buildkit.svc.cluster.local:1234 +BUILDKIT_SERVERNAME=buildkitd.buildkit.svc.cluster.local diff --git a/platform/pm-agent/README.md b/platform/pm-agent/README.md index 5247aff..809c6b5 100644 --- a/platform/pm-agent/README.md +++ b/platform/pm-agent/README.md @@ -47,9 +47,9 @@ Both are deliberate, and both fail **visibly** rather than silently: 1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and `PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder until the first - publish. `.gitea/workflows/images.yaml` builds `agent-runner` with - `infra/agent/build-image.sh` (the same script, and the same in-image verification, you can - run locally), and on a push to `main` publishes it to + publish. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit + (`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that + `infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to `gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-` and `main`. Renovate then follows the `main` tag through the `# renovate-digest:` marker above **both** lines and moves them together in one pull request. Nobody builds-and-pushes this image from From 04eb38584ab7c1885fcfb19bd766435e284239b2 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Thu, 29 Oct 2026 22:25:12 -0500 Subject: [PATCH 4/4] fix(conformance): workload-mTLS certificates may use a private CA, if they prove they are workload-internal R-TLS-01/docs/08 govern externally reachable TLS and ADR-0006 puts workload mTLS out of scope, but tls-lifecycle-automated required ACME for every Certificate, which platform/buildkit's internal CA cannot satisfy. Clause (5b) accepts the tls-scope=workload-mtls annotation only when the certificate proves it: no Gateway serves it, internal names only, namespaced selfSigned/ca issuer backed by an exempt CA certificate. Also fixes yamllint comma spacing in the buildkitd mounts. Co-Authored-By: Claude Opus 5.5 --- .../tls-lifecycle-automated/check.sh | 70 +++++++++++++++++++ decisions/2026-10-29-image-pipeline.md | 17 +++++ platform/buildkit/certificates.yaml | 15 ++++ platform/buildkit/deployment.yaml | 14 ++-- 4 files changed, 109 insertions(+), 7 deletions(-) diff --git a/conformance/tests/l0-static/tls-lifecycle-automated/check.sh b/conformance/tests/l0-static/tls-lifecycle-automated/check.sh index 294b4b1..7f1e5e9 100755 --- a/conformance/tests/l0-static/tls-lifecycle-automated/check.sh +++ b/conformance/tests/l0-static/tls-lifecycle-automated/check.sh @@ -42,6 +42,19 @@ # whose every solver is an automated challenge type (dns01 / http01) — the # same code path serves issuance AND renewal, so a solver a human must # complete would defeat both; +# (5b) the ONE exception to (5) — WORKLOAD mTLS, added 2026-10-29 — is a Certificate +# annotated `homelab.keithjackson.dev/tls-scope: workload-mtls`. The spec's TLS +# requirements are about EXTERNALLY REACHABLE services (R-TLS-01; docs/08), and +# ADR-0006 says outright that "mTLS between workloads is a separate concern … not +# covered here". A public ACME CA cannot issue for a `*.svc.cluster.local` name or a +# client-auth certificate at all. The annotation is NOT trusted: an exempt +# Certificate must PROVE it is workload-internal — its Secret is served by NO Gateway +# listener, every DNS name / commonName is cluster-internal (a bare label or +# `*.svc[.cluster.local]`), it carries no IP/URI/email SANs, and its issuer is a +# NAMESPACED Issuer in its own namespace whose spec is `selfSigned` or a `ca` backed +# by a Secret that a tracked, likewise-exempt Certificate produces. Issuance and +# renewal stay unattended (cert-manager renews CA-issued certs itself), and (9) +# still applies. Every exemption is printed. First user: platform/buildkit. # (6) each dns01 solver's provider credential is referenced BY NAME and that # Secret is delivered SOPS-encrypted from git via a kustomization — the # operator's one-time token hand-off (docs/08 OQ-4), never a per-cert step; @@ -265,6 +278,7 @@ CERT_SECRETS: dict[tuple[str, str], str] = { # (namespace, secretName) for rel, d in CERTS } listeners_checked = 0 +GATEWAY_SECRETS: set[tuple[str, str]] = set() # every (namespace, Secret) a listener serves for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"): gw_ns = ns_of(gw) for lis in ((gw.get("spec") or {}).get("listeners") or []): @@ -280,6 +294,7 @@ for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"): for r in refs: kind = str(r.get("kind", "Secret")) key = (str(r.get("namespace") or gw_ns), str(r.get("name"))) + GATEWAY_SECRETS.add(key) check(kind == "Secret", f"{rel}: listener {lis.get('name')!r} certificateRef kind is {kind!r}; only " "Secret refs can be traced to a cert-manager Certificate") @@ -324,8 +339,61 @@ CM_NS = ns_of(cm_releases[0][1], "cert-manager") if cm_releases else "cert-manag ISSUER_INDEX = {(d.get("kind"), name_of(d)): (rel, d) for rel, d in ISSUERS} solver_creds: list[tuple[str, str, str, str]] = [] # (issuer, ns, secretName, ref field) issuers_verified: set[str] = set() + +# (5b) workload-mTLS exemption — see the header. Proven, never taken on trust. +SCOPE_ANN = "homelab.keithjackson.dev/tls-scope" +INTERNAL_NAME = re.compile(r"^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*$") +NS_ISSUERS = {(ns_of(d), name_of(d)): (rel, d) for rel, d in of_kind("Issuer", "cert-manager.io/")} +EXEMPT_SECRETS = {(ns_of(d), str((d.get("spec") or {}).get("secretName", ""))) + for _, d in CERTS + if ((d.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN) == "workload-mtls"} +exempted: list[str] = [] + +def workload_mtls_ok(rel: str, cert: dict) -> bool: + """True if `cert` claims the workload-mTLS scope AND proves it; records failures.""" + ann = ((cert.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN) + if ann is None: + return False + cns, cname = ns_of(cert), name_of(cert) + spec = cert.get("spec") or {} + ok = True + def need(cond, msg): + nonlocal ok + check(cond, f"{rel}: Certificate {cns}/{cname} claims {SCOPE_ANN}={ann!r} but {msg}") + ok = ok and bool(cond) + need(ann == "workload-mtls", "the only accepted value is 'workload-mtls'") + need((cns, str(spec.get("secretName", ""))) not in GATEWAY_SECRETS, + "its Secret is served by a Gateway listener — that is external TLS, which must be ACME (R-TLS-01)") + names = [str(n) for n in (spec.get("dnsNames") or [])] + if spec.get("commonName"): + names.append(str(spec["commonName"])) + for n in names: + internal = INTERNAL_NAME.match(n) and ("." not in n or n.endswith(".svc") or n.endswith(".svc.cluster.local")) + need(bool(internal), f"name {n!r} is not cluster-internal (bare label or *.svc[.cluster.local])") + for f in ("ipAddresses", "uris", "emailAddresses"): + need(not spec.get(f), f"it carries {f}, which a workload-internal certificate has no use for") + ref = spec.get("issuerRef") or {} + need(str(ref.get("kind", "Issuer")) == "Issuer", + "its issuer is not a NAMESPACED Issuer — a ClusterIssuer could sign for any namespace") + hit = NS_ISSUERS.get((cns, str(ref.get("name", "")))) + need(hit is not None, f"its Issuer {ref.get('name')!r} is not tracked in namespace {cns!r}") + if hit is not None: + ispec = hit[1].get("spec") or {} + kinds = sorted(ispec) + need(kinds in (["selfSigned"], ["ca"]), + f"its Issuer is {kinds} — only selfSigned or ca are accepted for workload mTLS") + if kinds == ["ca"]: + need((cns, str((ispec.get("ca") or {}).get("secretName", ""))) in EXEMPT_SECRETS, + "its CA Issuer's Secret is not produced by a tracked workload-mtls Certificate " + "in the same namespace — the CA itself would have to be minted by hand") + if ok: + exempted.append(f"{cns}/{cname}") + return ok + for rel, cert in CERTS: spec = cert.get("spec") or {} + if workload_mtls_ok(rel, cert): + continue # (5b): proven workload-internal; (9) below still applies ref = spec.get("issuerRef") or {} kind = str(ref.get("kind", "Issuer")) per_env = resolve_per_env(str(ref.get("name", ""))) @@ -379,6 +447,8 @@ for rel, cert in CERTS: solver_creds.append((iname, want_ns, n, f"{provider}.{field}")) check(len(issuers_verified) > 0, "no issuer was reachable from any Certificate — (5) proved nothing") +if exempted: + print(f" (5b) workload-mTLS certificates exempted from ACME after proving it: {', '.join(exempted)}") # (6) the solver credential is delivered, SOPS-encrypted, from git — never per-cert by hand. check(len(solver_creds) > 0, diff --git a/decisions/2026-10-29-image-pipeline.md b/decisions/2026-10-29-image-pipeline.md index 40c3909..9620e35 100644 --- a/decisions/2026-10-29-image-pipeline.md +++ b/decisions/2026-10-29-image-pipeline.md @@ -85,6 +85,23 @@ valid_volumes) is **untouched**. - AppArmor enabled (`Y`). So this is a satisfied prerequisite, not an open one. +- **The TLS conformance check needed a narrow, self-proving exception** (`conformance/**`, so this + changes what green means; flagged for the reviewer). + - The first gate run failed L0 `tls-lifecycle-automated` clause (5): "every Certificate's + issuer must be ACME". That is broader than the spec it serves: + - R-TLS-01 and docs/08 concern *externally reachable* services; + - ADR-0006 says "mTLS between workloads is a separate concern … not covered here"; + - a public CA cannot issue for a `*.svc.cluster.local` name or a client-auth certificate. + - The new clause (5b) exempts a Certificate annotated + `homelab.keithjackson.dev/tls-scope: workload-mtls` **only if it proves it**: + - no Gateway listener serves its Secret; + - every DNS name and the commonName are a bare label or `*.svc[.cluster.local]`; + - it has no IP, URI or email SANs; + - its issuer is a namespaced `selfSigned`/`ca` Issuer in its own namespace; + - a `ca` issuer's Secret is produced by a likewise-exempt tracked Certificate. + - Exemptions are printed, and clause (9), the renewal arithmetic, still applies. + - **Negative-tested:** a public DNS name, and a Gateway serving `buildkitd-tls`, each fail the + check with a specific message. - **`infra/gitea/buildkit-client-secret.sh`** (operator action): copies `ca.crt`, `tls.crt` and `tls.key` from the Secret `buildkit/buildkit-client-ci` into the Actions secrets `BUILDKIT_CA`, `BUILDKIT_CLIENT_CERT` and `BUILDKIT_CLIENT_KEY`. Before writing anything it checks that: diff --git a/platform/buildkit/certificates.yaml b/platform/buildkit/certificates.yaml index ac0a280..2f5b2b0 100644 --- a/platform/buildkit/certificates.yaml +++ b/platform/buildkit/certificates.yaml @@ -14,6 +14,15 @@ # tls.key from the Secret below into the repository secrets BUILDKIT_CA, BUILDKIT_CLIENT_CERT and # BUILDKIT_CLIENT_KEY without printing them. Re-run it after every renewal (see renewBefore). # +# WHY A PRIVATE CA AND NOT LET'S ENCRYPT. The platform's TLS rule (R-TLS-01/02, docs/08, +# ADR-0006: ACME DNS-01) is about EXTERNALLY REACHABLE services; ADR-0006 itself says "mTLS between +# workloads is a separate concern … not covered here", and a public CA cannot issue for a +# `*.svc.cluster.local` name or a client-auth certificate anyway. buildkitd is ClusterIP-only. Each +# Certificate below is therefore annotated `homelab.keithjackson.dev/tls-scope: workload-mtls`, +# which the L0 check tls-lifecycle-automated accepts ONLY if the certificate proves it: no +# Gateway serves its Secret, every name is cluster-internal, and the issuer is a namespaced +# selfSigned/CA Issuer whose CA is itself issued here. Issuance and renewal stay unattended. +# # ROTATION. cert-manager renews both leaves `renewBefore` their expiry and rewrites the Secrets. # * server: buildkitd reads its certificate at START, so the pod must restart inside the 30-day # renewal window (a node reboot by kured, a Deployment rollout, or a manual restart). @@ -32,6 +41,8 @@ kind: Certificate metadata: name: buildkit-ca namespace: buildkit + annotations: + homelab.keithjackson.dev/tls-scope: workload-mtls # see the header: R-TLS-01 is about external TLS spec: isCA: true commonName: buildkit-ca @@ -58,6 +69,8 @@ kind: Certificate metadata: name: buildkitd-tls namespace: buildkit + annotations: + homelab.keithjackson.dev/tls-scope: workload-mtls spec: secretName: buildkitd-tls commonName: buildkitd.buildkit.svc.cluster.local @@ -83,6 +96,8 @@ kind: Certificate metadata: name: buildkit-client-ci namespace: buildkit + annotations: + homelab.keithjackson.dev/tls-scope: workload-mtls spec: secretName: buildkit-client-ci commonName: gitea-actions-image-pipeline diff --git a/platform/buildkit/deployment.yaml b/platform/buildkit/deployment.yaml index 67d826c..ae057ef 100644 --- a/platform/buildkit/deployment.yaml +++ b/platform/buildkit/deployment.yaml @@ -114,13 +114,13 @@ spec: limits: memory: 4Gi volumeMounts: - - { name: certs, mountPath: /certs, readOnly: true } - - { name: config, mountPath: /etc/buildkit, readOnly: true } - - { name: cache, mountPath: /home/user/.local/share/buildkit } - - { name: run, mountPath: /run/user } - - { name: tmpdir, mountPath: /home/user/.local/tmp } # the image's TMPDIR - - { name: tmp, mountPath: /tmp } - - { name: clicfg, mountPath: /home/user/.docker } # buildctl (the probes) writes here + - { name: certs, mountPath: /certs, readOnly: true } + - { name: config, mountPath: /etc/buildkit, readOnly: true } + - { name: cache, mountPath: /home/user/.local/share/buildkit } + - { name: run, mountPath: /run/user } + - { name: tmpdir, mountPath: /home/user/.local/tmp } # the image's TMPDIR + - { name: tmp, mountPath: /tmp } + - { name: clicfg, mountPath: /home/user/.docker } # buildctl (the probes) writes here volumes: - name: certs secret: