From aacf361bb8044ce481e22750b7d89df8a7b6ba11 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Thu, 29 Oct 2026 20:58:15 -0500 Subject: [PATCH 1/7] fix(renovate): the workflow still named renovate.json, so every run since #79 died with ENOENT MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #79 renamed the config to renovate.json5 and left configurationFile pointing at the old name. Runs 315-322 (2026-10-22..29) all failed in ~1s before Renovate started. Also add a ci/lint.sh assertion that every workflow configurationFile: resolves to a real path — valid YAML, so yamllint could not catch it, and the gate stayed green throughout. Co-Authored-By: Claude Opus 5.5 --- .gitea/workflows/renovate.yaml | 6 ++- ci/lint.sh | 15 ++++++ .../2026-10-29-renovate-config-filename.md | 48 +++++++++++++++++++ 3 files changed, 68 insertions(+), 1 deletion(-) create mode 100644 decisions/2026-10-29-renovate-config-filename.md diff --git a/.gitea/workflows/renovate.yaml b/.gitea/workflows/renovate.yaml index 91798e6..3fbb161 100644 --- a/.gitea/workflows/renovate.yaml +++ b/.gitea/workflows/renovate.yaml @@ -152,7 +152,11 @@ jobs: # Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers # includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`. # This is the one dependency in the repo whose updater is its own updater. - configurationFile: renovate.json + # MUST match the file's real name. #79 renamed it to renovate.json5 and left this line + # saying `renovate.json`; every scheduled run from 2026-10-18 then died in ~1s with + # ENOENT before Renovate started, and nothing in the gate noticed. ci/lint.sh now + # asserts that every `configurationFile:` here names a file that exists. + configurationFile: renovate.json5 token: ${{ secrets.RENOVATE_TOKEN }} env: # Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own diff --git a/ci/lint.sh b/ci/lint.sh index be703aa..e5307fe 100755 --- a/ci/lint.sh +++ b/ci/lint.sh @@ -27,3 +27,18 @@ fi yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${files[@]}" ok "yaml/lint passed (${#files[@]} files)" + +# A workflow's `configurationFile:` must name a file that exists. Renovate's did not after #79 +# renamed renovate.json -> renovate.json5, and every scheduled run failed with ENOENT for 11 days +# while the gate stayed green. Valid YAML, so yamllint cannot see it. +missing=0 +while IFS=: read -r wf _ value; do + value="${value#*configurationFile:}" + value="$(echo "$value" | sed -E 's/#.*//; s/^[[:space:]"'\'']+//; s/[[:space:]"'\'']+$//')" + if [ -n "$value" ] && [ ! -e "$value" ]; then + err "${wf}: configurationFile '${value}' does not exist in the repo" + missing=1 + fi +done < <(grep -nE '^[[:space:]]*configurationFile:' .gitea/workflows/*.y*ml 2>/dev/null || true) +[ "$missing" -eq 0 ] || exit 1 +ok "workflow configurationFile references resolve" diff --git a/decisions/2026-10-29-renovate-config-filename.md b/decisions/2026-10-29-renovate-config-filename.md new file mode 100644 index 0000000..bd30d4f --- /dev/null +++ b/decisions/2026-10-29-renovate-config-filename.md @@ -0,0 +1,48 @@ +--- +date: 2026-10-29 +branch: fix/renovate-config-filename +title: Renovate has failed every run since #79 — the workflow still named renovate.json +--- + +## What changed + +**`.gitea/workflows/renovate.yaml`** — `configurationFile: renovate.json` → `renovate.json5`. + +PR #79 (`fix/renovate-invalid-config`) renamed the config to `renovate.json5` and did not update +the one line in the workflow that names it. Measured 2026-10-29 from the Gitea Actions log of run +322 (task 513, schedule, 08:15 UTC): + + Error: ENOENT: no such file or directory, stat '/workspace/Home-Network/home-network-k3s-impl/renovate.json' + +Runs 315–322 (2026-10-22 → 10-29, every one listed) are all `failure` on the `schedule` event, +each failing in about one second. Renovate never started. `RENOVATE_TOKEN` **does** exist as a +repository Actions secret (listed by name via the API; value not read), so the token preflight +passes and this filename is the only blocker observed. + +**`ci/lint.sh`** — a new assertion after yamllint: every non-comment `configurationFile:` key in +`.gitea/workflows/*.y*ml` must name a path that exists. This is valid YAML, so yamllint cannot see +it, and the gate stayed green through 11 days of failures. + +## What I verified + +- The root cause, from the run log quoted above, and that `origin/main` has `renovate.json5` and + no `renovate.json` (`git ls-tree`). +- The new guard, both ways, run standalone: on this branch it prints + `workflow configurationFile references resolve`, rc 0. With the line reverted to `renovate.json` + it prints `configurationFile 'renovate.json' does not exist in the repo`, rc 1. My first version + also matched its own explanatory comment in the workflow and failed. That is why the grep is + anchored to non-comment lines. +- Toolbox gate: result and rc are reported in the PR, not predicted here. + +## What I did NOT verify + +- **Renovate has still never completed a run.** This removes the observed blocker. Others may + follow it. The test is a `workflow_dispatch` with `dry_run: true` after merge, and its log is + the first evidence that Renovate authenticates, extracts dependencies, and honours `.json5`. +- **The `enabledManagers: ["regex"]` vs `custom.regex` question** (#73, #77) is still open. The + same dry-run log answers it: look for the regex/custom manager's dependency-extraction count. +- **No alert for "Renovate keeps failing".** The guard prevents this specific cause from recurring. + Nothing notices a scheduled workflow that is red every day. That gap stays open. +- **Sensitive path.** `.gitea/**` and `ci/**` are both in `approval-policy.json`. CI will block + on `approval-policy` until the operator posts `/approve-override` as a review. The local gate + reports that stage as N/A. From da1b36eb64857f81852e75ec5c833fe32364cc3f Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Thu, 29 Oct 2026 21:49:38 -0500 Subject: [PATCH 2/7] =?UTF-8?q?fix(renovate):=20pin=20renovate-version=20?= =?UTF-8?q?=E2=80=94=20the=20action's=20default=20(39)=20rejects=20renovat?= =?UTF-8?q?e.json5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- .gitea/workflows/renovate.yaml | 5 +++++ decisions/2026-10-29-renovate-config-filename.md | 9 +++++++++ 2 files changed, 14 insertions(+) diff --git a/.gitea/workflows/renovate.yaml b/.gitea/workflows/renovate.yaml index 3fbb161..c313aac 100644 --- a/.gitea/workflows/renovate.yaml +++ b/.gitea/workflows/renovate.yaml @@ -158,6 +158,11 @@ jobs: # asserts that every `configurationFile:` here names a file that exists. configurationFile: renovate.json5 token: ${{ secrets.RENOVATE_TOKEN }} + # PINNED 2026-10-29. Unset, this action version runs Renovate `39` (action.yml + # `renovate-version` default at v41.0.9), i.e. 39.264.0, and renovate.json5 uses + # `managerFilePatterns` (>= 40.2.0): under 39.x validation fails with five errors and + # the run stops at CONFIG_VALIDATION. #79 validated against `latest`, not this. + renovate-version: 44.167.0 env: # Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own # header said were "set at runtime by the operator's renovate runner" — this is that diff --git a/decisions/2026-10-29-renovate-config-filename.md b/decisions/2026-10-29-renovate-config-filename.md index bd30d4f..cf26844 100644 --- a/decisions/2026-10-29-renovate-config-filename.md +++ b/decisions/2026-10-29-renovate-config-filename.md @@ -23,6 +23,13 @@ passes and this filename is the only blocker observed. `.gitea/workflows/*.y*ml` must name a path that exists. This is valid YAML, so yamllint cannot see it, and the gate stayed green through 11 days of failures. +**Second blocker, same PR:** the workflow did not pin a Renovate version, so +`renovatebot/github-action@v41.0.9` ran its default, Renovate **39** (`action.yml`: +`renovate-version` `default: '39'`, read at that tag). `renovate.json5` uses `managerFilePatterns`, +which needs >= 40.2.0. Under 39.264.0 the validator rejects the config with five errors. This was +found and reproduced by the image-pipeline worker (PR #81). #79 had validated against `latest`. Pinned +`renovate-version: 44.167.0`, the version the config validates clean against per that worker. + ## What I verified - The root cause, from the run log quoted above, and that `origin/main` has `renovate.json5` and @@ -36,6 +43,8 @@ it, and the gate stayed green through 11 days of failures. ## What I did NOT verify +- **The 39.x rejection and the 44.167.0 pass were not reproduced by me.** I confirmed only the + action's default (`'39'`). The validation results are the PR #81 worker's. - **Renovate has still never completed a run.** This removes the observed blocker. Others may follow it. The test is a `workflow_dispatch` with `dry_run: true` after merge, and its log is the first evidence that Renovate authenticates, extracts dependencies, and honours `.json5`. From e5391098302e7709e64b779fc1dbcb8b1551c778 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Thu, 29 Oct 2026 21:27:44 -0500 Subject: [PATCH 3/7] feat(images): build, verify and publish the three images in CI; Renovate carries the digests Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a workstation build/push. Nothing had ever been pushed to the Gitea registry. - .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as sha- + main, reading the digest back from the registry. Only changed images build. - Each image's own build-image.sh does the build and in-image verification, now engine-neutral (DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh. - Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex manager for them. The worker Job template is deliberately untracked (it would loop). - renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION. - Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh. Co-Authored-By: Claude Opus 5.5 --- .gitea/workflows/README.md | 33 ++ .gitea/workflows/images.yaml | 174 +++++++++++ .gitea/workflows/renovate.yaml | 18 +- .gitea/workflows/scripts/images.sh | 291 ++++++++++++++++++ conformance/incluster/README.md | 11 +- conformance/incluster/build-image.sh | 84 ++--- .../alert-rules-unit-tested/check.sh | 13 +- conformance/versions.env | 39 ++- decisions/2026-10-29-image-pipeline.md | 222 +++++++++++++ docs/plans/pm-agent-issue-to-pr.md | 10 +- infra/agent/build-image.sh | 140 +++++---- infra/agent/probe-validation.sh | 4 +- infra/agent/versions.env | 14 +- infra/agent/worker-job-template.yaml | 6 + infra/devcontainer/README.md | 16 + infra/devcontainer/build-image.sh | 108 +++++++ infra/devcontainer/dev.sh | 20 +- infra/gitea/conformance-trigger-kubeconfig.sh | 11 +- infra/gitea/registry-retention.sh | 130 ++++++++ platform/pm-agent/README.md | 21 +- platform/pm-agent/configmap.yaml | 13 +- platform/pm-agent/deployment.yaml | 7 +- renovate.json5 | 95 +++++- runbooks/pm-agent.md | 4 +- 24 files changed, 1274 insertions(+), 210 deletions(-) create mode 100644 .gitea/workflows/images.yaml create mode 100755 .gitea/workflows/scripts/images.sh create mode 100644 decisions/2026-10-29-image-pipeline.md create mode 100755 infra/devcontainer/build-image.sh create mode 100755 infra/gitea/registry-retention.sh diff --git a/.gitea/workflows/README.md b/.gitea/workflows/README.md index 952e6ce..c42412c 100644 --- a/.gitea/workflows/README.md +++ b/.gitea/workflows/README.md @@ -115,14 +115,47 @@ CI. That is a real coverage gap, not a phase. | variable | `REQUIRE_CLUSTER_TESTS` | `'true'` (Phase 1+) makes unverified cluster-needing touched reqs a hard block | | secret | `CONFORMANCE_TRIGGER_KUBECONFIG` | narrow creds to create/await the in-cluster Chainsaw Job (not cluster-admin) | +## How images are built + +This repo's three images — `toolbox` (`infra/devcontainer/`), `agent-runner` (`infra/agent/`, +built `FROM` the toolbox) and `conformance-runner` (`conformance/incluster/`) — are built, +verified and published by [`images.yaml`](images.yaml), never from a workstation +(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh); each +image is built by **its own** `build-image.sh`, the same script a developer runs locally, so the +in-image verification (non-root uid, read-only rootfs, tool presence, `PM_SELFTEST=1 +pm-dispatch`, the `conformance-clone` refusals) exists once. + +| Trigger | What happens | +|---|---| +| pull request touching an image's inputs | build + verify only. Nothing is pushed and **no registry credential is referenced** | +| push to `main` touching an image's inputs | build + verify + publish to `gitea.home.keithjackson.dev/home-network/` as `sha-` (immutable) and `main` (moving); the **registry-returned** digest is in the job summary | +| `workflow_dispatch` | the same as a push when run on `main` (all images by default); verify-only elsewhere | + +Only images whose inputs changed are built. The publish job **never writes to git**: Renovate +follows each image's `main` tag through a `# renovate-digest:` marker next to every digest +reference (`platform/pm-agent/`, `infra/agent/versions.env`, `conformance/versions.env`) and +opens the bump PR, which goes through the gate like anything else. A PR opened with the Actions +token would not trigger the required gate, which is why it is Renovate's job and not this one's. + +Credential: the repository secret **`REGISTRY_PUSH_TOKEN`** (scope `write:package`, owned by an +account with package write on the org). The automatic Actions token cannot push in Gitea 1.24.3 +— the reasoning, with source citations, is in the header of `images.yaml`. Retention: +[`infra/gitea/registry-retention.sh`](../../infra/gitea/registry-retention.sh), an operator +action, keeps every digest git pins plus the newest tags. + +Cost on this runner: the job shares the single act_runner slot with the gate, and the DinD store +is a node `emptyDir`; the job prunes it on exit. + ## Files ``` .gitea/workflows/ merge-gate.yaml the pipeline (lint → l0 → l1 → conformance → decision) + images.yaml build/verify/publish this repo's container images (see above) scripts/ install-tools.sh installs the pinned tools per stage (from ci/versions.env) decide.sh the R-TEST-03 merge decision over job results + images.sh the image pipeline's driver (what to build, base, push, digest) README.md this file ``` diff --git a/.gitea/workflows/images.yaml b/.gitea/workflows/images.yaml new file mode 100644 index 0000000..3d027bc --- /dev/null +++ b/.gitea/workflows/images.yaml @@ -0,0 +1,174 @@ +# Container images — built, verified and published by CI, never from a workstation. +# +# ═══════════════════════════════════════════════════════════════════════════════════════════════ +# WHY THIS FILE EXISTS (operator rule, 2026-10-29) +# ═══════════════════════════════════════════════════════════════════════════════════════════════ +# "Container image creation, hosting and lifecycle must be part of CI/CD and the codebase, never a +# one-off build/push from a workstation." Before this workflow, three images existed only as +# workstation builds and NOTHING had ever been pushed to the Gitea registry: the Home-Network org's +# package list was empty, platform/pm-agent/ pinned an all-zero digest and sat in ImagePullBackOff, +# and conformance/versions.env carried an empty digest. Both build scripts said "publishing is an +# operator action" — a step with no log, no recorded inputs and no verification anyone could read. +# +# WHAT IT DOES — the logic is in scripts/images.sh; this file only wires triggers and secrets. +# pull_request -> build + VERIFY only. No push, and no registry credential is referenced. +# push to main -> build + verify + PUBLISH to gitea.home.keithjackson.dev/home-network/ +# dispatch -> same as a push when run on main; verify-only on any other ref. +# Images: toolbox (infra/devcontainer), agent-runner (infra/agent, FROM toolbox) and +# conformance-runner (conformance/incluster). Only images whose inputs changed are built. +# +# THE DIGEST LIFECYCLE — this workflow NEVER writes to git. +# 1. publish pushes `sha-` (immutable) and `main` (moving), and prints the digest the +# REGISTRY returns in the job summary. +# 2. Renovate (renovate.yaml) follows the `main` tag of each image through the +# `# renovate-digest:` markers next to every digest reference, and opens the bump PR. +# 3. That PR goes through the gate like any other change, and Flux deploys it. +# The bump PR is Renovate's and not this job's because a PR opened with the Actions token does not +# trigger workflows, so it could never pass the REQUIRED gate check; Renovate's token is a real +# user's. +# +# THE PUSH CREDENTIAL — `REGISTRY_PUSH_TOKEN`, a repository Actions secret, referenced BY NAME. +# The automatic Actions token CANNOT be used. Gitea 1.24.3 source, verified: +# * services/auth/basic.go (Basic.Verify) turns a task token into the pseudo-user +# `gitea-actions` (ActionsUserID -2) and sets NO token scope; +# * routers/api/packages/container/container.go (Authenticate, the /v2/token exchange every +# docker client uses) then requires read:package / write:package / all scope and answers 401; +# * services/context/package.go (determineAccessMode) has `// TODO: ActionUser permission check` +# and no branch for that user, so even past login it would get at most READ on a public org. +# So: a personal access token with scope `write:package`, minted by an account that has package +# WRITE on the Home-Network org (org owner, or a team with the Packages unit at write). A dedicated +# bot account in a packages-only team is the least-privilege choice. The username sent with it is +# not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER` +# is optional and only labels the login. +# +# THE RUNNER (apps/wave-1/gitea-actions.yaml): act_runner 0.2.11, one job at a time, Docker +# executor on a privileged `docker:27-dind` sidecar whose image store is an emptyDir on the node +# (no sizeLimit) and whose memory limit is 2Gi — shared by the job container AND every build +# container. The toolbox is ~1.4 GB uncompressed and agent-runner adds ~0.3 GB on top of it, so the +# cleanup step below is not optional: without it each run would leave ~3 GB on a node that had +# 7 GB free when this was written, and the next eviction would take the runner with it. +name: images + +on: + pull_request: + branches: [main] + paths: + - infra/devcontainer/Containerfile + - infra/devcontainer/versions.env + - infra/devcontainer/build-image.sh + - infra/agent/** + - conformance/incluster/Containerfile + - conformance/incluster/conformance-clone + - conformance/incluster/build-image.sh + - .gitea/workflows/images.yaml + - .gitea/workflows/scripts/images.sh + push: + branches: [main] + paths: + - infra/devcontainer/Containerfile + - infra/devcontainer/versions.env + - infra/devcontainer/build-image.sh + - infra/agent/** + - conformance/incluster/Containerfile + - conformance/incluster/conformance-clone + - conformance/incluster/build-image.sh + - .gitea/workflows/images.yaml + - .gitea/workflows/scripts/images.sh + workflow_dispatch: + inputs: + images: + description: "Which images: 'all', or a space-separated subset of: toolbox agent-runner conformance-runner" + required: false + default: "all" + +concurrency: + # Never cancel a publish half-way: a pushed `sha-` tag with no matching `main` move is + # recoverable (a rerun skips what is already published) but pointless to provoke. + group: images-${{ github.ref }} + cancel-in-progress: false + +jobs: + images: + runs-on: ubuntu-latest + # Measured on the PR that introduced this file — see its decisions entry. Generous because a + # cold DinD store downloads the Fedora base and every pinned tool on each run. + timeout-minutes: 75 + steps: + - name: Checkout + uses: actions/checkout@v4 + with: + fetch-depth: 0 # the per-image change detection diffs against the base commit + + # ── pull requests: build + verify, NO credential ─────────────────────────────────────── + - name: Build and verify (pull request — nothing is pushed) + if: github.event_name == 'pull_request' + env: + MODE: verify + IMAGES: auto + PR_BASE_REF: ${{ github.base_ref }} + run: | + set -euo pipefail + BASE_SHA="$(git merge-base "origin/${PR_BASE_REF}" HEAD)" + export BASE_SHA + ./.gitea/workflows/scripts/images.sh + + # ── non-main dispatch: verify only ────────────────────────────────────────────────────── + - name: Build and verify (dispatch off main — nothing is pushed) + if: github.event_name == 'workflow_dispatch' && github.ref != 'refs/heads/main' + env: + MODE: verify + IMAGES: ${{ inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images }} + run: ./.gitea/workflows/scripts/images.sh + + # ── main: publish ─────────────────────────────────────────────────────────────────────── + - name: Preflight - the push token must exist, and its absence must be LOUD + if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main' + env: + REGISTRY_PUSH_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }} + run: | + set -euo pipefail + if [ -z "${REGISTRY_PUSH_TOKEN:-}" ]; then + echo "::error::REGISTRY_PUSH_TOKEN is not set on this repository." >&2 + cat >&2 <<'MSG' + + Images cannot be published without a registry credential, and this step fails rather + than skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would + make "nothing was published" look exactly like "published". That is how this repo went + a month with images that existed only on one workstation. + + WHY NOT THE BUILT-IN ACTIONS TOKEN: Gitea 1.24.3 gives it no package scope at the + registry's /v2/token exchange (routers/api/packages/container/container.go, + Authenticate) — see the header of .gitea/workflows/images.yaml. + + TO FIX: as an account with package WRITE on the Home-Network org (ideally a dedicated + bot in a packages-only team), create an access token with scope `write:package` and + nothing else, then add it to this repository as the Actions secret + REGISTRY_PUSH_TOKEN. Optionally set the Actions variable REGISTRY_PUSH_USER to that + account's name. Do not commit the token anywhere. Then re-run this workflow + (workflow_dispatch on main, images: all). + MSG + exit 1 + fi + echo "REGISTRY_PUSH_TOKEN is present (value not printed)." + + - name: Build, verify and publish (main) + if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main' + env: + MODE: publish + IMAGES: ${{ github.event_name == 'workflow_dispatch' && (inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images) || 'auto' }} + BASE_SHA: ${{ github.event.before }} + REGISTRY_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }} + REGISTRY_USER: ${{ vars.REGISTRY_PUSH_USER }} + run: ./.gitea/workflows/scripts/images.sh + + # ── always: give the DinD store back ──────────────────────────────────────────────────── + # The runner's image store is a node emptyDir with no sizeLimit (see header). Everything this + # job built or pulled is removed; `prune -a` cannot remove the job's own image, because the + # job container is still running on it. + - name: Clean the DinD image store + if: always() + run: | + docker logout gitea.home.keithjackson.dev >/dev/null 2>&1 || true + docker image prune -af >/dev/null || true + docker builder prune -af >/dev/null || true + docker system df || true diff --git a/.gitea/workflows/renovate.yaml b/.gitea/workflows/renovate.yaml index c313aac..f635ebf 100644 --- a/.gitea/workflows/renovate.yaml +++ b/.gitea/workflows/renovate.yaml @@ -60,8 +60,9 @@ # entry's "What I did NOT verify" — that gap is stated, not papered over. # # OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE: -# `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository` and -# `read:user`. Set it as an Actions secret on this repository. Nothing in this repo mints, +# `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository`, +# `read:user` and (since 2026-10-29) `read:package`. Set it as an Actions secret on this +# repository. Nothing in this repo mints, # stores or references its value; it is consumed by name only. # Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a # visible red is the correct state for "automation configured but not credentialed". @@ -139,8 +140,9 @@ jobs: seven files claimed "Renovate maintains this pin". TO FIX: create a Gitea access token for the bot account with scopes - `write:repository` and `read:user`, then add it to this repository as the Actions - secret RENOVATE_TOKEN. Do not commit it anywhere. + `write:repository`, `read:user` and `read:package` (the last one lets Renovate read + this repo's own images in the Gitea registry), then add it to this repository as the + Actions secret RENOVATE_TOKEN. Do not commit it anywhere. MSG exit 1 fi @@ -179,6 +181,14 @@ jobs: RENOVATE_REQUIRE_CONFIG: "required" RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }} LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }} + # ADDED 2026-10-29: registry read access for this repo's OWN images (toolbox, + # agent-runner, conformance-runner — published by .gitea/workflows/images.yaml and + # followed through `# renovate-digest:` markers). The Gitea registry answers 401 to + # anonymous reads, and Renovate scopes the platform token to hostType `gitea`, which a + # docker lookup never matches — so without this rule every digest lookup fails. Same + # token; it therefore needs scope `read:package` in addition to the ones listed in the + # preflight above. Gitea ignores the username for token auth (services/auth/basic.go). + RENOVATE_HOST_RULES: '[{"hostType":"docker","matchHost":"gitea.home.keithjackson.dev","username":"renovate","password":"${{ secrets.RENOVATE_TOKEN }}"}]' - name: Summary if: always() diff --git a/.gitea/workflows/scripts/images.sh b/.gitea/workflows/scripts/images.sh new file mode 100755 index 0000000..2d83fb4 --- /dev/null +++ b/.gitea/workflows/scripts/images.sh @@ -0,0 +1,291 @@ +#!/usr/bin/env bash +############################################################################### +# .gitea/workflows/scripts/images.sh — the image pipeline's driver (images.yaml calls this). +# +# Builds, verifies and (on main) PUBLISHES the three images this repo ships: +# +# toolbox infra/devcontainer/ the pinned dev/gate toolchain +# agent-runner infra/agent/ FROM toolbox; the PM dispatcher + worker +# conformance-runner conformance/incluster/ the in-cluster L2/L3 Chainsaw runner +# +# Each image is built by ITS OWN build-image.sh — the same script a developer runs locally — +# so the in-image verification (non-root uid, read-only rootfs, tool presence, pm-dispatch +# selftest, conformance-clone refusals, ...) is written once and runs in both places. This +# file only decides WHAT to build, in which order, FROM which base, and where it goes. +# +# MODES (env MODE): +# verify build + verify, never push, holds NO credential. Pull requests. +# publish build + verify + push to ${REGISTRY}/${NAMESPACE}/ with tags +# `sha-<12 hex>` (immutable: never re-pushed) and `main` (moving; Renovate follows +# it). The digest recorded is the one the REGISTRY returns, read back after the push. +# +# WHAT GETS BUILT (env IMAGES): `auto` (default) diffs BASE_SHA..HEAD_SHA against each image's +# inputs; otherwise a space-separated list. A change to the pipeline itself (images.yaml, this +# file) builds everything in verify mode, and nothing extra in publish mode: re-publishing +# unchanged inputs would only mint new digests and churn Renovate PRs. +# +# THE agent-runner BASE — the one ordering rule, and the reason this is not three jobs: +# * toolbox built in this run -> agent-runner builds FROM it (in publish mode, FROM the +# digest the registry just returned for it). +# * verify mode, toolbox not changed -> the toolbox is built anyway as a base. A pull request +# holds no registry credential, so it cannot pull the published one. +# * publish mode, toolbox not changed -> AGENT_BASE_IMAGE from infra/agent/versions.env, +# i.e. the published toolbox digest that Renovate keeps current. If that is still the +# all-zero placeholder the toolbox has never been published, so it is built and published +# first (this is how the very first run bootstraps). +# +# LOOP-FREEDOM. Renovate turns each publish into a digest-bump PR. Those bumps land in +# platform/pm-agent/ (no image input), conformance/versions.env (no image input) and +# infra/agent/versions.env (agent-runner input -> ONE more agent-runner build, whose own bump +# lands in platform/pm-agent/ and stops). The worker Job template is deliberately NOT tracked: +# it is baked INTO agent-runner, so bumping it would rebuild agent-runner, mint a new digest, +# bump it again, forever. pm-dispatch overwrites that field at runtime anyway. +############################################################################### +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" +cd "${REPO_ROOT}" + +MODE="${MODE:-verify}" +IMAGES="${IMAGES:-auto}" +ENGINE="${ENGINE:-docker}" +REGISTRY="${REGISTRY:-gitea.home.keithjackson.dev}" +NAMESPACE="${NAMESPACE:-home-network}" +HEAD_SHA="${HEAD_SHA:-$(git rev-parse HEAD)}" +BASE_SHA="${BASE_SHA:-}" +SUMMARY="${GITHUB_STEP_SUMMARY:-/dev/null}" +ZERO_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000" + +ALL_IMAGES=(toolbox agent-runner conformance-runner) +SHA_TAG="sha-${HEAD_SHA:0:12}" +MOVING_TAG="main" + +log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; } +info() { printf ' %s\n' "$*"; } +die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; printf -- '- **ERROR**: %s\n' "$*" >>"${SUMMARY}"; exit 1; } + +case "${MODE}" in verify|publish) ;; *) die "MODE must be verify or publish, got '${MODE}'" ;; esac + +# --- inputs per image: a change under any of these rebuilds that image --------------------- +# conformance-runner takes its chainsaw/kubectl/fedora pins from infra/devcontainer/versions.env. +# conformance/versions.env is NOT an input (it only RECORDS the published digest), and must not +# become one: Renovate writes it after every publish, and an input there would loop. +inputs() { + case "$1" in + toolbox) echo "infra/devcontainer/Containerfile infra/devcontainer/versions.env infra/devcontainer/build-image.sh" ;; + agent-runner) echo "infra/agent/" ;; + conformance-runner) echo "conformance/incluster/Containerfile conformance/incluster/conformance-clone conformance/incluster/build-image.sh infra/devcontainer/versions.env" ;; + esac +} +PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh" + +declare -A WANT=() BUILT=() DIGEST=() SECS=() NOTE=() + +changed_any() { # -> 0 if BASE_SHA..HEAD_SHA touches any + # shellcheck disable=SC2086 + [ -n "$(git diff --name-only "${BASE_SHA}" "${HEAD_SHA}" -- "$@")" ] +} + +select_images() { + if [ "${IMAGES}" != "auto" ]; then + for i in ${IMAGES}; do + case " ${ALL_IMAGES[*]} " in *" ${i} "*) WANT[$i]=1; NOTE[$i]="requested" ;; *) die "unknown image '${i}'" ;; esac + done + return + fi + if [ -z "${BASE_SHA}" ] || [[ "${BASE_SHA}" =~ ^0+$ ]] || ! git cat-file -e "${BASE_SHA}^{commit}" 2>/dev/null; then + info "no usable diff base ('${BASE_SHA}'): building everything" + for i in "${ALL_IMAGES[@]}"; do WANT[$i]=1; NOTE[$i]="no diff base"; done + return + fi + info "diff base ${BASE_SHA} .. ${HEAD_SHA}" + local i + for i in "${ALL_IMAGES[@]}"; do + # shellcheck disable=SC2046 + if changed_any $(inputs "$i"); then WANT[$i]=1; NOTE[$i]="inputs changed"; fi + done + # shellcheck disable=SC2086 + if [ "${MODE}" = verify ] && changed_any ${PIPELINE_FILES}; then + for i in "${ALL_IMAGES[@]}"; do [ -n "${WANT[$i]:-}" ] || { WANT[$i]=1; NOTE[$i]="pipeline changed"; }; done + fi +} + +# --- registry helpers (publish mode only) ---------------------------------------------------- +REG_TOKEN="" +registry_token() { + [ -n "${REG_TOKEN}" ] && return 0 + REG_TOKEN="$(curl -fsS --config <(printf 'user = "%s:%s"\n' "${REGISTRY_USER}" "${REGISTRY_TOKEN}") \ + "https://${REGISTRY}/v2/token?service=container_registry&scope=*" | jq -r '.token // empty')" \ + || die "could not obtain a registry bearer token from https://${REGISTRY}/v2/token" + [ -n "${REG_TOKEN}" ] || die "registry returned no bearer token" +} +# Prints the manifest digest the REGISTRY serves for :; nothing if it is absent (404). +# Any other answer (401, 5xx, no digest header) is fatal: "cannot tell" must never be read as +# "not published", or the immutable-tag guard would be bypassed by an auth hiccup. +registry_digest() { + registry_token + local hdrs code digest + hdrs="$(curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "${REG_TOKEN}") \ + -H 'Accept: application/vnd.oci.image.manifest.v1+json' \ + -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \ + -H 'Accept: application/vnd.oci.image.index.v1+json' \ + -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \ + "https://${REGISTRY}/v2/${NAMESPACE}/$1/manifests/$2" | tr -d '\r')" \ + || die "registry unreachable while reading $1:$2" + code="$(awk 'NR==1{print $2}' <<<"${hdrs}")" + digest="$(awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' <<<"${hdrs}" | tail -1)" + case "${code}" in + 200) [[ "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]] || die "registry answered 200 for $1:$2 with no usable digest" + echo "${digest}" ;; + 404) ;; + *) die "registry answered HTTP ${code} for $1:$2 — cannot tell whether it exists" ;; + esac +} + +local_ref() { echo "${REGISTRY}/${NAMESPACE}/$1:ci-${HEAD_SHA:0:12}"; } + +build_one() { # [agent base] + local img="$1" base="${2:-}" t0 tag + tag="ci-${HEAD_SHA:0:12}" + t0=$(date +%s) + log "BUILD + VERIFY ${img}" + case "${img}" in + toolbox) + ENGINE="${ENGINE}" IMAGE="$(local_ref toolbox)" BUILD_REVISION="${HEAD_SHA}" \ + ./infra/devcontainer/build-image.sh ;; + agent-runner) + ENGINE="${ENGINE}" IMAGE_TAG="${tag}" AGENT_BASE_IMAGE="${base}" BUILD_REVISION="${HEAD_SHA}" \ + ./infra/agent/build-image.sh ;; + conformance-runner) + ENGINE="${ENGINE}" IMAGE_TAG="${tag}" BUILD_REVISION="${HEAD_SHA}" \ + ./conformance/incluster/build-image.sh ;; + esac + BUILT[$img]=1 + SECS[$img]=$(( $(date +%s) - t0 )) + info "${img}: built and verified in ${SECS[$img]}s" +} + +push_one() { # -> sets DIGEST[image] to the registry-returned digest + local img="$1" repo src existing pushed got t0 + repo="${REGISTRY}/${NAMESPACE}/${img}" + src="$(local_ref "${img}")" + t0=$(date +%s) + log "PUSH ${img} -> ${repo}:{${SHA_TAG},${MOVING_TAG}}" + existing="$(registry_digest "${img}" "${SHA_TAG}")" + if [ -n "${existing}" ]; then + # Immutable tag: never overwrite. A rebuild is not byte-reproducible, so re-pushing would + # silently change what `sha-` means after something may already have pinned it. + die "${repo}:${SHA_TAG} already exists (${existing}). Tags here are immutable; this run must not + replace it. (publish skips images that already exist at this commit BEFORE building them, + so reaching this line means the tag appeared mid-run.)" + fi + "${ENGINE}" tag "${src}" "${repo}:${SHA_TAG}" + "${ENGINE}" tag "${src}" "${repo}:${MOVING_TAG}" + pushed="$("${ENGINE}" push "${repo}:${SHA_TAG}" | tee /dev/stderr | grep -oE 'digest: sha256:[0-9a-f]{64}' | tail -1 | cut -d' ' -f2)" + "${ENGINE}" push "${repo}:${MOVING_TAG}" >/dev/null + # The AUTHORITATIVE digest is what the registry serves, read back — not what the client + # printed. They should agree; if they do not, nothing downstream may trust either. + got="$(registry_digest "${img}" "${SHA_TAG}")" + [ -n "${got}" ] || die "pushed ${repo}:${SHA_TAG} but the registry does not serve it" + [ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] \ + || die "${img}: client reported ${pushed} but the registry serves ${got}" + [ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \ + || die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})" + DIGEST[$img]="${got}" + SECS[$img]=$(( ${SECS[$img]:-0} + $(date +%s) - t0 )) + info "${img}: registry digest ${got}" +} + +# --- main --------------------------------------------------------------------------------- +log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}" +command -v "${ENGINE}" >/dev/null || die "${ENGINE} is not on PATH in the job container" +"${ENGINE}" info >/dev/null 2>&1 \ + || die "the ${ENGINE} daemon is not reachable from the job container — the runner did not + hand it a Docker socket (act_runner container.docker_host). Nothing can be built." +"${ENGINE}" version 2>/dev/null | sed -n '1,12p' || true +"${ENGINE}" system df || true + +if [ "${MODE}" = publish ]; then + : "${REGISTRY_TOKEN:?REGISTRY_TOKEN is required in publish mode}" + REGISTRY_USER="${REGISTRY_USER:-ci-image-publisher}" + printf '%s' "${REGISTRY_TOKEN}" | "${ENGINE}" login "${REGISTRY}" -u "${REGISTRY_USER}" --password-stdin >/dev/null \ + || die "docker login ${REGISTRY} failed — check REGISTRY_PUSH_TOKEN (scope write:package, owner has package write on the org)" +fi + +select_images + +# Publish idempotence: an image already published at THIS commit is not rebuilt. This is what +# makes a re-run after a half-finished publish safe — and it is the only way a rerun can +# finish, because the immutable sha tag must never be pushed twice. +if [ "${MODE}" = publish ]; then + for i in "${ALL_IMAGES[@]}"; do + [ -n "${WANT[$i]:-}" ] || continue + d="$(registry_digest "${i}" "${SHA_TAG}")" + if [ -n "${d}" ]; then + info "${i}: already published at ${SHA_TAG} (${d}); not rebuilding" + unset "WANT[$i]"; DIGEST[$i]="${d}"; NOTE[$i]="already published at this commit" + fi + done +fi + +# Decide the agent-runner base (see header). +AGENT_BASE="" +if [ -n "${WANT[agent-runner]:-}" ]; then + # shellcheck disable=SC1091 + PINNED_BASE="$(. ./infra/agent/versions.env; echo "${AGENT_BASE_IMAGE}")" + if [ "${MODE}" = verify ] && [ -z "${WANT[toolbox]:-}" ]; then + WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)" + elif [ "${MODE}" = publish ] && [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then + case "${PINNED_BASE}" in + *@"${ZERO_DIGEST}") WANT[toolbox]=1; NOTE[toolbox]="never published; required as agent-runner's base" ;; + *@sha256:*) AGENT_BASE="${PINNED_BASE}" ;; + *) die "infra/agent/versions.env AGENT_BASE_IMAGE='${PINNED_BASE}' is not digest-pinned; CI will not build on a floating base" ;; + esac + fi +fi + +yn() { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; } +for i in "${ALL_IMAGES[@]}"; do info "plan: ${i}: $(yn "${WANT[$i]:-}" BUILD skip) (${NOTE[$i]:-inputs unchanged})"; done + +if [ -n "${WANT[toolbox]:-}" ]; then + build_one toolbox + if [ "${MODE}" = publish ]; then push_one toolbox; fi +fi +if [ -n "${WANT[agent-runner]:-}" ]; then + if [ -z "${AGENT_BASE}" ]; then + if [ "${MODE}" = publish ]; then + AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}" + else + AGENT_BASE="$(local_ref toolbox)" + fi + fi + info "agent-runner base: ${AGENT_BASE}" + build_one agent-runner "${AGENT_BASE}" + if [ "${MODE}" = publish ]; then push_one agent-runner; fi +fi +if [ -n "${WANT[conformance-runner]:-}" ]; then + build_one conformance-runner + if [ "${MODE}" = publish ]; then push_one conformance-runner; fi +fi + +# --- summary ------------------------------------------------------------------------------ +{ + echo "## Images (${MODE})" + echo + echo "commit \`${HEAD_SHA}\`" + echo + echo "| image | built + verified | seconds | published digest (registry-returned) | note |" + echo "|---|---|---|---|---|" + for i in "${ALL_IMAGES[@]}"; do + d="${DIGEST[$i]:-}" + echo "| \`${i}\` | $(yn "${BUILT[$i]:-}" yes no) | ${SECS[$i]:--} | $(yn "${d}" "\`${REGISTRY}/${NAMESPACE}/${i}@${d}\`" -) | ${NOTE[$i]:-inputs unchanged} |" + done + echo + if [ "${MODE}" = publish ]; then + echo "Tags pushed: \`${SHA_TAG}\` (immutable) and \`${MOVING_TAG}\` (moving)." + echo "Nothing here writes to git. Renovate opens the digest-bump PR on its next run." + else + echo "Verify mode: nothing was pushed and no registry credential was present." + fi +} >>"${SUMMARY}" +log "done" diff --git a/conformance/incluster/README.md b/conformance/incluster/README.md index 5728e4b..3d306dc 100644 --- a/conformance/incluster/README.md +++ b/conformance/incluster/README.md @@ -61,10 +61,13 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02). > digest-pinned to the conformance-runner image, `command` exactly > `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and > `env`/`envFrom` were the same shape of hole and are closed in the same policy. -- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the - image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it - prints the exact `podman push --digestfile …` command for the operator instead. See - `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin. +- `build-image.sh` — builds the above and **verifies the toolchain by running it** inside the + image as uid 65532 with `--read-only`. It **never pushes**, and neither does anyone from a + workstation: CI (`.gitea/workflows/images.yaml`) runs this same script, and on a push to + `main` publishes the verified image under `sha-` and `main`. Renovate then moves + `CONFORMANCE_RUNNER_IMAGE_DIGEST` in `conformance/versions.env` onto the registry-returned + digest in a pull request (that file is a sensitive path, so the bump needs an + administrator's `/approve-override`). `--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the default verify stays offline so an air-gapped build is fully checked, not half-checked). diff --git a/conformance/incluster/build-image.sh b/conformance/incluster/build-image.sh index 54a8d28..c836484 100755 --- a/conformance/incluster/build-image.sh +++ b/conformance/incluster/build-image.sh @@ -1,37 +1,41 @@ #!/usr/bin/env bash ############################################################################### # conformance/incluster/build-image.sh — build the pinned `conformance-runner` -# image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1. +# image and prove it works. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1. # -# ./build-image.sh # build + verify + print the digest and push command +# ./build-image.sh # build + verify + print the image id # ./build-image.sh --no-verify # build only (skip the in-image tool proof) # ./build-image.sh --verify-clone # ALSO really clone this repo with conformance-clone. # # Opt-in because it needs the Gitea host reachable; the # # default verify is offline so an air-gapped build is # # still fully checked rather than half-checked. # ENGINE=docker ./build-image.sh # build with docker instead of podman +# IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this) # -# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is -# not an agent's to hold, and the pushed digest is what a fail-closed admission policy -# then trusts. So this builds locally, proves the toolchain works INSIDE the image as the -# non-root uid the policy demands, and prints the exact `podman push` line for a human. +# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either. +# Publishing is done by CI: .gitea/workflows/images.yaml calls this script, and on a push to +# `main` it pushes what this script built AND verified to the Gitea registry, records the +# registry-returned digest in the job summary, and Renovate then opens the pull request that +# moves conformance/versions.env onto that digest. So this script's job is to build and to +# prove the toolchain works INSIDE the image as the non-root uid the policy demands — the same +# proof locally and in CI. It runs under ENGINE=docker in CI (Docker-in-Docker), so nothing +# below may be podman-only. # # VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the # single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same # pins the toolbox gate runs with. Nothing in this file invents a version. # -# THE DIGEST CAVEAT — read it before pasting a digest into anything. -# The digest printed here is the LOCAL manifest digest. It is normally identical to the -# registry digest (podman pushes the manifest it already has rather than recompressing), -# but that is a property of the push, not a guarantee. So the push command below uses -# `--digestfile`: the digest the registry actually returns is the authoritative one, and -# that is the one that goes into conformance/versions.env. +# NO DIGEST IS PRINTED. A manifest digest does not exist until a registry holds the manifest, +# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at +# push time (.gitea/workflows/scripts/images.sh) and Renovate carries it into +# conformance/versions.env. ############################################################################### set -euo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_ROOT="$(cd "${HERE}/../.." && pwd)" ENGINE="${ENGINE:-podman}" +TAG_OVERRIDE="${IMAGE_TAG:-}" VERIFY=1 VERIFY_CLONE=0 @@ -64,9 +68,9 @@ source "${TOOLCHAIN_ENV}" # gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path # is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here. IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner" -# The tag is build convenience only (a human-readable handle for the push). Admission -# ignores it: only the digest is ever referenced from a Job. -IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}" +# The tag is a local handle only. CI re-tags what it publishes (git SHA + `main`), and +# admission ignores tags entirely: only the digest is ever referenced from a Job. +IMAGE_TAG="${TAG_OVERRIDE:-chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}}" IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}" log "Building ${IMAGE_REF}" @@ -76,7 +80,11 @@ info " CHAINSAW ${CHAINSAW_VERSION}" info " KUBECTL ${KUBECTL_VERSION}" info " FEDORA ${FEDORA_VERSION}" -"${ENGINE}" build \ +# Provenance: CI passes the commit it is building, so a pulled image names its source tree. +LABEL_ARGS=() +[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}") + +"${ENGINE}" build "${LABEL_ARGS[@]}" \ --file "${HERE}/Containerfile" \ --tag "${IMAGE_REF}" \ --build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \ @@ -184,46 +192,12 @@ if [ "${VERIFY_CLONE}" -eq 1 ]; then log "conformance-clone verified against the real repository" fi -# --- digest ------------------------------------------------------------------ -DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")" +# --- result ------------------------------------------------------------------ IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")" SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")" log "Built ${IMAGE_REF}" -info "local manifest digest ${DIGEST}" -info "image id ${IMAGE_ID}" -info "size $(( SIZE / 1024 / 1024 )) MiB" - -cat < sha256:.... THIS is the authoritative digest, not the local one above. - - # conformance/versions.env - CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)" - -WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml -is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not -${IMAGE_REPO}@sha256:. A tag reference is denied, -and so is any other repository path. - -TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard -\`${IMAGE_REPO}@sha256:*\` to be replaced with the -LITERAL digest once the image exists, closing the "anyone who can push to this path" gap. -That is a separate, human-reviewed PR against the security boundary. -================================================================================ -EOF +info "image id ${IMAGE_ID}" +info "size $(( SIZE / 1024 / 1024 )) MiB" +info "Not pushed. CI publishes this image from main (.gitea/workflows/images.yaml) and" +info "Renovate moves conformance/versions.env onto the registry-returned digest." diff --git a/conformance/tests/l0-static/alert-rules-unit-tested/check.sh b/conformance/tests/l0-static/alert-rules-unit-tested/check.sh index eb31f03..ae7c593 100755 --- a/conformance/tests/l0-static/alert-rules-unit-tested/check.sh +++ b/conformance/tests/l0-static/alert-rules-unit-tested/check.sh @@ -29,7 +29,8 @@ # THEN (1) promtool is a PINNED member of the toolchain: PROMTOOL_VERSION is declared in BOTH # infra/devcontainer/versions.env and ci/versions.env and the two AGREE (the repo's # lockstep rule), the Containerfile takes it as a build ARG, installs it from the -# pinned prometheus release and records it for provenance, dev.sh passes it, and the +# pinned prometheus release and records it for provenance, build-image.sh (the one +# recipe `dev.sh build` and CI both use) passes it, and the # CI runner installs it for the l0 stage; # (2) the promtool ACTUALLY IN USE reports exactly that pinned version — a unit test is # evidence only if the engine matches the one prod evaluates rules with; @@ -246,10 +247,12 @@ check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text, "infra/devcontainer/Containerfile does not record promtool in /etc/dev-toolchain-versions — " "that provenance line is how an operator tells a stale image from a current one") -DEVSH = ROOT / "infra/devcontainer/dev.sh" -check(DEVSH.is_file() and "--build-arg PROMTOOL_VERSION=" in DEVSH.read_text(), - "infra/devcontainer/dev.sh does not pass --build-arg PROMTOOL_VERSION — the ARG would be " - "empty and the build would fetch a bogus URL") +# The build recipe moved from dev.sh into build-image.sh on 2026-10-29, so that `dev.sh build` +# and the CI image pipeline (.gitea/workflows/images.yaml) share ONE recipe. Check the recipe. +BUILDSH = ROOT / "infra/devcontainer/build-image.sh" +check(BUILDSH.is_file() and "--build-arg PROMTOOL_VERSION=" in BUILDSH.read_text(), + "infra/devcontainer/build-image.sh does not pass --build-arg PROMTOOL_VERSION — the ARG " + "would be empty and the build would fetch a bogus URL") INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh" if INSTALL.is_file(): diff --git a/conformance/versions.env b/conformance/versions.env index 731d469..8325387 100644 --- a/conformance/versions.env +++ b/conformance/versions.env @@ -40,32 +40,29 @@ PYTHON_MIN="3.9" # gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:* # Any other path, and any TAG reference, is DENIED at admission. Do not "tidy" these values. CONFORMANCE_RUNNER_IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner" -# Build handle only. Admission never sees the tag; a Job must reference the digest. +# Local build handle only. Admission never sees the tag; a Job must reference the digest. +# CI publishes under `sha-` and `main` instead (.gitea/workflows/images.yaml). CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44" # # ############################################################################ -# # DIGEST DELIBERATELY EMPTY — THE IMAGE HAS NOT BEEN PUBLISHED YET. # +# # THE DIGEST IS WRITTEN BY RENOVATE, NOT BY HAND. # # ############################################################################ -# The image has been BUILT and its toolchain verified locally (non-root uid 65532, -# read-only rootfs: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4, chainsaw 0.2.12, -# and `./run.sh --layer l2 --no-cluster` completing with rc 0). It has NOT been pushed: -# pushing needs a registry credential, and publishing the artifact a fail-closed security -# boundary then trusts is an OPERATOR action, not an agent's. +# CI builds and verifies the image (non-root uid 65532, read-only rootfs, the +# conformance-clone refusals) and, on a push to main, publishes it and prints the digest the +# REGISTRY returned (.gitea/workflows/images.yaml). Renovate follows the image's `main` tag +# through the marker below and opens the pull request that moves this value. Nobody pushes +# this image from a workstation, and nobody pastes a digest here. # -# OPERATOR, to complete the pin: -# 1. ./conformance/incluster/build-image.sh # rebuild, verify, print the commands -# 2. podman login gitea.home.keithjackson.dev -# 3. podman push --digestfile /tmp/conformance-runner.digest \ -# "${CONFORMANCE_RUNNER_IMAGE_REPO}:${CONFORMANCE_RUNNER_IMAGE_TAG}" -# 4. Set CONFORMANCE_RUNNER_IMAGE_DIGEST below to `cat /tmp/conformance-runner.digest` -# — the digest the REGISTRY returned, not the local manifest digest the build printed. -# 5. Land steps 4 in a PR (this file is GitOps input, not a live edit). -# -# UNTIL THEN: this is empty on purpose, NOT a placeholder to be guessed at. An empty value -# makes any consumer that renders the Job fail visibly; a fake or stale digest would instead -# produce an ImagePullBackOff that looks like a cluster fault. Do not invent one. -CONFORMANCE_RUNNER_IMAGE_DIGEST="" -CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_DIGEST:+${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}}" +# ALL-ZERO UNTIL THE FIRST PUBLISH. Before 2026-10-29 this was EMPTY, so that a consumer +# rendering the Job would fail visibly. It is now the same impossible all-zero digest +# platform/pm-agent/ uses, because Renovate can only replace a digest that is there to match. +# A consumer MUST treat the all-zero value as "not published" (no consumer exists yet: the +# L2/L3 trigger is unwired, docs/plans/l2-l3-conformance-ci-wiring.md). Note also that +# chainsaw-job.yaml's @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ takes the 64 hex digits WITHOUT the +# `sha256:` prefix this value carries — the trigger must strip it. +# renovate-digest: image=gitea.home.keithjackson.dev/home-network/conformance-runner tag=main +CONFORMANCE_RUNNER_IMAGE_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000" +CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}" # # AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for # its `@sha256:*` wildcard to be replaced with the LITERAL digest once the image exists — diff --git a/decisions/2026-10-29-image-pipeline.md b/decisions/2026-10-29-image-pipeline.md new file mode 100644 index 0000000..7fa2979 --- /dev/null +++ b/decisions/2026-10-29-image-pipeline.md @@ -0,0 +1,222 @@ +--- +date: 2026-10-29 +branch: feat/image-pipeline +title: "Images are built, verified and published by CI; Renovate carries the digests into git" +--- + +## What changed + +Operator rule (2026-10-29): container image creation, hosting and lifecycle belong to CI/CD and +the codebase, never to a one-off workstation build/push. Before this branch the Home-Network org's +package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and +both `build-image.sh` scripts said "publishing is an operator action". + +- **`.gitea/workflows/images.yaml`** + **`.gitea/workflows/scripts/images.sh`** (the logic). + - Pull request: build and verify only. No push, and the step references no secret at all. + - Push to `main`, or `workflow_dispatch` on `main`: build, verify, then push to + `gitea.home.keithjackson.dev/home-network/{toolbox,agent-runner,conformance-runner}`. + Tags are `sha-<12 hex>` (immutable: the driver refuses to overwrite one) and `main` (moving). + The digest is read back from the registry with a HEAD on the manifest, compared with what the + client printed, and written to the job summary. + - Only images whose inputs changed are built. A re-run on the same commit skips images that are + already published. + - agent-runner builds FROM the toolbox built in the same run (on main, FROM its just-pushed + digest). Otherwise: + - in verify mode the toolbox is built anyway as its base, because a PR holds no credential; + - in publish mode it uses `AGENT_BASE_IMAGE` from `infra/agent/versions.env`; + - if that is still the zero placeholder, the toolbox is built and published first. This is + how the first run on main bootstraps. + - An always-run step prunes the DinD image store. +- **One build recipe per image, reused by CI.** + - New `infra/devcontainer/build-image.sh`. `dev.sh build` now delegates to it. It has an + in-image verification step that the toolbox never had: every tool is on PATH, and + promtool, kubectl, chainsaw, kyverno and sops report their pins. + - `infra/agent/build-image.sh` and `conformance/incluster/build-image.sh` were made + engine-neutral, because CI runs `ENGINE=docker` against DinD: + - `podman image exists` became `image inspect`, with a pull fallback; + - `{{.Digest}}` and `--digestfile` are gone. No local digest is printed, since only the + registry's digest is meaningful; + - `probe-validation.sh` is fed over **stdin** instead of a bind mount. Under DinD a bind + source is resolved on the daemon's filesystem, which would silently mount an empty + directory; + - `probe-validation.sh` itself gained `${BASH_SOURCE[0]:-$0}` so it survives `bash -s` + under `set -u`. + - `AGENT_BASE_IMAGE` and `IMAGE_TAG` can be overridden from the environment, and + `BUILD_REVISION` adds an `org.opencontainers.image.revision` label. + - The "publishing is an operator action" text was replaced throughout. +- **Digest references.** Each now carries a `# renovate-digest: image= tag=main` marker + on the line above it: + - `platform/pm-agent/deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`), which share + one dependency, so they bump in one PR; + - `infra/agent/versions.env` `AGENT_BASE_IMAGE`, which was `home-k3s-dev:local` and is now + the published toolbox by digest (zero placeholder until the first publish); + - `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to the + zero placeholder because Renovate can only replace a digest it can match. + + The admission policies are **untouched**; the `@sha256:` form is preserved. +- **`renovate.json5`**: + - a new `custom.regex` manager for the markers; + - `enabledManagers` `regex` → `custom.regex`; + - a packageRule that disables the `kubernetes` manager for these three repos. Otherwise it + would also see the digest-only lines and look them up as `latest`; + - an `own-image` label; + - `ignorePaths` narrowed from `conformance/**` to its subdirectories. `ignorePaths` applies + to custom managers too, so the blanket glob hid `conformance/versions.env`. +- **`renovate.yaml`**: `renovate-version: 44.167.0` pinned, plus a `RENOVATE_HOST_RULES` docker + rule for the Gitea registry, using the same `RENOVATE_TOKEN`, which now also needs + `read:package`. See below for why the version pin is a real fix and not tidiness. +- **`infra/gitea/registry-retention.sh`**: dry run by default, `--apply` to delete. This is an + operator action. Per image it keeps: + - `main` and `latest`; + - the 10 newest `sha-` tags; + - every tag whose digest is pinned anywhere in `origin/main`; + - every digest-named version. +- **`infra/gitea/conformance-trigger-kubeconfig.sh`**: the "runner image is digest-pinned" + precondition grepped the literal line `CONFORMANCE_RUNNER_IMAGE="${…_REPO}@${…_DIGEST}"` + for 64 hex digits, so it **could never pass**. It now sources the file, evaluates the value, + and rejects the zero placeholder. +- **`conformance/tests/l0-static/alert-rules-unit-tested/check.sh`**: its "the toolbox build + passes `--build-arg PROMTOOL_VERSION`" assertion now reads `infra/devcontainer/build-image.sh` + instead of `dev.sh`. The gate caught this on the first run, because the recipe had moved. +- **Docs**: `platform/pm-agent/README.md`, `runbooks/pm-agent.md` (two rows), + `docs/plans/pm-agent-issue-to-pr.md` (follow-up 1), `conformance/incluster/README.md`, + `infra/devcontainer/README.md`, and a new "How images are built" section in + `.gitea/workflows/README.md`. + +### Decisions and pushback + +- **The push credential is a PAT secret, `REGISTRY_PUSH_TOKEN`; the Actions token cannot push.** + Checked against Gitea **v1.24.3** source: + - `services/auth/basic.go` `Basic.Verify` turns a task token into the pseudo-user + `gitea-actions` (ID −2) and sets no token scope; + - `routers/api/packages/container/container.go` `Authenticate` (the `/v2/token` exchange every + docker client performs) requires `read:package`/`write:package`/`all` scope, so it answers + 401. The Actions token cannot even *pull*; + - `services/context/package.go` `determineAccessMode` has `// TODO: ActionUser permission check` + and no branch for that user. Even past login it would get at most read on a public org; + - `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3. + + The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight + pattern. +- **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate: + - `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` → + `custom.regex`; + - that migration runs on every repository run (`lib/workers/repository/init/merge.ts` → + `migrateAndValidate`), not only in the validator; + - `lib/modules/manager/index.ts` `getEnabledManagersList` accepts either spelling; + - `custom.regex` exists since 37.51.1 (renovatebot/renovate#24079). +- **The open question hid a bigger bug: the Renovate config on main is invalid for the Renovate + the workflow actually runs.** + - `renovatebot/github-action@v41.0.9` defaults `renovate-version` to `39` + (`action.yml`, `src/docker.ts`), which is 39.264.0. + - The validator used in #79 was `ghcr.io/renovatebot/renovate` with no tag, i.e. `latest`, + 44.167.0. + - **Measured:** `renovate:39.264.0 renovate-config-validator` on main's `renovate.json5` gives + `ERROR: Found errors in configuration` with five errors, all `managerFilePatterns`. + `fileMatch` was renamed in 40.2.0. + - So once #80 fixes the filename, every run would still die at `CONFIG_VALIDATION`. + - Fixed by pinning `renovate-version: 44.167.0`. This branch's config validates clean on it. + The only `--strict` complaint is a pending migration of the pre-existing top-level + `"docker": {"enabled": true}` block, which I left alone. +- **Unworkable as briefed: tracking the worker Job template's digest.** + - `infra/agent/worker-job-template.yaml` is `COPY`ed into agent-runner. A Renovate bump there + changes agent-runner's inputs, CI publishes a new digest, Renovate bumps it again, and so on + forever. + - `pm-dispatch` overwrites that field from `PM_WORKER_IMAGE` at runtime + (`pm-dispatch.py:649`), so nothing is lost by leaving it all-zero. It is commented as + deliberately untracked. + - For the same loop reason, `conformance/versions.env` is **not** an input of the + conformance-runner build, and a pipeline-only change does not republish on main. +- **Changed from the brief: retention is a script, not a Gitea cleanup rule.** + - Gitea 1.24.3 has **no API** for cleanup rules (none in `routers/api/v1/api.go`; web + session+CSRF forms only, in `routers/web/web.go`). So a rule cannot be created, or + drift-asserted, as code. **`ci/gitea-settings.sh` was therefore not extended.** A cheaper + proxy (count versions per image) would need `read:package` on `SETTINGS_API_TOKEN`, which + it lacks. + - A built-in rule is also **unsafe here**. Gitea stores a manifest as a file of the TAG + version it was pushed under (`routers/api/packages/container/manifest.go` + `createPackageAndVersion`), and resolves a by-digest pull from those files + (`models/packages/container/search.go`). Deleting the tag a deployed digest was pushed under + breaks the `@sha256:` pull, and the rule's "newest N" knows nothing about what git pins. The + script keeps every digest git pins. +- **Auto-merge.** Digest bumps of these images auto-merge on a green gate (the existing + `digest` rule). The exception is the conformance-runner bump: `conformance/versions.env` is a + sensitive path, so it waits for an administrator's `/approve-override` every time. + +## What I verified + +- **Local gate, pinned toolbox, from this worktree, committed HEAD.** Command: + `ENGINE= DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`. + The shim adds the `.git` mount. Result: see the PR body for the rc and stage list. They are + recorded there rather than here because this entry is part of the tree being gated. +- **Local podman builds through the CI driver**, run as + `MODE=verify ENGINE=podman BASE_SHA=$(git merge-base origin/main HEAD) .gitea/workflows/scripts/images.sh`. + It returned rc 0 and built and verified all three images: + - toolbox: every tool on PATH, and the five pins matched; + - agent-runner: `PM_SELFTEST=1 pm-dispatch` OK, and the stdin-fed `probe-validation.sh` + gave "20 refusal(s), 0 failure(s)"; + - conformance-runner: all seven `conformance-clone` refusals. +- **Other driver paths, run locally:** + - agent-runner alone builds the toolbox as its base; + - an empty diff builds nothing; + - publish mode without a token dies before doing anything; + - `infra/agent/build-image.sh` with the zero placeholder dies with build-locally + instructions. +- **Renovate, with the pinned 44.167.0 image, locally:** + - `renovate-config-validator` exits 0; + - a `--platform=local` `extract` dry run finds all four marker references (custom regex manager, + `currentValue: main`, `currentDigest: sha256:000…`); + - a `lookup` dry run shows the `kubernetes`-manager duplicates as `skipReason: disabled` and the + marker deps taking the "unsupported/unversioned value main → getDigest(main)" path. That + lookup currently 404s because nothing is published yet. +- **Renovate 39.264.0 rejects main's current `renovate.json5`.** Validator output reproduced + above. +- **`registry-retention.sh` dry run against the live Gitea (read-only):** zero versions per image, + nothing to delete. The token exchange and the package listing both answered. +- **Live, read-only measurements** (kubectl and the Gitea API): + - the runner pod is on k3s-prod-1, which had **7 GB** nodefs available; + - DinD holds only the 1.54 GB act image; buildx is v0.20.1; the daemon is 27.5.1 on overlay2 + with cgroup v2; + - Gitea `/data` is 9.7G, 1% used; + - the Gateway's Envoy (Cilium) has `streamIdleTimeout: 300s`, `maxStreamDuration: 0s`, and + **no route `timeout`** on the gitea route; + - Gitea's `LIMIT_SIZE_CONTAINER` is unset (default −1 = unlimited, `modules/setting/packages.go`). + +## What I did NOT verify + +- **Nothing has been pushed to the registry, by me or by CI.** The publish path has never run: + - the registry HEAD/digest read-back; + - the immutable-tag guard; + - "already published at this commit" idempotence; + - agent-runner `FROM @sha256:`; + - `docker login` with a PAT. + + The first merge to main is the first test, and it needs `REGISTRY_PUSH_TOKEN` to exist first. +- **Large-layer push through the Gateway is UNVERIFIED.** Envoy has no request-body cap and no + stream duration cap here. But with no route `timeout` set, Envoy's **default 15 s upstream + timeout** applies from the end of the request to the end of the response. That covers Gitea + finalising a blob: copying the upload into package storage on Longhorn. A ~250 MB compressed + layer should finish well inside 15 s, but I could not measure it without pushing. If the first + publish shows 504s on blob PUTs, set `timeouts.request` on the gitea HTTPRoute. +- **DinD fit is estimated, not measured** (until the PR run reports): + - memory: 2Gi is shared by the job container and every build container; + - disk: the toolbox, agent-runner, conformance-runner, Fedora base and build cache should + total roughly 3 GB on top of the 1.5 GB act image, against 7 GB free on the node, with + `dind-storage` having no sizeLimit. +- **That act_runner hands the job container a Docker socket is inferred** from act_runner's + DOCKER_HOST handling, not measured. The driver fails loudly if it does not. +- **Renovate has still never completed a run here.** Three things block it: + - #80, the `configurationFile` name; + - `RENOVATE_TOKEN` gaining `read:package`; + - this branch's version pin. + + `renovatebot/github-action@v41.0.9` running a 44.x image is untested. The action only passes + the tag to `docker run`. +- **Pods cannot pull these images yet regardless.** The kubelet pull credential is PR #61 + (`pm-agent-registry` dockerconfigjson, `read:package`), still open. Until it merges, a correct + digest still ends in ImagePullBackOff (401), as before. +- **`registry-retention.sh --apply` has never run**, and its keep logic has only run against an + empty registry. +- **`git diff` change detection on Gitea `push` events relies on `github.event.before`.** A + force-push or a first push falls back to building everything, which is intended but not + observed. diff --git a/docs/plans/pm-agent-issue-to-pr.md b/docs/plans/pm-agent-issue-to-pr.md index ad87805..9f18152 100644 --- a/docs/plans/pm-agent-issue-to-pr.md +++ b/docs/plans/pm-agent-issue-to-pr.md @@ -244,10 +244,12 @@ about runtime behaviour in this document is a design claim, not a measurement. (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with the credential decision now being local Gitea accounts with PATs, that means a `dockerconfigjson` for an account whose token carries `read:package`. Only then does the - digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both - placeholder digests. Expect the build itself to need iteration too: the nodejs module - stream and the npm install have never run against this base. The same two blockers sit in - front of `conformance-runner`, which shares the registry path and has never been pulled. + digest matter. **Superseded 2026-10-29:** the build-and-push is no longer an operator step. + `.gitea/workflows/images.yaml` builds and verifies the image with + `infra/agent/build-image.sh`, publishes it on a push to `main`, and Renovate opens the one + PR that replaces both placeholder digests (`decisions/2026-10-29-image-pipeline.md`). The + same two blockers sit in front of `conformance-runner`, which shares the registry path and + has never been pulled. 2. **Create the credentials** per `pm-agent-credentials.sops.yaml.example`, and **verify the bot's permission is `write`, not `admin`** — that measurement is the control that keeps the agent out of its own approval gate. diff --git a/infra/agent/build-image.sh b/infra/agent/build-image.sh index eb65585..e854f4e 100755 --- a/infra/agent/build-image.sh +++ b/infra/agent/build-image.sh @@ -1,26 +1,36 @@ #!/usr/bin/env bash ############################################################################### -# infra/agent/build-image.sh — build the pinned `agent-runner` image and print its digest. +# infra/agent/build-image.sh — build the pinned `agent-runner` image and prove it works. # -# ./build-image.sh # build + verify + print the digest and push command +# ./build-image.sh # build + verify + print the image id # ./build-image.sh --no-verify # build only (skip the in-image proof) # ENGINE=docker ./build-image.sh # build with docker instead of podman +# AGENT_BASE_IMAGE=home-k3s-dev:local ./build-image.sh +# # build on a LOCAL toolbox instead of the pinned one +# # (`infra/devcontainer/dev.sh build` makes that tag) +# IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this) # -# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is not -# an agent's to hold, and the pushed digest is what a fail-closed admission policy then -# trusts. So this builds locally, proves the toolchain and both entrypoints work INSIDE the -# image as the non-root uid with the read-only root filesystem the policy forces, and prints -# the exact push line for a human. +# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either. +# Publishing is done by CI: .gitea/workflows/images.yaml calls this script, and on a push to +# `main` it pushes what this script built AND verified to the Gitea registry, records the +# registry-returned digest in the job summary, and Renovate then opens the pull request that +# moves platform/pm-agent/ onto that digest. So this script's job is to build, and to prove +# the toolchain and both entrypoints work INSIDE the image as the non-root uid with the +# read-only root filesystem the admission policy forces — the same proof locally and in CI. # -# VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the four pins this image -# adds (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds -# the platform toolchain the base already carries. Nothing in this file invents a version. +# VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the pins this image adds +# (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds the +# platform toolchain the base already carries. Nothing in this file invents a version. # -# THE DIGEST CAVEAT — read it before pasting a digest into anything. The digest printed here -# is the LOCAL manifest digest. It is normally identical to the registry digest, but that is -# a property of the push rather than a guarantee, so the push command below uses -# `--digestfile`: the digest the REGISTRY returns is the authoritative one, and that is the -# one that goes into platform/pm-agent/configmap.yaml. +# THE BASE. AGENT_BASE_IMAGE in versions.env is the PUBLISHED toolbox, pinned by digest and +# kept current by Renovate. An AGENT_BASE_IMAGE in the environment overrides it — CI passes +# the toolbox it built in the same run, and a developer can pass `home-k3s-dev:local`. +# +# ENGINE PORTABILITY. CI runs this with ENGINE=docker against a Docker-in-Docker daemon, so +# nothing below may be podman-only (`image exists`, `--digestfile`, `{{.Digest}}`), and +# nothing may BIND-MOUNT a file from this checkout into a container: under DinD the daemon +# resolves bind sources on ITS filesystem, where this checkout does not exist, and silently +# mounts an empty directory instead. Files go in over stdin. ############################################################################### set -euo pipefail @@ -46,8 +56,11 @@ command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGI # --- pins: sourced, never invented ------------------------------------------- AGENT_ENV="${HERE}/versions.env" [ -f "${AGENT_ENV}" ] || die "missing ${AGENT_ENV} — the pins live there, not here" +BASE_OVERRIDE="${AGENT_BASE_IMAGE:-}" +TAG_OVERRIDE="${IMAGE_TAG:-}" # shellcheck disable=SC1090 source "${AGENT_ENV}" +[ -z "${BASE_OVERRIDE}" ] || AGENT_BASE_IMAGE="${BASE_OVERRIDE}" : "${AGENT_BASE_IMAGE:?AGENT_BASE_IMAGE not set in ${AGENT_ENV}}" : "${NODEJS_MAJOR:?NODEJS_MAJOR not set in ${AGENT_ENV}}" : "${CLAUDE_CODE_VERSION:?CLAUDE_CODE_VERSION not set in ${AGENT_ENV}}" @@ -55,16 +68,28 @@ source "${AGENT_ENV}" : "${AGENT_MODEL:?AGENT_MODEL not set in ${AGENT_ENV}}" : "${AGENT_IMAGE_REPO:?AGENT_IMAGE_REPO not set in ${AGENT_ENV}}" -# --- the base must already exist --------------------------------------------- -# Deliberately NOT built on demand. The toolbox build is its own reviewed artefact with its -# own pins, and silently building it here would mean this script could change the toolchain -# a worker gates with as a side effect of rebuilding the agent image. -if ! "${ENGINE}" image exists "${AGENT_BASE_IMAGE}" 2>/dev/null; then - die "base image '${AGENT_BASE_IMAGE}' is not present. Build the pinned toolbox first: - ${REPO_ROOT}/infra/devcontainer/dev.sh build" +# --- the base must exist, locally or in the registry --------------------------- +# Deliberately NOT built on demand. The toolbox is its own reviewed artefact with its own +# pins, and silently building it here would mean this script could change the toolchain a +# worker gates with as a side effect of rebuilding the agent image. +case "${AGENT_BASE_IMAGE}" in + *@sha256:0000000000000000000000000000000000000000000000000000000000000000) + die "AGENT_BASE_IMAGE is still the all-zero placeholder in ${AGENT_ENV}: the toolbox has + not been published yet. Either let CI publish it (.gitea/workflows/images.yaml) and + merge Renovate's digest bump, or build on a local toolbox: + ${REPO_ROOT}/infra/devcontainer/dev.sh build + AGENT_BASE_IMAGE=home-k3s-dev:local $0" ;; +esac +if ! "${ENGINE}" image inspect "${AGENT_BASE_IMAGE}" >/dev/null 2>&1; then + info "base ${AGENT_BASE_IMAGE} is not local; pulling it" + "${ENGINE}" pull "${AGENT_BASE_IMAGE}" >/dev/null \ + || die "base image '${AGENT_BASE_IMAGE}' is neither local nor pullable (registry login?). + To build on a local toolbox instead: + ${REPO_ROOT}/infra/devcontainer/dev.sh build + AGENT_BASE_IMAGE=home-k3s-dev:local $0" fi -IMAGE_TAG="claude${CLAUDE_CODE_VERSION}-node${NODEJS_MAJOR}" +IMAGE_TAG="${TAG_OVERRIDE:-claude${CLAUDE_CODE_VERSION}-node${NODEJS_MAJOR}}" IMAGE_REF="${AGENT_IMAGE_REPO}:${IMAGE_TAG}" log "Building ${IMAGE_REF}" @@ -75,7 +100,11 @@ info " node ${NODEJS_MAJOR}" info " claude ${CLAUDE_CODE_VERSION} (floor ${CLAUDE_CODE_MIN_VERSION})" info " model ${AGENT_MODEL}" -"${ENGINE}" build \ +# Provenance: CI passes the commit it is building, so a pulled image names its source tree. +LABEL_ARGS=() +[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}") + +"${ENGINE}" build "${LABEL_ARGS[@]}" \ --file "${HERE}/Containerfile" \ --tag "${IMAGE_REF}" \ --build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}" \ @@ -138,63 +167,32 @@ if [ "${VERIFY}" -eq 1 ]; then # that will actually run in the cluster. The probe is mounted read-only rather than baked # in: it is a build-time check, not something a worker Job has any business carrying. log "Probing pm-worker's input validation inside the image" - "${ENGINE}" run --rm \ + # + # The probe goes in over STDIN, not as a bind mount: under Docker-in-Docker (CI) a bind + # source is resolved on the daemon's filesystem, where this file does not exist, and the + # container would get an empty directory. PM_WORKER_PATH points it at the baked worker, + # since the probe's own default (a sibling of the probe file) has no meaning on stdin. + "${ENGINE}" run --rm -i \ --user 65532:65532 \ --read-only \ --tmpfs /workspace:rw,mode=1777 \ --tmpfs /tmp:rw,mode=1777 \ --network none \ - --volume "${HERE}/probe-validation.sh:/probe-validation.sh:ro,Z" \ + -e PM_WORKER_PATH=/usr/local/bin/pm-worker \ --entrypoint "" \ "${IMAGE_REF}" \ - /bin/bash /probe-validation.sh + /bin/bash -s < "${HERE}/probe-validation.sh" log "Image verified" fi -# --- digest ------------------------------------------------------------------ -DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")" +# --- result ------------------------------------------------------------------ +# No digest is printed: a manifest digest does not exist until a registry has the manifest, +# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at +# push time and prints it in the job summary (.gitea/workflows/scripts/images.sh). SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")" - log "Built ${IMAGE_REF}" -info "local manifest digest ${DIGEST}" -info "size $(( SIZE / 1024 / 1024 )) MiB" - -cat < sha256:.... THIS is the authoritative digest, not the local one above. - - # platform/pm-agent/configmap.yaml - PM_WORKER_IMAGE: "${AGENT_IMAGE_REPO}@\$(cat /tmp/agent-runner.digest)" - -WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/pm-agent-job-shape.yaml is a -fail-closed Enforce policy. It DENIES any worker Job whose image is not -${AGENT_IMAGE_REPO}@sha256: — a tag reference is denied, -and so is any other repository path. The dispatcher also refuses to start if -PM_WORKER_IMAGE carries no digest, so the failure names its own cause. - -THE SAME IMAGE RUNS THE DISPATCHER. platform/pm-agent/deployment.yaml pins it too, so a -new digest is a two-line change in one pull request: the Deployment's image and -PM_WORKER_IMAGE. They must be the same digest — the dispatcher and its workers share the -committed Job template and the env-name contract, and a mismatched pair would disagree -about it. - -TIGHTEN THE BOUNDARY AFTERWARDS: the policy admits the wildcard -\`${AGENT_IMAGE_REPO}@sha256:*\`, so anyone who can push to that registry path can publish -a different digest. Replacing the wildcard with the LITERAL digest closes that gap; it is a -one-line change, reviewed in a pull request, which is exactly where a change to the -security boundary belongs. -================================================================================ -EOF +info "image id $("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")" +info "size $(( SIZE / 1024 / 1024 )) MiB" +info "Not pushed. CI publishes this image from main (.gitea/workflows/images.yaml); see" +info "platform/pm-agent/README.md for how a new digest reaches the cluster." diff --git a/infra/agent/probe-validation.sh b/infra/agent/probe-validation.sh index 042d004..d888700 100755 --- a/infra/agent/probe-validation.sh +++ b/infra/agent/probe-validation.sh @@ -22,7 +22,9 @@ ############################################################################### set -uo pipefail -HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# `:-$0` because build-image.sh feeds this script over STDIN (`bash -s`), where BASH_SOURCE is +# empty and `set -u` would abort on it; PM_WORKER_PATH is always set in that case. +HERE="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)" WORKER="${PM_WORKER_PATH:-${HERE}/pm-worker}" [ -r "$WORKER" ] || { echo "cannot read ${WORKER}" >&2; exit 2; } diff --git a/infra/agent/versions.env b/infra/agent/versions.env index bd0a136..676d66c 100644 --- a/infra/agent/versions.env +++ b/infra/agent/versions.env @@ -9,10 +9,16 @@ # truth and, the first time the two drifted, the worker's "green" would stop meaning the # gate's green. So this file pins ONLY what the toolbox does not carry. -# The toolbox image the runner is layered on. Built by `infra/devcontainer/dev.sh build` -# (tag `home-k3s-dev:local`); infra/agent/build-image.sh asserts it exists rather than -# silently building a different base. -AGENT_BASE_IMAGE=home-k3s-dev:local +# The toolbox image the runner is layered on: the PUBLISHED toolbox, by digest. CI builds and +# publishes it (.gitea/workflows/images.yaml) and Renovate moves this line onto each new digest +# of its `main` tag via the marker below, in a pull request — nobody edits it by hand. +# +# ALL-ZERO UNTIL THE FIRST PUBLISH. The first run of the image pipeline on main sees this +# placeholder, builds and publishes the toolbox first, and builds agent-runner FROM that fresh +# digest. For a LOCAL build on a local toolbox, override it: +# infra/devcontainer/dev.sh build && AGENT_BASE_IMAGE=home-k3s-dev:local infra/agent/build-image.sh +# renovate-digest: image=gitea.home.keithjackson.dev/home-network/toolbox tag=main +AGENT_BASE_IMAGE=gitea.home.keithjackson.dev/home-network/toolbox@sha256:0000000000000000000000000000000000000000000000000000000000000000 # Node.js — the runtime for the Claude Code CLI, installed from the Fedora base's module # stream. A major version, not a patch: the CLI supports the active LTS line and the diff --git a/infra/agent/worker-job-template.yaml b/infra/agent/worker-job-template.yaml index 6687282..62d043b 100644 --- a/infra/agent/worker-job-template.yaml +++ b/infra/agent/worker-job-template.yaml @@ -80,6 +80,12 @@ spec: # Replaced by the dispatcher with the digest-pinned image. A tag here would be # DENIED by admission; the dispatcher refuses a non-digest PM_WORKER_IMAGE at # startup so the error names the real cause. + # + # DELIBERATELY ALL-ZERO, AND DELIBERATELY NOT TRACKED BY RENOVATE. This file is + # baked INTO agent-runner, so it cannot carry agent-runner's own digest: bumping it + # would change the image's inputs, CI would publish a new digest, Renovate would bump + # it again — forever. pm-dispatch overwrites this field from PM_WORKER_IMAGE + # (platform/pm-agent/configmap.yaml, which IS tracked) on every Job it creates. image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000 imagePullPolicy: IfNotPresent # The pinned entrypoint, with NO args. Args on a pinned command are diff --git a/infra/devcontainer/README.md b/infra/devcontainer/README.md index 99cb8fd..3cfbaa3 100644 --- a/infra/devcontainer/README.md +++ b/infra/devcontainer/README.md @@ -34,6 +34,22 @@ cd infra/devcontainer ``` `ENGINE=docker ./dev.sh` to use docker instead of rootless podman. +### The published toolbox + +`./dev.sh build` delegates to [`build-image.sh`](build-image.sh), which is the **one** build +recipe: CI runs the same script (with its in-image verification — every tool executes, and the +version-sensitive ones report their `versions.env` pins) and, on a push to `main` that changes +`Containerfile`, `versions.env` or `build-image.sh`, **publishes** the result as +`gitea.home.keithjackson.dev/home-network/toolbox` (tags `sha-` and `main`) — +see [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). `agent-runner` is +built `FROM` that published digest (`infra/agent/versions.env`, kept current by Renovate). +Nobody pushes this image from a workstation. To use the published one locally instead of +building, after `podman login gitea.home.keithjackson.dev` (a token with `read:package`): + +```bash +IMAGE=gitea.home.keithjackson.dev/home-network/toolbox:main ./dev.sh run make -C ci gate +``` + ## Choosing a cluster — `CLUSTER=dev|prod` ```bash diff --git a/infra/devcontainer/build-image.sh b/infra/devcontainer/build-image.sh new file mode 100755 index 0000000..9467a8e --- /dev/null +++ b/infra/devcontainer/build-image.sh @@ -0,0 +1,108 @@ +#!/usr/bin/env bash +############################################################################### +# infra/devcontainer/build-image.sh — build the pinned dev `toolbox` image and prove it works. +# +# ./build-image.sh # build + verify (tag: home-k3s-dev:local) +# ./build-image.sh --no-verify # build only +# IMAGE=foo:bar ./build-image.sh # build under another local name +# ENGINE=docker ./build-image.sh # build with docker instead of podman +# +# ONE BUILD DEFINITION, TWO CALLERS. `dev.sh build` calls this for local development, and the +# CI image pipeline (.gitea/workflows/images.yaml -> .gitea/workflows/scripts/images.sh) calls +# it to build the image it then PUBLISHES as gitea.home.keithjackson.dev/home-network/toolbox. +# Keeping the build args in exactly one place is what makes "the toolbox CI publishes" and "the +# toolbox you gate with locally" the same recipe. +# +# THIS SCRIPT NEVER PUSHES. Publishing is CI's job (images.yaml, on push to main), not a +# workstation's: an image that exists only because someone ran `podman push` from a laptop has +# no build log, no recorded inputs and no verification anyone else can read. +# +# VERSIONS ARE NOT DEFINED HERE. infra/devcontainer/versions.env is the single source of truth. +############################################################################### +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ENGINE="${ENGINE:-podman}" +IMAGE="${IMAGE:-home-k3s-dev:local}" + +VERIFY=1 +for arg in "$@"; do + case "$arg" in + --no-verify) VERIFY=0 ;; + -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;; + *) echo "unknown argument: $arg" >&2; exit 2 ;; + esac +done + +log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; } +die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; } + +command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)" + +# shellcheck disable=SC1091 +source "${HERE}/versions.env" + +# Provenance: CI passes the commit it is building, so a pulled image names its source tree. +LABEL_ARGS=() +[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}") + +log "Building ${IMAGE}" +"$ENGINE" build "${LABEL_ARGS[@]}" \ + --build-arg FEDORA_VERSION="$FEDORA_VERSION" \ + --build-arg OPENTOFU_VERSION="$OPENTOFU_VERSION" \ + --build-arg AGE_VERSION="$AGE_VERSION" \ + --build-arg SOPS_VERSION="$SOPS_VERSION" \ + --build-arg FLUX_VERSION="$FLUX_VERSION" \ + --build-arg KUBECTL_VERSION="$KUBECTL_VERSION" \ + --build-arg HELM_VERSION="$HELM_VERSION" \ + --build-arg KYVERNO_VERSION="$KYVERNO_VERSION" \ + --build-arg KYVERNO_JSON_VERSION="$KYVERNO_JSON_VERSION" \ + --build-arg CHAINSAW_VERSION="$CHAINSAW_VERSION" \ + --build-arg YAMLLINT_VERSION="$YAMLLINT_VERSION" \ + --build-arg YQ_VERSION="$YQ_VERSION" \ + --build-arg CHECK_JSONSCHEMA_VERSION="$CHECK_JSONSCHEMA_VERSION" \ + --build-arg PROMTOOL_VERSION="$PROMTOOL_VERSION" \ + -t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}" + +# --- verify ------------------------------------------------------------------ +# A build that succeeded is not evidence the toolbox can run the gate. Every tool the gate and +# the runbooks call must EXECUTE, and the version-sensitive ones must report the version +# versions.env pins — promtool above all, because ci/versions.env asserts a promtool/Prometheus +# lockstep and a drifted engine evaluates alert-rule tests differently (see versions.env). +if [ "${VERIFY}" -eq 1 ]; then + log "Verifying the toolchain inside ${IMAGE}" + "$ENGINE" run --rm --network none --entrypoint "" \ + -e PROMTOOL_VERSION="${PROMTOOL_VERSION}" \ + -e KUBECTL_VERSION="${KUBECTL_VERSION}" \ + -e CHAINSAW_VERSION="${CHAINSAW_VERSION}" \ + -e KYVERNO_VERSION="${KYVERNO_VERSION}" \ + -e SOPS_VERSION="${SOPS_VERSION}" \ + "$IMAGE" /bin/bash -euo pipefail -c ' + echo "--- baked versions ---"; cat /etc/dev-toolchain-versions + echo "--- every tool is on PATH ---" + for t in tofu age age-keygen sops flux kubectl helm kyverno kyverno-json chainsaw yq \ + yamllint check-jsonschema promtool ansible-playbook git make jq diff python3 \ + wg qrencode; do + if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t"; + else printf " MISSING %s\n" "$t"; missing=1; fi + done + [ -z "${missing:-}" ] || { echo "toolbox is missing tools"; exit 1; } + echo "--- version-sensitive tools report their pins ---" + check() { # name pin actual-output + if printf "%s" "$3" | grep -qF "$2"; then printf " ok %-10s %s\n" "$1" "$2"; + else printf " DRIFT %-10s want %s, got: %s\n" "$1" "$2" "$(printf "%s" "$3" | head -1)"; drift=1; fi + } + check promtool "${PROMTOOL_VERSION}" "$(promtool --version 2>&1)" + check kubectl "${KUBECTL_VERSION}" "$(kubectl version --client 2>&1)" + check chainsaw "${CHAINSAW_VERSION}" "$(chainsaw version 2>&1)" + check kyverno "${KYVERNO_VERSION}" "$(kyverno version 2>&1)" + check sops "${SOPS_VERSION}" "$(sops --version --disable-version-check 2>&1 || sops --version 2>&1)" + [ -z "${drift:-}" ] || { echo "a tool does not report its pinned version"; exit 1; } + echo "--- python can import what the renderer and the gate need ---" + python3 -c "import yaml; print(\" PyYAML\", yaml.__version__)" + ' + log "Toolbox verified" +fi + +log "Built ${IMAGE}" +printf ' image id %s\n' "$("$ENGINE" image inspect --format '{{.Id}}' "$IMAGE")" diff --git a/infra/devcontainer/dev.sh b/infra/devcontainer/dev.sh index d86561c..29701d6 100755 --- a/infra/devcontainer/dev.sh +++ b/infra/devcontainer/dev.sh @@ -72,22 +72,10 @@ esac CLUSTER_KUBECONFIG="${REPO_ROOT}/infra/ansible/.cache/${CLUSTER}.kubeconfig" build() { - "$ENGINE" build \ - --build-arg FEDORA_VERSION="$FEDORA_VERSION" \ - --build-arg OPENTOFU_VERSION="$OPENTOFU_VERSION" \ - --build-arg AGE_VERSION="$AGE_VERSION" \ - --build-arg SOPS_VERSION="$SOPS_VERSION" \ - --build-arg FLUX_VERSION="$FLUX_VERSION" \ - --build-arg KUBECTL_VERSION="$KUBECTL_VERSION" \ - --build-arg HELM_VERSION="$HELM_VERSION" \ - --build-arg KYVERNO_VERSION="$KYVERNO_VERSION" \ - --build-arg KYVERNO_JSON_VERSION="$KYVERNO_JSON_VERSION" \ - --build-arg CHAINSAW_VERSION="$CHAINSAW_VERSION" \ - --build-arg YAMLLINT_VERSION="$YAMLLINT_VERSION" \ - --build-arg YQ_VERSION="$YQ_VERSION" \ - --build-arg CHECK_JSONSCHEMA_VERSION="$CHECK_JSONSCHEMA_VERSION" \ - --build-arg PROMTOOL_VERSION="$PROMTOOL_VERSION" \ - -t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}" + # The build recipe lives in build-image.sh so that CI (.gitea/workflows/images.yaml), which + # builds and PUBLISHES this image, and a local `dev.sh build` cannot drift apart. --no-verify + # keeps the local loop as fast as it always was; CI runs the verification. + IMAGE="$IMAGE" ENGINE="$ENGINE" "${HERE}/build-image.sh" --no-verify } run() { diff --git a/infra/gitea/conformance-trigger-kubeconfig.sh b/infra/gitea/conformance-trigger-kubeconfig.sh index 4b7e30d..9016409 100755 --- a/infra/gitea/conformance-trigger-kubeconfig.sh +++ b/infra/gitea/conformance-trigger-kubeconfig.sh @@ -252,11 +252,16 @@ step "preconditions — repo side (these block ONLY arming ${VARIABLE}; the secr # R1: the runner image is built and digest-pinned. Without it the Job cannot run run.sh at # all (the stock chainsaw image has no bash/python3/kubectl/git) — plan §3 item 1. +# EVALUATED, not grepped: the line is `CONFORMANCE_RUNNER_IMAGE="${..._REPO}@${..._DIGEST}"`, so +# its literal text never contains 64 hex digits and the grep this replaced could never pass. The +# all-zero digest is the "not published yet" placeholder (Renovate replaces it after CI's first +# publish, .gitea/workflows/images.yaml) and must not count as pinned. VERSIONS="${REPO_ROOT}/conformance/versions.env" -if grep -Eq '^[[:space:]]*CONFORMANCE_RUNNER_IMAGE=.*@sha256:[0-9a-f]{64}' "$VERSIONS" 2>/dev/null; then - pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env" +runner_img="$( (set +u; . "$VERSIONS" >/dev/null 2>&1; printf '%s' "${CONFORMANCE_RUNNER_IMAGE:-}") )" +if [[ "$runner_img" =~ @sha256:[0-9a-f]{64}$ ]] && [[ ! "$runner_img" =~ @sha256:0{64}$ ]]; then + pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env (${runner_img##*@})" else - fail_arm "conformance/versions.env has no digest-pinned CONFORMANCE_RUNNER_IMAGE (plan §3 item 1) — the Job cannot run ./run.sh" + fail_arm "conformance/versions.env has no PUBLISHED digest for CONFORMANCE_RUNNER_IMAGE (got '${runner_img}') — the image pipeline has not published it yet, or Renovate's bump is unmerged (plan §3 item 1)" fi # R2: the Job template is the real thing, not the skeleton: the admitted image AND an diff --git a/infra/gitea/registry-retention.sh b/infra/gitea/registry-retention.sh new file mode 100755 index 0000000..e62f6c0 --- /dev/null +++ b/infra/gitea/registry-retention.sh @@ -0,0 +1,130 @@ +#!/usr/bin/env bash +# Registry retention for this repo's OWN images (toolbox, agent-runner, conformance-runner): +# keep what is deployed and what is recent, delete the rest. DRY RUN unless --apply. +# +# ./infra/gitea/registry-retention.sh # show what would be deleted +# ./infra/gitea/registry-retention.sh --apply # delete it +# KEEP=15 ./infra/gitea/registry-retention.sh # keep the 15 newest sha- tags (default 10) +# +# RUNNING IT IS AN OPERATOR ACTION. It deletes registry content, so it is not wired into CI and +# no agent runs it with --apply. +# +# WHY A SCRIPT, AND NOT GITEA'S BUILT-IN "CLEANUP RULE" — measured against Gitea 1.24.3's source: +# +# 1. There is NO API for cleanup rules. routers/api/v1/api.go has no such route; the rules live +# only behind the web UI's session+CSRF forms (routers/web/web.go, org settings +# `/packages/rules/add`). A rule therefore cannot be created, read or drift-checked as code — +# which is also why ci/gitea-settings.sh cannot assert one exists. +# 2. A built-in rule would DELETE WHAT IS DEPLOYED. Every reference to these images is +# digest-only (`@sha256:...`, required by admission). Gitea stores a manifest as a +# file OF THE TAG VERSION it was pushed under (routers/api/packages/container/manifest.go +# createPackageAndVersion), and resolves a by-digest pull by searching those files +# (models/packages/container/search.go). So deleting the tag a digest was pushed under makes +# that digest unpullable — MANIFEST_UNKNOWN, i.e. ImagePullBackOff on the next reschedule. +# The built-in rule keeps "the newest N" and a name pattern, and knows nothing about which +# digest git currently pins. Renovate's bump PRs can lag, so "newest N" is not a guarantee. +# +# WHAT IS KEPT, per image: +# * the `main` and `latest` tags; +# * the KEEP newest `sha-` tags (by creation time); +# * EVERY tag whose manifest digest is referenced anywhere in the tracked tree at REF +# (default origin/main) — i.e. whatever Flux is deploying, and the toolbox digest +# agent-runner builds FROM; +# * every version whose NAME is a digest (`sha256:...`): those are index children, and Gitea's +# own cleanup handles them (services/packages/container/cleanup.go ShouldBeSkipped). +# Everything else is deleted. Gitea's `cleanup_packages` cron then reclaims the unreferenced +# blobs (default: daily, blobs older than 24h) — space comes back the day after, not instantly. +# +# THE STORAGE IT PROTECTS. Packages live on the SAME 10Gi Longhorn PVC as every repository and +# Gitea's database (catalog/gitea.yaml). The toolbox is ~1.4 GB uncompressed and every main push +# that touches its inputs adds a complete, non-reproducible copy. Unpruned, the registry is what +# fills the volume the git host lives on. +# +# CREDENTIAL: the one `git` has stored for the host (as the other scripts here). It needs +# `read:package` to list and `write:package` to --apply, and the account needs package write on +# the org. The secret is never printed. +set -euo pipefail +HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; NS=home-network +IMAGES=(toolbox agent-runner conformance-runner) +KEEP="${KEEP:-10}" +REF="${REF:-origin/main}" +APPLY=0 +for a in "$@"; do + case "$a" in + --apply) APPLY=1 ;; + -h|--help) sed -n '2,/^set -euo/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;; + *) echo "unknown argument: $a" >&2; exit 2 ;; + esac +done +[[ "$KEEP" =~ ^[0-9]+$ ]] && [ "$KEEP" -ge 1 ] || { echo "KEEP must be a positive integer" >&2; exit 2; } +command -v jq >/dev/null || { echo "jq is required" >&2; exit 1; } +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" + +git -C "$REPO_ROOT" fetch -q origin main 2>/dev/null || echo " (could not fetch origin/main; using the local ${REF})" >&2 +git -C "$REPO_ROOT" rev-parse -q --verify "${REF}^{commit}" >/dev/null || { echo "no such ref: ${REF}" >&2; exit 1; } + +creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)" +U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")" +[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; } +echo "authenticating as '${U}' (secret not shown)" +api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; } + +REG_TOKEN="$(api "https://${HOST}/v2/token?service=container_registry" | jq -r '.token // empty')" +[ -n "$REG_TOKEN" ] || { echo "registry token exchange failed (does the credential have read:package?)" >&2; exit 1; } +manifest_digest() { # -> digest, or nothing + curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "$REG_TOKEN") \ + -H 'Accept: application/vnd.oci.image.manifest.v1+json' \ + -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \ + -H 'Accept: application/vnd.oci.image.index.v1+json' \ + -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \ + "https://${HOST}/v2/${NS}/$1/manifests/$2" \ + | tr -d '\r' | awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' | tail -1 +} + +TOTAL_DEL=0 +for img in "${IMAGES[@]}"; do + echo + echo "==> ${NS}/${img}" + # Digests git pins at REF. The marker convention puts the digest on the line after + # `# renovate-digest: image=`; the plain `@sha256:` grep catches any other use. + repo="${HOST}/${NS}/${img}" + pinned="$( { git -C "$REPO_ROOT" grep -h -A1 -F "renovate-digest: image=${repo} " "$REF" -- . 2>/dev/null || true + git -C "$REPO_ROOT" grep -h -F "${repo}@sha256:" "$REF" -- . 2>/dev/null || true; } \ + | grep -oE 'sha256:[0-9a-f]{64}' | grep -v 'sha256:0\{64\}' | sort -u || true)" + echo " pinned at ${REF}: $(wc -w <<<"$pinned") digest(s)" + + versions="[]"; page=1 + while :; do + chunk="$(api "https://${HOST}/api/v1/packages/${OWNER}?type=container&q=${img}&limit=50&page=${page}")" + jq -e 'type=="array"' >/dev/null <<<"$chunk" || { echo " package list failed: $(head -c 200 <<<"$chunk")" >&2; exit 1; } + [ "$(jq length <<<"$chunk")" -gt 0 ] || break + versions="$(jq -c --argjson a "$versions" --arg n "$img" '$a + [ .[] | select(.name==$n) | {version, created_at} ]' <<<"$chunk")" + page=$((page + 1)) + done + echo " versions: $(jq length <<<"$versions")" + + keep_sha="$(jq -r '[ .[] | select(.version|startswith("sha-")) ] | sort_by(.created_at) | reverse | .[:'"$KEEP"'][] .version' <<<"$versions")" + while IFS= read -r v; do + [ -n "$v" ] || continue + case "$v" in + main|latest) echo " keep ${v} (moving tag)"; continue ;; + sha256:*) echo " keep ${v} (digest-named version; Gitea's cleanup owns those)"; continue ;; + esac + if grep -qxF "$v" <<<"$keep_sha"; then echo " keep ${v} (among the ${KEEP} newest)"; continue; fi + d="$(manifest_digest "$img" "$v")" + if [ -z "$d" ]; then echo " keep ${v} (could not read its digest — never delete what cannot be checked)"; continue; fi + if grep -qxF "$d" <<<"$pinned"; then echo " keep ${v} (${d} is pinned at ${REF})"; continue; fi + if [ "$APPLY" -eq 1 ]; then + code="$(api -o /dev/null -w '%{http_code}' -X DELETE \ + "https://${HOST}/api/v1/packages/${OWNER}/container/${img}/$(jq -rn --arg v "$v" '$v|@uri')")" + if [ "$code" = 204 ]; then echo " DELETED ${v} (${d})"; TOTAL_DEL=$((TOTAL_DEL + 1)); + else echo " FAILED ${v} HTTP ${code}" >&2; fi + else + echo " would delete ${v} (${d})"; TOTAL_DEL=$((TOTAL_DEL + 1)) + fi + done < <(jq -r '.[].version' <<<"$versions") +done + +echo +if [ "$APPLY" -eq 1 ]; then echo "deleted ${TOTAL_DEL} version(s). Blobs are reclaimed by Gitea's cleanup_packages cron." +else echo "DRY RUN: ${TOTAL_DEL} version(s) would be deleted. Re-run with --apply to delete them."; fi diff --git a/platform/pm-agent/README.md b/platform/pm-agent/README.md index b201f37..5247aff 100644 --- a/platform/pm-agent/README.md +++ b/platform/pm-agent/README.md @@ -45,11 +45,18 @@ waive the sensitive-path human review on its own pull request. Both are deliberate, and both fail **visibly** rather than silently: -1. **The image does not exist.** `platform/pm-agent/deployment.yaml` and - `PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder. Build it with - `infra/agent/build-image.sh` (which verifies it and refuses to push), push it as the - operator, then update **both** places with the digest `podman push --digestfile` returned, - in one pull request. Until then: `ImagePullBackOff`. +1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and + `PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder until the first + publish. `.gitea/workflows/images.yaml` builds `agent-runner` with + `infra/agent/build-image.sh` (the same script, and the same in-image verification, you can + run locally), and on a push to `main` publishes it to + `gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-` and `main`. + Renovate then follows the `main` tag through the `# renovate-digest:` marker above **both** + lines and moves them together in one pull request. Nobody builds-and-pushes this image from + a workstation, and nobody pastes a digest. Until the first publish, and until that PR + merges: `ImagePullBackOff`. The one operator prerequisite on the publishing side is the + `REGISTRY_PUSH_TOKEN` Actions secret (see the header of `.gitea/workflows/images.yaml`); + without it the publish job fails loudly rather than skipping. **Two things have to be true before any digest will pull, and neither is declared in this repository.** Both were measured after PR #57 merged — this service is the first workload @@ -78,8 +85,8 @@ Both are deliberate, and both fail **visibly** rather than silently: account whose token carries `read:package`, referenced from the Deployment and the committed worker Job template — not built yet, because minting that token is yours. - The same two apply to `conformance-runner`: same registry path, same push-then-pin model, - never pulled either. + The same two apply to `conformance-runner`: same registry path, same publish-then-Renovate + model, never pulled either. 2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off — which Gitea account to create, which scopes, and why it must *not* be an administrator — diff --git a/platform/pm-agent/configmap.yaml b/platform/pm-agent/configmap.yaml index 16de902..61104a1 100644 --- a/platform/pm-agent/configmap.yaml +++ b/platform/pm-agent/configmap.yaml @@ -52,12 +52,13 @@ data: # contract. A tag here is refused at startup (and would be DENIED by admission anyway). # # PLACEHOLDER UNTIL THE IMAGE IS PUBLISHED. The all-zero digest below is deliberately - # impossible: `infra/agent/build-image.sh` builds and verifies the image, the OPERATOR - # pushes it, and the authoritative digest from `--digestfile` replaces this value in a - # reviewed pull request (the same two-step the conformance-runner image follows, and for - # the same reason: a registry credential is not an agent's to hold). Until then the - # dispatcher starts, fails to pull, and says so — which is the visible failure, not a - # silent one. + # impossible. CI builds, verifies and — on main — publishes agent-runner + # (.gitea/workflows/images.yaml), and Renovate replaces this value with the digest the + # REGISTRY returned for the image's `main` tag, in a pull request that also moves the + # Deployment's image (same marker, same branch). Nobody pushes this image from a workstation. + # Until the first publish the dispatcher starts, fails to pull, and says so — which is the + # visible failure, not a silent one. + # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000" # --- the model ------------------------------------------------------------- diff --git a/platform/pm-agent/deployment.yaml b/platform/pm-agent/deployment.yaml index 96c8812..fdd4adb 100644 --- a/platform/pm-agent/deployment.yaml +++ b/platform/pm-agent/deployment.yaml @@ -66,9 +66,12 @@ spec: - name: dispatch # THE SAME IMAGE THE WORKERS RUN, at the same digest. They share the committed Job # template and the env-name contract baked into it, so a mismatched pair would - # disagree about the shape the admission policy enforces. Replaced together, in one - # pull request, with the digest `infra/agent/build-image.sh` printed — the all-zero + # disagree about the shape the admission policy enforces. Replaced together, in ONE + # Renovate pull request: both lines carry the same marker, so Renovate treats them as + # one dependency and bumps them in the same branch. The digest comes from CI's publish + # of agent-runner's `main` tag (.gitea/workflows/images.yaml) — the all-zero # placeholder below is impossible on purpose (see configmap.yaml PM_WORKER_IMAGE). + # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000 imagePullPolicy: IfNotPresent command: diff --git a/renovate.json5 b/renovate.json5 index f6decdf..48e5ee4 100644 --- a/renovate.json5 +++ b/renovate.json5 @@ -22,9 +22,28 @@ // podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \ // ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json5 // -// NOTE on what was NOT wrong: `enabledManagers` containing bare `regex` was suspected of needing to -// be `custom.regex`. The validator accepts `regex`, so that suspicion was unfounded and nothing -// here changes it. +// CORRECTION (2026-10-29): THE VALIDATOR ABOVE WAS NOT THE RENOVATE THE WORKFLOW RUNS. The bare +// `ghcr.io/renovatebot/renovate` tag is `latest` (44.167.0 when this was measured), while +// renovatebot/github-action@v41.0.9 defaults `renovate-version` to `39` (its action.yml), i.e. +// 39.264.0. Validated against THAT, this file failed with five errors — every +// `managerFilePatterns` key ("Custom Manager contains disallowed fields", "Invalid configuration +// option: flux.managerFilePatterns", ...), because 39.x only knows `fileMatch`; the rename landed +// in 40.2.0. So even after the workflow found this file, every run would have stopped at +// CONFIG_VALIDATION. renovate.yaml now pins `renovate-version` to the exact release this file is +// validated against. VALIDATE WITH THAT SAME TAG: +// podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \ +// ghcr.io/renovatebot/renovate: \ +// renovate-config-validator /tmp/renovate.json5 +// (Not `--strict`: that also fails on a pending migration of the pre-existing top-level +// `"docker": {"enabled": true}` block at the bottom, which is a warning, not an error.) +// +// `regex` vs `custom.regex` in enabledManagers, RESOLVED FROM SOURCE (Renovate 39.264.0 and +// later): both work. lib/config/migrations/custom/enabled-managers-migration.ts rewrites +// `regex` -> `custom.regex`, and that migration runs on every real repository run +// (lib/workers/repository/init/merge.ts -> migrateAndValidate), not only in the validator; +// lib/modules/manager/index.ts getEnabledManagersList accepts either spelling. `custom.regex` is +// the canonical name (introduced in 37.51.1, renovatebot/renovate#24079), so it is used below and +// nothing depends on the migration any more. { "$schema": "https://docs.renovatebot.com/renovate-schema.json", // Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17 // RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags. @@ -51,7 +70,7 @@ "labels": ["dependencies", "automated-update"], "ignoreUnstable": true, "respectLatest": true, - "enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"], + "enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "custom.regex", "github-actions"], "flux": { "managerFilePatterns": [ "/(^|/)platform/.+\\.ya?ml$/", @@ -93,6 +112,42 @@ "#\\s*renovate:\\s*datasource=(?\\S+)\\s+depName=(?\\S+)(?:\\s+versioning=(?\\S+))?\\s*\\n.*?(?v?[0-9]+\\.[0-9]+\\.[0-9]+)" ], "versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}" + }, + // ADDED 2026-10-29 — THIS REPO'S OWN IMAGES (toolbox, agent-runner, conformance-runner). + // CI publishes them (.gitea/workflows/images.yaml) under an immutable `sha-` tag and a + // moving `main` tag, and never writes to git. Every reference to them is DIGEST-ONLY + // (`@sha256:`, no tag), because the admission policies + // (platform/kyverno/admission/pm-agent-job-shape.yaml, conformance-trigger-job-shape.yaml) + // admit exactly that form. A digest alone gives Renovate nothing to follow, so the tag to + // follow rides in a marker comment on the line ABOVE the reference: + // + // # renovate-digest: image=// tag=main + // sha256:<64 hex> + // + // `main` is not a version, so Renovate skips the release lookup and proposes a `digest` + // update whenever the tag's digest moves (lib/workers/repository/process/lookup/index.ts: + // an unversioned currentValue WITH a currentDigest yields updateType `digest`). The + // `sha256:` prefix is INSIDE currentDigest on purpose: Renovate replaces that capture + // verbatim with the new digest, which carries the prefix. + // + // DELIBERATELY NOT TRACKED: infra/agent/worker-job-template.yaml. It is baked INTO + // agent-runner, so bumping it would rebuild agent-runner and loop forever; pm-dispatch + // overwrites that field from PM_WORKER_IMAGE at runtime anyway. + // REGISTRY AUTH: the Gitea registry answers 401 to anonymous reads, and the platform token is + // scoped to hostType `gitea`, which a docker lookup never matches + // (lib/util/host-rules.ts find). renovate.yaml supplies a hostType `docker` rule. + { + "customType": "regex", + "description": "This repo's own CI-published images, pinned by digest, following the tag named in a `# renovate-digest:` marker on the previous line.", + "managerFilePatterns": [ + "/(^|/)versions\\.env$/", + "/(^|/)platform/pm-agent/.+\\.ya?ml$/" + ], + "matchStrings": [ + "#\\s*renovate-digest:\\s*image=(?\\S+)\\s+tag=(?\\S+)\\s*\\n[^\\n]*?(?sha256:[a-f0-9]{64})" + ], + "datasourceTemplate": "docker", + "versioningTemplate": "docker" } ], "packageRules": [ @@ -151,6 +206,26 @@ "needs-review", "blast-radius" ] + }, + { + "description": "ADDED 2026-10-29. This repo's own images are followed ONLY through the `# renovate-digest:` markers above. The `kubernetes` manager also sees platform/pm-agent/deployment.yaml and infra/agent/worker-job-template.yaml, where the image has a digest and no tag; left on, it would look those up as `latest` (a tag CI never pushes) and, for the template, re-open the self-referential loop the custom manager deliberately avoids.", + "matchManagers": ["kubernetes"], + "matchPackageNames": [ + "gitea.home.keithjackson.dev/home-network/toolbox", + "gitea.home.keithjackson.dev/home-network/agent-runner", + "gitea.home.keithjackson.dev/home-network/conformance-runner" + ], + "enabled": false + }, + { + "description": "ADDED 2026-10-29. Digest bumps for this repo's own images: the image was built and verified by CI before it was published, so these auto-merge on a green gate like any digest (rule 1). Labelled so they are easy to find. NOTE: the conformance-runner bump edits conformance/versions.env, a sensitive path (infra/gitea/approval-policy.json), so it will wait for an administrator's /approve-override regardless of this rule.", + "matchManagers": ["custom.regex"], + "matchPackageNames": [ + "gitea.home.keithjackson.dev/home-network/toolbox", + "gitea.home.keithjackson.dev/home-network/agent-runner", + "gitea.home.keithjackson.dev/home-network/conformance-runner" + ], + "addLabels": ["own-image"] } ], // WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A @@ -168,10 +243,20 @@ // shape, not software to keep current. `conformance/**` was already ignored for the same // reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise // have opened. + // conformance/** NARROWED 2026-10-29 to its subdirectories: ignorePaths applies to custom + // managers too (lib/workers/repository/extract/file-match.ts getMatchingFiles -> + // filterIgnoredFiles), so the blanket glob also hid conformance/versions.env, which now carries + // the conformance-runner digest Renovate must maintain. Everything else under conformance/ + // stays ignored, for the original reason (test fixtures with deliberately fixed shapes). "ignorePaths": [ "**/*.example", "**/*.sops.yaml", - "conformance/**", + "conformance/config/**", + "conformance/incluster/**", + "conformance/lib/**", + "conformance/requirements/**", + "conformance/runs/**", + "conformance/tests/**", "platform/image-automation/per-service/**", "platform/kyverno/tests/**" ], diff --git a/runbooks/pm-agent.md b/runbooks/pm-agent.md index 7f30319..a99410e 100644 --- a/runbooks/pm-agent.md +++ b/runbooks/pm-agent.md @@ -67,10 +67,10 @@ Work through these in order; the first three are by far the most likely. | Symptom in the log | Cause | Fix | |---|---|---| | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` | -| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first | +| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been published; the committed digest is the deliberate placeholder | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first | | `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` | | `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours | -| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` | +| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | restore the digest form; the digest comes from CI's publish (`.gitea/workflows/images.yaml` job summary) and Renovate's bump PR, never from a hand push | | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob | | `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) | | `at the worker cap` / `worker limit reached` | the per-cycle cap, draining a backlog one issue at a time | expected; it will continue next cycle | From 8f35234fcc8a2613195f5be95886c2fbad021643 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Thu, 29 Oct 2026 21:44:38 -0500 Subject: [PATCH 4/7] fix(images): name the measured runner blocker; the job container gets no Docker daemon PR #81's first images run failed in 6s: act_runner 0.2.11 forces docker_host '-' for a tcp:// DOCKER_HOST, and gitea/act drops every bind when valid_volumes is empty. Giving jobs the privileged DinD daemon is node-root for any workflow author, so it is left to the operator; options are in the handoff entry. The driver's failure now names the cause. Co-Authored-By: Claude Opus 5.5 --- .gitea/workflows/images.yaml | 9 ++++ .gitea/workflows/scripts/images.sh | 9 +++- decisions/2026-10-29-image-pipeline.md | 67 ++++++++++++++++++++++++-- 3 files changed, 80 insertions(+), 5 deletions(-) diff --git a/.gitea/workflows/images.yaml b/.gitea/workflows/images.yaml index 3d027bc..6170fe4 100644 --- a/.gitea/workflows/images.yaml +++ b/.gitea/workflows/images.yaml @@ -41,6 +41,15 @@ # not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER` # is optional and only labels the login. # +# ┌─ BLOCKER, MEASURED ON PR #81 (run 325): THIS RUNNER CANNOT BUILD IMAGES YET ───────────────┐ +# │ The job container gets no Docker daemon. act_runner 0.2.11 forces container.docker_host to │ +# │ "-" whenever DOCKER_HOST is tcp:// (internal/app/cmd/daemon.go), and with valid_volumes: [] │ +# │ gitea/act strips every bind including the socket (pkg/container/docker_run.go │ +# │ sanitizeConfig). So this job fails at its first check, loudly, until the operator chooses │ +# │ how workflow code may reach a daemon — see decisions/2026-10-29-image-pipeline.md. It is │ +# │ not a required check, so it does not block merges; it is red, which is the honest state. │ +# └──────────────────────────────────────────────────────────────────────────────────────────────┘ +# # THE RUNNER (apps/wave-1/gitea-actions.yaml): act_runner 0.2.11, one job at a time, Docker # executor on a privileged `docker:27-dind` sidecar whose image store is an emptyDir on the node # (no sizeLimit) and whose memory limit is 2Gi — shared by the job container AND every build diff --git a/.gitea/workflows/scripts/images.sh b/.gitea/workflows/scripts/images.sh index 2d83fb4..a36a950 100755 --- a/.gitea/workflows/scripts/images.sh +++ b/.gitea/workflows/scripts/images.sh @@ -200,8 +200,13 @@ push_one() { # -> sets DIGEST[image] to the registry-returned digest log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}" command -v "${ENGINE}" >/dev/null || die "${ENGINE} is not on PATH in the job container" "${ENGINE}" info >/dev/null 2>&1 \ - || die "the ${ENGINE} daemon is not reachable from the job container — the runner did not - hand it a Docker socket (act_runner container.docker_host). Nothing can be built." + || die "the ${ENGINE} daemon is not reachable from the job container, so nothing can be built. + MEASURED ON THIS RUNNER (PR #81, 2026-10-29): act_runner 0.2.11 never gives a job a + daemon. Its DOCKER_HOST is tcp:// (the DinD sidecar), and internal/app/cmd/daemon.go then + forces container.docker_host to '-' (\"can't be mounted\"); separately, with + valid_volumes: [] gitea/act's sanitizeConfig drops EVERY bind, the socket included. + Fixing it means handing workflow code the privileged DinD daemon — an operator security + decision, set out in decisions/2026-10-29-image-pipeline.md. Not a fault in this script." "${ENGINE}" version 2>/dev/null | sed -n '1,12p' || true "${ENGINE}" system df || true diff --git a/decisions/2026-10-29-image-pipeline.md b/decisions/2026-10-29-image-pipeline.md index 7fa2979..32e5a68 100644 --- a/decisions/2026-10-29-image-pipeline.md +++ b/decisions/2026-10-29-image-pipeline.md @@ -83,6 +83,56 @@ both `build-image.sh` scripts said "publishing is an operator action". `infra/devcontainer/README.md`, and a new "How images are built" section in `.gitea/workflows/README.md`. +### BLOCKER — the runner cannot build images, and fixing that is a security decision + +**Measured on this PR's own CI run (images run 325): the job failed in 6 s** with +`failed to connect to the docker API at unix:///var/run/docker.sock ... no such file`. The job +container is given **no Docker daemon at all**, for two independent reasons, both read from the +exact versions deployed: + +- act_runner **v0.2.11** `internal/app/cmd/daemon.go`: when `DOCKER_HOST` has a scheme other than + `unix`/`npipe` — here `tcp://localhost:2376`, the DinD sidecar — it sets + `cfg.Container.DockerHost = "-"`, which means "never mount into job containers". +- gitea/act **v0.261.2** (act_runner's `go.mod` replace) `pkg/container/docker_run.go` + `sanitizeConfig`: with `valid_volumes` empty (the runner has no config file, so the default + `[]`), `hostConfig.Binds = []string{}`. **Every** bind is dropped, the daemon socket included, + and so is any `volumes:` a workflow asks for. + +I assumed the opposite while designing (from act's `getDockerDaemonSocketMountPath`, which never +gets the chance to run). That was wrong, and the PR run is what showed it. + +**Why I did not fix it in this PR.** Any fix gives workflow code the DinD daemon. That daemon is +`privileged: true`, so `docker run --privileged -v /:/host` from a job is **root on the node**. +Today a malicious workflow — for example one pushed by a prompt-injected pm-agent worker, which +holds a repo write token — runs in an unprivileged container and holds no cluster credential. +After the fix it does not. That is a change to the security boundary, and it is the operator's +call. It also cannot be tested from a PR anyway: the runner only changes after merge + Flux. + +**Options, for the operator:** + +1. **Socket to every job (smallest change):** in `apps/wave-1/gitea-actions.yaml`, + - mount a shared `emptyDir` at `/var/run` in both the `dind` sidecar and the `runner`, so + dockerd's own `unix:///var/run/docker.sock` is visible at the same path in both; + - set the runner's `DOCKER_HOST=unix:///var/run/docker.sock`, and drop `DOCKER_TLS_VERIFY` + and `DOCKER_CERT_PATH`, because TLS over a unix socket breaks the client; + - add an act_runner config file (`CONFIG_FILE`) with `container.valid_volumes: + ["/var/run/docker.sock"]` and `runner.file: /data/.runner`, so the existing registration + is kept. + + Consequence: any workflow on any branch can reach node root. +2. **Option 1 on a SECOND runner** with its own label (e.g. `image-builder`) and its own DinD, + with `images.yaml` `runs-on` that label. The gate runner stays socket-less, and image builds + stop competing with the gate for the single slot and its 7 GB node disk. **It is not a + security boundary**: Gitea cannot restrict which workflow may target a label, so any branch + could still `runs-on: image-builder`. +3. **A builder that is not a privileged daemon**, e.g. a rootless BuildKit sidecar exposed to + jobs. That bounds a job to BuildKit's rootless sandbox instead of node root. It needs the + verification steps rewritten, because they *run* the image (as uid 65532, `--read-only`), + which BuildKit alone does not do. This is a larger piece of work. + +Nothing in this PR has to change for options 1 or 2 except `runs-on` for option 2. The driver +already produced verified images with a stock docker 27 CLI (see *What I verified*). + ### Decisions and pushback - **The push credential is a PAT secret, `REGISTRY_PUSH_TOKEN`; the Actions token cannot push.** @@ -156,6 +206,14 @@ both `build-image.sh` scripts said "publishing is an operator action". - agent-runner: `PM_SELFTEST=1 pm-dispatch` OK, and the stdin-fed `probe-validation.sh` gave "20 refusal(s), 0 failure(s)"; - conformance-runner: all seven `conformance-clone` refusals. +- **The docker-CLI code path**, i.e. what CI runs. I ran the same driver inside + `docker:27-cli` (`ENGINE=docker`, `DOCKER_BUILDKIT=0`) against podman's Docker-compatible API + socket. Exit code 0; all three images built and verified (selftest OK; probe OK, 20 refusals). + This covers the docker-specific calls (`image inspect` formats, `run -i` with stdin, + `--tmpfs`, `--read-only`, `--user`). It does **not** cover BuildKit or a real DinD daemon. +- **This PR's CI, run 325 (`images`): FAILED in 6 s**, because there is no Docker daemon in the + job container (see BLOCKER). Gate run 324 failed only on `approval-policy` (expected, + sensitive paths). Every other stage passed there, including `gitea-settings`. - **Other driver paths, run locally:** - agent-runner alone builds the toolbox as its base; - an empty diff builds nothing; @@ -198,13 +256,16 @@ both `build-image.sh` scripts said "publishing is an operator action". finalising a blob: copying the upload into package storage on Longhorn. A ~250 MB compressed layer should finish well inside 15 s, but I could not measure it without pushing. If the first publish shows 504s on blob PUTs, set `timeouts.request` on the gitea HTTPRoute. -- **DinD fit is estimated, not measured** (until the PR run reports): +- **DinD fit is estimated, not measured** (no build has run there, see the BLOCKER): - memory: 2Gi is shared by the job container and every build container; - disk: the toolbox, agent-runner, conformance-runner, Fedora base and build cache should total roughly 3 GB on top of the 1.5 GB act image, against 7 GB free on the node, with `dind-storage` having no sizeLimit. -- **That act_runner hands the job container a Docker socket is inferred** from act_runner's - DOCKER_HOST handling, not measured. The driver fails loudly if it does not. +- **No image has been built in this runner's DinD**, because it cannot be done until the BLOCKER + is resolved. Two things are therefore unmeasured until then: + - DinD memory (2Gi) and disk fit; + - BuildKit (`buildx` v0.20.1 is on the daemon) as opposed to the legacy builder I tested + with. - **Renovate has still never completed a run here.** Three things block it: - #80, the `configurationFile` name; - `RENOVATE_TOKEN` gaining `read:package`; From f65c12b3953f8adf6a14d43dc7096f48ee9311ba Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Thu, 29 Oct 2026 22:21:03 -0500 Subject: [PATCH 5/7] feat(images): build on rootless BuildKit over mTLS; verification moves into Containerfile verify stages Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched. - platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined, read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP :1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it). No Kyverno exception: every applied policy matches app-namespace=true only. - Each Containerfile names its shipped stage and ends in a `verify` stage holding every former `docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now. - images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes. Fails loudly when the builder secrets are absent or buildkitd is unreachable. - infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets. Co-Authored-By: Claude Opus 5.5 --- .gitea/workflows/README.md | 24 +- .gitea/workflows/images.yaml | 54 +- .gitea/workflows/scripts/images.sh | 315 +++++++----- clusters/base/buildkit.yaml | 21 + clusters/base/kustomization.yaml | 1 + conformance/incluster/Containerfile | 88 +++- conformance/incluster/README.md | 10 +- conformance/incluster/build-image.sh | 114 ++--- .../alert-rules-unit-tested/check.sh | 4 +- decisions/2026-10-29-image-pipeline.md | 474 ++++++++++-------- docs/plans/pm-agent-issue-to-pr.md | 4 +- infra/agent/Containerfile | 82 ++- infra/agent/build-image.sh | 130 ++--- infra/agent/probe-validation.sh | 10 +- infra/devcontainer/Containerfile | 50 +- infra/devcontainer/README.md | 8 +- infra/devcontainer/build-image.sh | 98 ++-- infra/gitea/buildkit-client-secret.sh | 104 ++++ platform/buildkit/README.md | 74 +++ platform/buildkit/certificates.yaml | 100 ++++ platform/buildkit/configmap.yaml | 24 + platform/buildkit/deployment.yaml | 142 ++++++ platform/buildkit/kustomization.yaml | 13 + platform/buildkit/namespace.yaml | 26 + platform/buildkit/networkpolicy.yaml | 88 ++++ platform/buildkit/service.yaml | 19 + platform/buildkit/versions.env | 15 + platform/pm-agent/README.md | 6 +- 28 files changed, 1490 insertions(+), 608 deletions(-) create mode 100644 clusters/base/buildkit.yaml create mode 100755 infra/gitea/buildkit-client-secret.sh create mode 100644 platform/buildkit/README.md create mode 100644 platform/buildkit/certificates.yaml create mode 100644 platform/buildkit/configmap.yaml create mode 100644 platform/buildkit/deployment.yaml create mode 100644 platform/buildkit/kustomization.yaml create mode 100644 platform/buildkit/namespace.yaml create mode 100644 platform/buildkit/networkpolicy.yaml create mode 100644 platform/buildkit/service.yaml create mode 100644 platform/buildkit/versions.env diff --git a/.gitea/workflows/README.md b/.gitea/workflows/README.md index c42412c..ace6abc 100644 --- a/.gitea/workflows/README.md +++ b/.gitea/workflows/README.md @@ -120,10 +120,20 @@ CI. That is a real coverage gap, not a phase. This repo's three images — `toolbox` (`infra/devcontainer/`), `agent-runner` (`infra/agent/`, built `FROM` the toolbox) and `conformance-runner` (`conformance/incluster/`) — are built, verified and published by [`images.yaml`](images.yaml), never from a workstation -(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh); each -image is built by **its own** `build-image.sh`, the same script a developer runs locally, so the -in-image verification (non-root uid, read-only rootfs, tool presence, `PM_SELFTEST=1 -pm-dispatch`, the `conformance-clone` refusals) exists once. +(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh). + +**The builder is rootless BuildKit** ([`platform/buildkit/`](../../platform/buildkit/README.md)), +reached over mTLS with a pinned `buildx` — no job here touches a Docker daemon, privileged or +otherwise (operator decision on PR #81). The client certificate reaches jobs as the Actions +secrets `BUILDKIT_CA`, `BUILDKIT_CLIENT_CERT`, `BUILDKIT_CLIENT_KEY`. + +**Verification is a build target.** Each Containerfile ends in a `verify` stage that runs every +in-image assertion at build time as the non-root uid with `RUN --network=none` (tool presence and +pinned versions, `PM_SELFTEST=1 pm-dispatch`, the pm-worker probe, the `conformance-clone` +refusals, no writable path outside the workspace). The pipeline builds `-verify` before it +builds, let alone pushes, the image. Build args come from each image's own +`build-image.sh --print-build-args`, which is also the local podman path — so the recipe and the +proof exist once. | Trigger | What happens | |---|---| @@ -143,8 +153,10 @@ account with package write on the org). The automatic Actions token cannot push [`infra/gitea/registry-retention.sh`](../../infra/gitea/registry-retention.sh), an operator action, keeps every digest git pins plus the newest tags. -Cost on this runner: the job shares the single act_runner slot with the gate, and the DinD store -is a node `emptyDir`; the job prunes it on exit. +Cost: the job shares the single act_runner slot with the gate, but the build itself runs in the +`buildkitd` pod, not in DinD, so the runner's disk and memory are untouched. Until +`platform/buildkit` is deployed and its client certificate copied +(`infra/gitea/buildkit-client-secret.sh`), the job fails loudly at its builder preflight. ## Files diff --git a/.gitea/workflows/images.yaml b/.gitea/workflows/images.yaml index 6170fe4..bbe700c 100644 --- a/.gitea/workflows/images.yaml +++ b/.gitea/workflows/images.yaml @@ -41,21 +41,22 @@ # not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER` # is optional and only labels the login. # -# ┌─ BLOCKER, MEASURED ON PR #81 (run 325): THIS RUNNER CANNOT BUILD IMAGES YET ───────────────┐ -# │ The job container gets no Docker daemon. act_runner 0.2.11 forces container.docker_host to │ -# │ "-" whenever DOCKER_HOST is tcp:// (internal/app/cmd/daemon.go), and with valid_volumes: [] │ -# │ gitea/act strips every bind including the socket (pkg/container/docker_run.go │ -# │ sanitizeConfig). So this job fails at its first check, loudly, until the operator chooses │ -# │ how workflow code may reach a daemon — see decisions/2026-10-29-image-pipeline.md. It is │ -# │ not a required check, so it does not block merges; it is red, which is the honest state. │ -# └──────────────────────────────────────────────────────────────────────────────────────────────┘ +# THE BUILDER — rootless BuildKit (operator decision on PR #81, 2026-10-29). +# The act runner's job containers get NO Docker daemon (act_runner 0.2.11 forces docker_host "-" +# for a tcp:// DinD, and gitea/act strips every bind when valid_volumes is empty — measured on +# this PR's first run), and handing them the privileged DinD daemon would be root on the node for +# anyone who can push a branch. So images are built by platform/buildkit/: a ROOTLESS buildkitd +# (not privileged; uid 1000 in a user namespace), reached over mTLS at +# buildkitd.buildkit.svc.cluster.local:1234 with a pinned `buildx` (remote driver). Its client +# certificate reaches jobs as three Actions secrets — BUILDKIT_CA, BUILDKIT_CLIENT_CERT, +# BUILDKIT_CLIENT_KEY — copied from cert-manager by infra/gitea/buildkit-client-secret.sh. +# Pull requests get the client certificate (they must build) but never a registry credential. +# +# UNTIL platform/buildkit IS DEPLOYED (Flux applies it only after merge) THIS JOB FAILS, LOUDLY, +# at its builder preflight. A skipped Gitea job reports `success`, so it is red on purpose. +# +# VERIFICATION is each Containerfile's `verify` stage, built before anything is pushed. # -# THE RUNNER (apps/wave-1/gitea-actions.yaml): act_runner 0.2.11, one job at a time, Docker -# executor on a privileged `docker:27-dind` sidecar whose image store is an emptyDir on the node -# (no sizeLimit) and whose memory limit is 2Gi — shared by the job container AND every build -# container. The toolbox is ~1.4 GB uncompressed and agent-runner adds ~0.3 GB on top of it, so the -# cleanup step below is not optional: without it each run would leave ~3 GB on a node that had -# 7 GB free when this was written, and the next eviction would take the runner with it. name: images on: @@ -71,6 +72,7 @@ on: - conformance/incluster/build-image.sh - .gitea/workflows/images.yaml - .gitea/workflows/scripts/images.sh + - platform/buildkit/versions.env push: branches: [main] paths: @@ -83,6 +85,7 @@ on: - conformance/incluster/build-image.sh - .gitea/workflows/images.yaml - .gitea/workflows/scripts/images.sh + - platform/buildkit/versions.env workflow_dispatch: inputs: images: @@ -115,6 +118,11 @@ jobs: MODE: verify IMAGES: auto PR_BASE_REF: ${{ github.base_ref }} + # The BUILDER's client certificate (platform/buildkit). Not a registry credential: + # a pull request can build, never push. + BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }} + BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }} + BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }} run: | set -euo pipefail BASE_SHA="$(git merge-base "origin/${PR_BASE_REF}" HEAD)" @@ -127,6 +135,9 @@ jobs: env: MODE: verify IMAGES: ${{ inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images }} + BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }} + BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }} + BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }} run: ./.gitea/workflows/scripts/images.sh # ── main: publish ─────────────────────────────────────────────────────────────────────── @@ -168,16 +179,7 @@ jobs: BASE_SHA: ${{ github.event.before }} REGISTRY_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }} REGISTRY_USER: ${{ vars.REGISTRY_PUSH_USER }} + BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }} + BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }} + BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }} run: ./.gitea/workflows/scripts/images.sh - - # ── always: give the DinD store back ──────────────────────────────────────────────────── - # The runner's image store is a node emptyDir with no sizeLimit (see header). Everything this - # job built or pulled is removed; `prune -a` cannot remove the job's own image, because the - # job container is still running on it. - - name: Clean the DinD image store - if: always() - run: | - docker logout gitea.home.keithjackson.dev >/dev/null 2>&1 || true - docker image prune -af >/dev/null || true - docker builder prune -af >/dev/null || true - docker system df || true diff --git a/.gitea/workflows/scripts/images.sh b/.gitea/workflows/scripts/images.sh index a36a950..3243d83 100755 --- a/.gitea/workflows/scripts/images.sh +++ b/.gitea/workflows/scripts/images.sh @@ -4,35 +4,41 @@ # # Builds, verifies and (on main) PUBLISHES the three images this repo ships: # -# toolbox infra/devcontainer/ the pinned dev/gate toolchain -# agent-runner infra/agent/ FROM toolbox; the PM dispatcher + worker +# toolbox infra/devcontainer/ the pinned dev/gate toolchain +# agent-runner infra/agent/ FROM toolbox; the PM dispatcher + worker # conformance-runner conformance/incluster/ the in-cluster L2/L3 Chainsaw runner # -# Each image is built by ITS OWN build-image.sh — the same script a developer runs locally — -# so the in-image verification (non-root uid, read-only rootfs, tool presence, pm-dispatch -# selftest, conformance-clone refusals, ...) is written once and runs in both places. This -# file only decides WHAT to build, in which order, FROM which base, and where it goes. +# THE BUILDER (operator decision on PR #81): a ROOTLESS BuildKit daemon, platform/buildkit/, +# reached over mTLS. No CI job may reach a privileged daemon, so there is no `docker build` and no +# `docker run` here: this script drives buildkitd with a pinned `buildx` (remote driver) and +# nothing else. The client certificate arrives as Actions secrets (BUILDKIT_CA, +# BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY; infra/gitea/buildkit-client-secret.sh puts them +# there). If the daemon is unreachable this FAILS LOUDLY — it never skips. +# +# VERIFICATION IS A BUILD TARGET. Each Containerfile ends in a `verify` stage that runs every +# in-image assertion at build time (as the non-root uid, with RUN --network=none); this script +# builds `-verify` and only then the image. Build args come from each image's own +# `build-image.sh --print-build-args`, the same list the local podman path uses. # # MODES (env MODE): -# verify build + verify, never push, holds NO credential. Pull requests. -# publish build + verify + push to ${REGISTRY}/${NAMESPACE}/ with tags +# verify build the verify targets, never push, hold no registry credential. Pull requests. +# publish verify, then build + push to ${REGISTRY}/${NAMESPACE}/ with tags # `sha-<12 hex>` (immutable: never re-pushed) and `main` (moving; Renovate follows # it). The digest recorded is the one the REGISTRY returns, read back after the push. # # WHAT GETS BUILT (env IMAGES): `auto` (default) diffs BASE_SHA..HEAD_SHA against each image's # inputs; otherwise a space-separated list. A change to the pipeline itself (images.yaml, this -# file) builds everything in verify mode, and nothing extra in publish mode: re-publishing -# unchanged inputs would only mint new digests and churn Renovate PRs. +# file, platform/buildkit/versions.env) builds everything in verify mode, and nothing extra in +# publish mode: re-publishing unchanged inputs would only mint new digests and churn Renovate PRs. # -# THE agent-runner BASE — the one ordering rule, and the reason this is not three jobs: -# * toolbox built in this run -> agent-runner builds FROM it (in publish mode, FROM the -# digest the registry just returned for it). -# * verify mode, toolbox not changed -> the toolbox is built anyway as a base. A pull request -# holds no registry credential, so it cannot pull the published one. -# * publish mode, toolbox not changed -> AGENT_BASE_IMAGE from infra/agent/versions.env, -# i.e. the published toolbox digest that Renovate keeps current. If that is still the -# all-zero placeholder the toolbox has never been published, so it is built and published -# first (this is how the very first run bootstraps). +# THE agent-runner BASE: +# * verify mode -> FROM the toolbox built in this run, linked with a bake `target:` context — +# a pull request holds no registry credential and that toolbox exists in no registry. If the +# toolbox's inputs did not change it is built anyway, as the base. +# * publish mode, toolbox published in this run -> FROM the digest the registry just returned. +# * publish mode otherwise -> AGENT_BASE_IMAGE from infra/agent/versions.env (the published +# toolbox digest Renovate keeps current). If that is still the all-zero placeholder the +# toolbox has never been published, so it is built and published first (the bootstrap). # # LOOP-FREEDOM. Renovate turns each publish into a digest-bump PR. Those bumps land in # platform/pm-agent/ (no image input), conformance/versions.env (no image input) and @@ -46,9 +52,14 @@ set -euo pipefail REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" cd "${REPO_ROOT}" +# shellcheck disable=SC1091 +BK_HOST_OVERRIDE="${BUILDKIT_HOST:-}"; BK_NAME_OVERRIDE="${BUILDKIT_SERVERNAME:-}" +. ./platform/buildkit/versions.env +BUILDKIT_HOST="${BK_HOST_OVERRIDE:-${BUILDKIT_HOST}}" +BUILDKIT_SERVERNAME="${BK_NAME_OVERRIDE:-${BUILDKIT_SERVERNAME}}" + MODE="${MODE:-verify}" IMAGES="${IMAGES:-auto}" -ENGINE="${ENGINE:-docker}" REGISTRY="${REGISTRY:-gitea.home.keithjackson.dev}" NAMESPACE="${NAMESPACE:-home-network}" HEAD_SHA="${HEAD_SHA:-$(git rev-parse HEAD)}" @@ -63,13 +74,15 @@ MOVING_TAG="main" log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; } info() { printf ' %s\n' "$*"; } die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; printf -- '- **ERROR**: %s\n' "$*" >>"${SUMMARY}"; exit 1; } +yn() { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; } case "${MODE}" in verify|publish) ;; *) die "MODE must be verify or publish, got '${MODE}'" ;; esac +for t in git jq curl sha256sum; do command -v "$t" >/dev/null || die "the job image lacks '$t'"; done -# --- inputs per image: a change under any of these rebuilds that image --------------------- -# conformance-runner takes its chainsaw/kubectl/fedora pins from infra/devcontainer/versions.env. -# conformance/versions.env is NOT an input (it only RECORDS the published digest), and must not -# become one: Renovate writes it after every publish, and an input there would loop. +# --- per-image facts ------------------------------------------------------------------------ +# Inputs: a change under any of these rebuilds that image. conformance-runner takes its pins from +# infra/devcontainer/versions.env. conformance/versions.env is NOT an input (it only RECORDS the +# published digest), and must not become one: Renovate writes it after every publish -> a loop. inputs() { case "$1" in toolbox) echo "infra/devcontainer/Containerfile infra/devcontainer/versions.env infra/devcontainer/build-image.sh" ;; @@ -77,16 +90,17 @@ inputs() { conformance-runner) echo "conformance/incluster/Containerfile conformance/incluster/conformance-clone conformance/incluster/build-image.sh infra/devcontainer/versions.env" ;; esac } -PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh" +context_of() { case "$1" in toolbox) echo infra/devcontainer ;; agent-runner) echo infra/agent ;; conformance-runner) echo conformance/incluster ;; esac; } +PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh platform/buildkit/versions.env" declare -A WANT=() BUILT=() DIGEST=() SECS=() NOTE=() changed_any() { # -> 0 if BASE_SHA..HEAD_SHA touches any - # shellcheck disable=SC2086 [ -n "$(git diff --name-only "${BASE_SHA}" "${HEAD_SHA}" -- "$@")" ] } select_images() { + local i if [ "${IMAGES}" != "auto" ]; then for i in ${IMAGES}; do case " ${ALL_IMAGES[*]} " in *" ${i} "*) WANT[$i]=1; NOTE[$i]="requested" ;; *) die "unknown image '${i}'" ;; esac @@ -99,7 +113,6 @@ select_images() { return fi info "diff base ${BASE_SHA} .. ${HEAD_SHA}" - local i for i in "${ALL_IMAGES[@]}"; do # shellcheck disable=SC2046 if changed_any $(inputs "$i"); then WANT[$i]=1; NOTE[$i]="inputs changed"; fi @@ -110,6 +123,114 @@ select_images() { fi } +# --- the builder: pinned buildx + mTLS to rootless buildkitd --------------------------------- +WORK="$(mktemp -d)" +trap 'rm -rf "${WORK}"' EXIT # client key + registry auth live only here, only this long +export BUILDX_CONFIG="${WORK}/buildx" DOCKER_CONFIG="${WORK}/docker" +# Single-manifest images: the admission policies, Renovate and registry-retention.sh all reason +# about one manifest per tag. (Provenance attestations would turn every push into an index.) +export BUILDX_NO_DEFAULT_ATTESTATIONS=1 +mkdir -p "${BUILDX_CONFIG}" "${DOCKER_CONFIG}" "${WORK}/certs" +echo '{}' > "${DOCKER_CONFIG}/config.json" + +setup_builder() { + log "Builder: rootless buildkitd at ${BUILDKIT_HOST} (mTLS)" + BUILDX="${BUILDX_BIN:-}" + if [ -z "${BUILDX}" ]; then + BUILDX="${WORK}/buildx-bin" + curl -fsSL --retry 3 -o "${BUILDX}" \ + "https://github.com/docker/buildx/releases/download/v${BUILDX_VERSION}/buildx-v${BUILDX_VERSION}.linux-amd64" \ + || die "could not download buildx v${BUILDX_VERSION}" + echo "${BUILDX_SHA256} ${BUILDX}" | sha256sum -c --quiet - \ + || die "buildx v${BUILDX_VERSION} does not match the sha256 pinned in platform/buildkit/versions.env — refusing to run it" + chmod +x "${BUILDX}" + fi + info "client: $("${BUILDX}" version)" + + local missing=() + [ -n "${BUILDKIT_CA:-}" ] || missing+=(BUILDKIT_CA) + [ -n "${BUILDKIT_CLIENT_CERT:-}" ] || missing+=(BUILDKIT_CLIENT_CERT) + [ -n "${BUILDKIT_CLIENT_KEY:-}" ] || missing+=(BUILDKIT_CLIENT_KEY) + [ "${#missing[@]}" -eq 0 ] || die "the builder's client certificate is missing: ${missing[*]} not set. + These Actions secrets are copied from the cert-manager Secret buildkit/buildkit-client-ci by + infra/gitea/buildkit-client-secret.sh (an operator action, after platform/buildkit is + deployed). Without them nothing can be built — this fails rather than skipping ON PURPOSE." + umask 077 + printf '%s\n' "${BUILDKIT_CA}" > "${WORK}/certs/ca.pem" + printf '%s\n' "${BUILDKIT_CLIENT_CERT}" > "${WORK}/certs/cert.pem" + printf '%s\n' "${BUILDKIT_CLIENT_KEY}" > "${WORK}/certs/key.pem" + umask 022 + + "${BUILDX}" create --name ci-remote --driver remote \ + --driver-opt "cacert=${WORK}/certs/ca.pem,cert=${WORK}/certs/cert.pem,key=${WORK}/certs/key.pem,servername=${BUILDKIT_SERVERNAME}" \ + "${BUILDKIT_HOST}" >/dev/null + if ! out="$(timeout 90 "${BUILDX}" inspect --bootstrap --builder ci-remote 2>&1)"; then + printf '%s\n' "${out}" >&2 + die "buildkitd is UNREACHABLE at ${BUILDKIT_HOST} — nothing can be built, and this job fails + rather than skipping. In order of likelihood: + 1. platform/buildkit is not deployed yet. Flux applies it only after it merges to main + (Kustomization 'buildkit'); a pull request that ADDS it cannot reach it. + 2. the BUILDKIT_* secrets are stale (the client cert was renewed) or from another CA: + re-run infra/gitea/buildkit-client-secret.sh. + 3. the NetworkPolicy (platform/buildkit/networkpolicy.yaml) does not admit this runner." + fi + printf '%s\n' "${out}" | grep -E '^(Name|Driver|Status|BuildKit version|Platforms):' | sed 's/^/ /' || true +} + +# KEY=VALUE lines from an image's build-image.sh --print-build-args -> a JSON object. +build_args_json() { + local ctx; ctx="$(context_of "$1")" + "./${ctx}/build-image.sh" --print-build-args | jq -Rn '[inputs | select(length>0) | capture("^(?[^=]+)=(?.*)$")] | map({(.k): .v}) | add' +} + +# bake_run +bake_run() { + local f="$1"; shift + "${BUILDX}" bake --builder ci-remote --progress plain -f "$f" \ + --metadata-file "${WORK}/meta.json" "$@" +} + +# Write a bake file with a verify target and a ship target for one image. +# $1 image, $2 agent base mode: "" | "link" | +write_bake() { + local img="$1" base="${2:-}" ctx args f + ctx="$(context_of "$img")" + args="$(build_args_json "$img")" + f="${WORK}/bake-${img}.json" + local contexts='{}' + if [ "${img}" = agent-runner ]; then + if [ "${base}" = link ]; then + # FROM ${AGENT_BASE_IMAGE} resolves to the named context below, i.e. the toolbox target + # of this same bake file — built here, never pushed anywhere. + args="$(jq -c '. + {AGENT_BASE_IMAGE: "toolbox-in-this-run"}' <<<"${args}")" + contexts='{"toolbox-in-this-run": "target:toolbox"}' + else + args="$(jq -c --arg b "${base}" '. + {AGENT_BASE_IMAGE: $b}' <<<"${args}")" + fi + fi + jq -n --arg img "$img" --arg ctx "$ctx" --argjson args "$args" --argjson contexts "$contexts" \ + --arg tbargs "$( [ "${base}" = link ] && build_args_json toolbox || echo '{}')" \ + --arg rev "${HEAD_SHA}" --arg repo "${REGISTRY}/${NAMESPACE}/${img}" \ + --arg sha "${SHA_TAG}" --arg moving "${MOVING_TAG}" --arg mode "${MODE}" ' + {target: ( + { + ($img + "-verify"): {context: $ctx, dockerfile: "Containerfile", target: "verify", + args: $args, contexts: $contexts, output: ["type=cacheonly"]}, + ($img): ({context: $ctx, dockerfile: "Containerfile", target: $img, args: $args, + contexts: $contexts, + labels: {"org.opencontainers.image.revision": $rev}} + + (if $mode == "publish" + then {tags: [($repo + ":" + $sha), ($repo + ":" + $moving)], output: ["type=image,push=true"]} + else {output: ["type=cacheonly"]} end)) + } + + (if ($contexts | has("toolbox-in-this-run")) + then {toolbox: {context: "infra/devcontainer", dockerfile: "Containerfile", target: "toolbox", + args: ($tbargs | fromjson), output: ["type=cacheonly"]}} + else {} end) + )}' > "$f" + echo "$f" +} + # --- registry helpers (publish mode only) ---------------------------------------------------- REG_TOKEN="" registry_token() { @@ -120,8 +241,8 @@ registry_token() { [ -n "${REG_TOKEN}" ] || die "registry returned no bearer token" } # Prints the manifest digest the REGISTRY serves for :; nothing if it is absent (404). -# Any other answer (401, 5xx, no digest header) is fatal: "cannot tell" must never be read as -# "not published", or the immutable-tag guard would be bypassed by an auth hiccup. +# Any other answer is fatal: "cannot tell" must never be read as "not published", or the +# immutable-tag guard would be bypassed by an auth hiccup. registry_digest() { registry_token local hdrs code digest @@ -142,87 +263,50 @@ registry_digest() { esac } -local_ref() { echo "${REGISTRY}/${NAMESPACE}/$1:ci-${HEAD_SHA:0:12}"; } - -build_one() { # [agent base] - local img="$1" base="${2:-}" t0 tag - tag="ci-${HEAD_SHA:0:12}" +# build_one [agent base] — verify, then (publish) build + push + read back. +build_one() { + local img="$1" base="${2:-}" t0 f pushed got existing t0=$(date +%s) - log "BUILD + VERIFY ${img}" - case "${img}" in - toolbox) - ENGINE="${ENGINE}" IMAGE="$(local_ref toolbox)" BUILD_REVISION="${HEAD_SHA}" \ - ./infra/devcontainer/build-image.sh ;; - agent-runner) - ENGINE="${ENGINE}" IMAGE_TAG="${tag}" AGENT_BASE_IMAGE="${base}" BUILD_REVISION="${HEAD_SHA}" \ - ./infra/agent/build-image.sh ;; - conformance-runner) - ENGINE="${ENGINE}" IMAGE_TAG="${tag}" BUILD_REVISION="${HEAD_SHA}" \ - ./conformance/incluster/build-image.sh ;; - esac + f="$(write_bake "$img" "$base")" + log "VERIFY ${img} (Containerfile target: verify, on rootless BuildKit)" + bake_run "$f" "${img}-verify" BUILT[$img]=1 - SECS[$img]=$(( $(date +%s) - t0 )) - info "${img}: built and verified in ${SECS[$img]}s" -} - -push_one() { # -> sets DIGEST[image] to the registry-returned digest - local img="$1" repo src existing pushed got t0 - repo="${REGISTRY}/${NAMESPACE}/${img}" - src="$(local_ref "${img}")" - t0=$(date +%s) - log "PUSH ${img} -> ${repo}:{${SHA_TAG},${MOVING_TAG}}" - existing="$(registry_digest "${img}" "${SHA_TAG}")" - if [ -n "${existing}" ]; then + if [ "${MODE}" = publish ]; then + existing="$(registry_digest "${img}" "${SHA_TAG}")" # Immutable tag: never overwrite. A rebuild is not byte-reproducible, so re-pushing would # silently change what `sha-` means after something may already have pinned it. - die "${repo}:${SHA_TAG} already exists (${existing}). Tags here are immutable; this run must not - replace it. (publish skips images that already exist at this commit BEFORE building them, - so reaching this line means the tag appeared mid-run.)" + [ -z "${existing}" ] || die "${REGISTRY}/${NAMESPACE}/${img}:${SHA_TAG} already exists (${existing}); tags are immutable" + log "BUILD + PUSH ${img} -> ${REGISTRY}/${NAMESPACE}/${img}:{${SHA_TAG},${MOVING_TAG}}" + bake_run "$f" "${img}" + pushed="$(jq -r --arg t "$img" '.[$t]["containerimage.digest"] // empty' "${WORK}/meta.json")" + # The AUTHORITATIVE digest is what the registry serves, read back — not what the client + # reported. They should agree; if they do not, nothing downstream may trust either. + got="$(registry_digest "${img}" "${SHA_TAG}")" + [ -n "${got}" ] || die "pushed ${img}:${SHA_TAG} but the registry does not serve it" + [ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] || die "${img}: client reported ${pushed} but the registry serves ${got}" + [ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \ + || die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})" + DIGEST[$img]="${got}" + info "${img}: registry digest ${got}" fi - "${ENGINE}" tag "${src}" "${repo}:${SHA_TAG}" - "${ENGINE}" tag "${src}" "${repo}:${MOVING_TAG}" - pushed="$("${ENGINE}" push "${repo}:${SHA_TAG}" | tee /dev/stderr | grep -oE 'digest: sha256:[0-9a-f]{64}' | tail -1 | cut -d' ' -f2)" - "${ENGINE}" push "${repo}:${MOVING_TAG}" >/dev/null - # The AUTHORITATIVE digest is what the registry serves, read back — not what the client - # printed. They should agree; if they do not, nothing downstream may trust either. - got="$(registry_digest "${img}" "${SHA_TAG}")" - [ -n "${got}" ] || die "pushed ${repo}:${SHA_TAG} but the registry does not serve it" - [ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] \ - || die "${img}: client reported ${pushed} but the registry serves ${got}" - [ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \ - || die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})" - DIGEST[$img]="${got}" - SECS[$img]=$(( ${SECS[$img]:-0} + $(date +%s) - t0 )) - info "${img}: registry digest ${got}" + SECS[$img]=$(( $(date +%s) - t0 )) + info "${img}: done in ${SECS[$img]}s" } # --- main --------------------------------------------------------------------------------- log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}" -command -v "${ENGINE}" >/dev/null || die "${ENGINE} is not on PATH in the job container" -"${ENGINE}" info >/dev/null 2>&1 \ - || die "the ${ENGINE} daemon is not reachable from the job container, so nothing can be built. - MEASURED ON THIS RUNNER (PR #81, 2026-10-29): act_runner 0.2.11 never gives a job a - daemon. Its DOCKER_HOST is tcp:// (the DinD sidecar), and internal/app/cmd/daemon.go then - forces container.docker_host to '-' (\"can't be mounted\"); separately, with - valid_volumes: [] gitea/act's sanitizeConfig drops EVERY bind, the socket included. - Fixing it means handing workflow code the privileged DinD daemon — an operator security - decision, set out in decisions/2026-10-29-image-pipeline.md. Not a fault in this script." -"${ENGINE}" version 2>/dev/null | sed -n '1,12p' || true -"${ENGINE}" system df || true +select_images if [ "${MODE}" = publish ]; then : "${REGISTRY_TOKEN:?REGISTRY_TOKEN is required in publish mode}" REGISTRY_USER="${REGISTRY_USER:-ci-image-publisher}" - printf '%s' "${REGISTRY_TOKEN}" | "${ENGINE}" login "${REGISTRY}" -u "${REGISTRY_USER}" --password-stdin >/dev/null \ - || die "docker login ${REGISTRY} failed — check REGISTRY_PUSH_TOKEN (scope write:package, owner has package write on the org)" -fi - -select_images - -# Publish idempotence: an image already published at THIS commit is not rebuilt. This is what -# makes a re-run after a half-finished publish safe — and it is the only way a rerun can -# finish, because the immutable sha tag must never be pushed twice. -if [ "${MODE}" = publish ]; then + # buildx (and through it buildkitd, which does the push) reads registry auth from this file. + jq -n --arg r "${REGISTRY}" --arg a "$(printf '%s:%s' "${REGISTRY_USER}" "${REGISTRY_TOKEN}" | base64 -w0)" \ + '{auths: {($r): {auth: $a}}}' > "${DOCKER_CONFIG}/config.json" + chmod 600 "${DOCKER_CONFIG}/config.json" + registry_token + # Publish idempotence: an image already published at THIS commit is not rebuilt. That is what + # makes a re-run after a half-finished publish safe — the immutable tag is never pushed twice. for i in "${ALL_IMAGES[@]}"; do [ -n "${WANT[$i]:-}" ] || continue d="$(registry_digest "${i}" "${SHA_TAG}")" @@ -236,11 +320,11 @@ fi # Decide the agent-runner base (see header). AGENT_BASE="" if [ -n "${WANT[agent-runner]:-}" ]; then - # shellcheck disable=SC1091 PINNED_BASE="$(. ./infra/agent/versions.env; echo "${AGENT_BASE_IMAGE}")" - if [ "${MODE}" = verify ] && [ -z "${WANT[toolbox]:-}" ]; then - WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)" - elif [ "${MODE}" = publish ] && [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then + if [ "${MODE}" = verify ]; then + AGENT_BASE=link + [ -n "${WANT[toolbox]:-}" ] || { WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)"; } + elif [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then case "${PINNED_BASE}" in *@"${ZERO_DIGEST}") WANT[toolbox]=1; NOTE[toolbox]="never published; required as agent-runner's base" ;; *@sha256:*) AGENT_BASE="${PINNED_BASE}" ;; @@ -249,37 +333,30 @@ if [ -n "${WANT[agent-runner]:-}" ]; then fi fi -yn() { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; } for i in "${ALL_IMAGES[@]}"; do info "plan: ${i}: $(yn "${WANT[$i]:-}" BUILD skip) (${NOTE[$i]:-inputs unchanged})"; done - -if [ -n "${WANT[toolbox]:-}" ]; then - build_one toolbox - if [ "${MODE}" = publish ]; then push_one toolbox; fi +if [ -z "${WANT[toolbox]:-}${WANT[agent-runner]:-}${WANT[conformance-runner]:-}" ]; then + info "nothing to build" +else + setup_builder fi + +if [ -n "${WANT[toolbox]:-}" ]; then build_one toolbox; fi if [ -n "${WANT[agent-runner]:-}" ]; then if [ -z "${AGENT_BASE}" ]; then - if [ "${MODE}" = publish ]; then - AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}" - else - AGENT_BASE="$(local_ref toolbox)" - fi + AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}" fi - info "agent-runner base: ${AGENT_BASE}" + info "agent-runner base: $( [ "${AGENT_BASE}" = link ] && echo "the toolbox built in this run (bake target: context)" || echo "${AGENT_BASE}")" build_one agent-runner "${AGENT_BASE}" - if [ "${MODE}" = publish ]; then push_one agent-runner; fi -fi -if [ -n "${WANT[conformance-runner]:-}" ]; then - build_one conformance-runner - if [ "${MODE}" = publish ]; then push_one conformance-runner; fi fi +if [ -n "${WANT[conformance-runner]:-}" ]; then build_one conformance-runner; fi # --- summary ------------------------------------------------------------------------------ { echo "## Images (${MODE})" echo - echo "commit \`${HEAD_SHA}\`" + echo "commit \`${HEAD_SHA}\` — builder: rootless BuildKit \`${BUILDKIT_HOST}\` (mTLS)" echo - echo "| image | built + verified | seconds | published digest (registry-returned) | note |" + echo "| image | verify target built | seconds | published digest (registry-returned) | note |" echo "|---|---|---|---|---|" for i in "${ALL_IMAGES[@]}"; do d="${DIGEST[$i]:-}" diff --git a/clusters/base/buildkit.yaml b/clusters/base/buildkit.yaml new file mode 100644 index 0000000..dedb3e8 --- /dev/null +++ b/clusters/base/buildkit.yaml @@ -0,0 +1,21 @@ +--- +# Flux Kustomization -> ./platform/buildkit — rootless BuildKit, the image builder CI uses +# (.gitea/workflows/images.yaml). Its own Kustomization so a stuck Certificate or a failing +# builder can never stall any other tier (Flux health-checks Certificates under wait:true). +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: buildkit + namespace: flux-system +spec: + interval: 10m + retryInterval: 2m + timeout: 5m + prune: true + wait: true + dependsOn: + - name: platform # cert-manager CRDs (Issuer/Certificate) come from its HelmRelease + sourceRef: + kind: GitRepository + name: flux-system + path: ./platform/buildkit diff --git a/clusters/base/kustomization.yaml b/clusters/base/kustomization.yaml index 5f816bf..d56e149 100644 --- a/clusters/base/kustomization.yaml +++ b/clusters/base/kustomization.yaml @@ -98,4 +98,5 @@ resources: # deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon. # See clusters/base/kured.yaml and platform/kured/helmrelease.yaml. - kured.yaml # Flux Kustomization -> ./platform/kured (dependsOn platform) + - buildkit.yaml # Flux Kustomization -> ./platform/buildkit (dependsOn platform) - cluster-vars.yaml # overlay knob surface (base defaults) diff --git a/conformance/incluster/Containerfile b/conformance/incluster/Containerfile index 35419dc..e864869 100644 --- a/conformance/incluster/Containerfile +++ b/conformance/incluster/Containerfile @@ -98,7 +98,10 @@ RUN set -eux; \ # --- runtime ----------------------------------------------------------------- ARG FEDORA_VERSION=44 -FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION} +# Stage `conformance-runner` is the image that ships; stage `verify` (end of file) is a TEST stage +# built FROM it that only runs assertions and is never pushed. build-image.sh and the CI +# pipeline build `--target verify` to prove it and `--target conformance-runner` for the image. +FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION} AS conformance-runner ARG CHAINSAW_VERSION= ARG KUBECTL_VERSION= @@ -198,3 +201,86 @@ WORKDIR /workspace # prepend something the policy cannot see. `bash` is the interactive fallback for a human # debugging the image locally. CMD ["/bin/bash"] + +# ============================================================================= +# verify — a TEST stage, never shipped. Every assertion conformance/incluster/build-image.sh used +# to make with `run --user 65532 --read-only --tmpfs /workspace` now runs here, at build time, +# because CI builds on rootless BuildKit where no container runtime exists. +# +# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE: +# * uid 65532 ............ exact (USER inherited from the conformance-runner stage). +# * no network ........... exact: RUN --network=none, and the first check proves it. +# * --read-only rootfs ... NOT reproducible (a RUN step always has a writable layer). +# Approximated by permission: everything uid 65532 can write must lie +# under /workspace, /tmp or /var/tmp. In the Job only /workspace (and +# /results) are writable — the image points HOME and TMPDIR there — and +# readOnlyRootFilesystem is enforced by +# platform/kyverno/admission/conformance-trigger-job-shape.yaml. +# ============================================================================= +FROM conformance-runner AS verify +RUN --network=none /bin/bash -euo pipefail <<'VERIFY' +echo "--- no network (the build-time stand-in for the Job's isolation) ---" +if getent hosts gitea.home.keithjackson.dev >/dev/null 2>&1 || timeout 3 bash -c '/dev/null; then + echo "UNEXPECTED: network reachable in the verify stage"; exit 1 +fi +echo "confirmed: no network" +echo "--- id ---"; id +echo "--- HOME ---"; echo "HOME=${HOME}" +echo "--- bash ---"; bash --version | head -1 +echo "--- python3 ---"; python3 --version +echo "--- git ---"; git --version +echo "--- kubectl ---"; kubectl version --client +echo "--- chainsaw ---"; chainsaw version +echo "--- baked versions ---"; cat /etc/conformance-runner-versions +echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---" +stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone +if [ -w /usr/local/bin/conformance-clone ]; then + echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1 +fi +echo "confirmed: not writable by uid $(id -u)" +# Each of these MUST fail. Assert the non-zero exit as well as the message: a check that only +# greps stderr would pass a build where the guard had stopped rejecting. +expect_fail() { + local label="$1"; shift + local out rc=0 + out="$(env "$@" 2>&1)" || rc=$? + echo "--- ${label} ---" + printf "%s\n" "${out}" | head -2 + if [ "${rc}" -eq 0 ]; then + echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1 + fi + echo "(rejected, exit ${rc})" +} +expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \ + conformance-clone --layer l2 +expect_fail "conformance-clone rejects a missing SHA" \ + conformance-clone +expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \ + CONFORMANCE_GIT_SHA=main conformance-clone +expect_fail "conformance-clone rejects an UNRENDERED placeholder" \ + CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone +expect_fail "conformance-clone rejects a HOME off the workspace volume" \ + CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \ + HOME=/etc conformance-clone +expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \ + CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \ + CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone +expect_fail "conformance-clone refuses a clone target off the workspace volume" \ + CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \ + CONFORMANCE_SRC=/etc/src conformance-clone +echo "--- the root filesystem is not writable by the runner (expected: a failure) ---" +if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi +echo "confirmed: / is not writable" +echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---" +stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \ + | grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)" +if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf ' %s\n' ${stray} | head -20; exit 1; fi +echo "confirmed: no writable path outside /workspace, /tmp, /var/tmp" +echo "--- HOME is writable (the workspace emptyDir) ---" +touch "${HOME}/.probe" || { echo "UNEXPECTED: ${HOME} not writable"; exit 1; } +echo "confirmed: ${HOME} is writable" +echo "--- uid is non-zero ---" +[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; } +echo "confirmed: running as uid $(id -u), not root" +echo "CONFORMANCE-RUNNER VERIFIED" +VERIFY diff --git a/conformance/incluster/README.md b/conformance/incluster/README.md index 3d306dc..215154b 100644 --- a/conformance/incluster/README.md +++ b/conformance/incluster/README.md @@ -61,10 +61,12 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02). > digest-pinned to the conformance-runner image, `command` exactly > `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and > `env`/`envFrom` were the same shape of hole and are closed in the same policy. -- `build-image.sh` — builds the above and **verifies the toolchain by running it** inside the - image as uid 65532 with `--read-only`. It **never pushes**, and neither does anyone from a - workstation: CI (`.gitea/workflows/images.yaml`) runs this same script, and on a push to - `main` publishes the verified image under `sha-` and `main`. Renovate then moves +- `build-image.sh` — builds the above and **verifies the toolchain by running it**: the + Containerfile's `verify` stage executes every tool and every `conformance-clone` refusal as + uid 65532 with no network, at build time. It **never pushes**, and neither does anyone from a + workstation: CI (`.gitea/workflows/images.yaml`) builds the same Containerfile and `verify` + stage on rootless BuildKit (`platform/buildkit/`), and on a push to `main` publishes the + verified image under `sha-` and `main`. Renovate then moves `CONFORMANCE_RUNNER_IMAGE_DIGEST` in `conformance/versions.env` onto the registry-returned digest in a pull request (that file is a sensitive path, so the bump needs an administrator's `/approve-override`). diff --git a/conformance/incluster/build-image.sh b/conformance/incluster/build-image.sh index c836484..c5b1063 100755 --- a/conformance/incluster/build-image.sh +++ b/conformance/incluster/build-image.sh @@ -3,7 +3,8 @@ # conformance/incluster/build-image.sh — build the pinned `conformance-runner` # image and prove it works. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1. # -# ./build-image.sh # build + verify + print the image id +# ./build-image.sh # build the verify target, then the image +# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes) # ./build-image.sh --no-verify # build only (skip the in-image tool proof) # ./build-image.sh --verify-clone # ALSO really clone this repo with conformance-clone. # # Opt-in because it needs the Gitea host reachable; the @@ -13,13 +14,12 @@ # IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this) # # THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either. -# Publishing is done by CI: .gitea/workflows/images.yaml calls this script, and on a push to -# `main` it pushes what this script built AND verified to the Gitea registry, records the -# registry-returned digest in the job summary, and Renovate then opens the pull request that -# moves conformance/versions.env onto that digest. So this script's job is to build and to -# prove the toolchain works INSIDE the image as the non-root uid the policy demands — the same -# proof locally and in CI. It runs under ENGINE=docker in CI (Docker-in-Docker), so nothing -# below may be podman-only. +# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on +# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and +# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest +# in the job summary; Renovate then opens the pull request that moves conformance/versions.env +# onto that digest. The proof that the image works is the Containerfile's `verify` stage, which +# this script (podman locally) and CI both build. # # VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the # single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same @@ -39,9 +39,11 @@ TAG_OVERRIDE="${IMAGE_TAG:-}" VERIFY=1 VERIFY_CLONE=0 +PRINT_ARGS=0 for arg in "$@"; do case "$arg" in --no-verify) VERIFY=0 ;; + --print-build-args) PRINT_ARGS=1 ;; --verify-clone) VERIFY_CLONE=1 ;; -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;; *) echo "unknown argument: $arg" >&2; exit 2 ;; @@ -52,8 +54,6 @@ log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; } info() { printf ' %s\n' "$*"; } die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; } -command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)" - # --- pins: sourced, never invented ------------------------------------------- TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env" [ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here" @@ -63,6 +63,13 @@ source "${TOOLCHAIN_ENV}" : "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}" : "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}" +# THE build args, in ONE place; `--print-build-args` hands them to the CI pipeline, which builds +# this Containerfile on rootless BuildKit. +BUILD_ARGS=("CHAINSAW_VERSION=${CHAINSAW_VERSION}" "KUBECTL_VERSION=${KUBECTL_VERSION}" "FEDORA_VERSION=${FEDORA_VERSION}") +if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi + +command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)" + # --- image name: fixed by the admission policy ------------------------------- # platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY # gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path @@ -84,86 +91,27 @@ info " FEDORA ${FEDORA_VERSION}" LABEL_ARGS=() [ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}") -"${ENGINE}" build "${LABEL_ARGS[@]}" \ - --file "${HERE}/Containerfile" \ - --tag "${IMAGE_REF}" \ - --build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \ - --build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \ - --build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \ - "${HERE}" +ARG_FLAGS=() +for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done # --- verify: the point of the whole exercise --------------------------------- -# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the -# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root -# filesystem — the shape the admission policy forces. A pass here is the only evidence -# this image is usable; a build that merely succeeded is not. +# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So the proof is that +# the tools EXECUTE, as the NON-ROOT uid, with no network, that conformance-clone is not writable +# by that uid and refuses every hostile input, and that nothing outside /workspace is writable. +# That proof is the Containerfile's `verify` stage, built here and in CI alike (CI has no +# container runtime — rootless BuildKit only). See the stage's header for what a build step can +# and cannot reproduce of the Job's shape. A build that merely succeeded is not evidence. if [ "${VERIFY}" -eq 1 ]; then - log "Verifying the toolchain inside the image (non-root, read-only rootfs)" - "${ENGINE}" run --rm \ - --user 65532:65532 \ - --read-only \ - --tmpfs /workspace:rw,mode=1777 \ - --entrypoint "" \ - "${IMAGE_REF}" \ - /bin/bash -euo pipefail -c ' - echo "--- id ---"; id - echo "--- HOME ---"; echo "HOME=${HOME}" - echo "--- bash ---"; bash --version | head -1 - echo "--- python3 ---"; python3 --version - echo "--- git ---"; git --version - echo "--- kubectl ---"; kubectl version --client - echo "--- chainsaw ---"; chainsaw version - echo "--- baked versions ---"; cat /etc/conformance-runner-versions - echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---" - stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone - if [ -w /usr/local/bin/conformance-clone ]; then - echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1 - fi - echo "confirmed: not writable by uid $(id -u)" - # Each of these MUST fail. Assert the non-zero exit as well as the message: a check - # that only greps stderr would pass a build where the guard had stopped rejecting. - # (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the - # expected failure — without it the verify dies at the first negative case.) - expect_fail() { - local label="$1"; shift - local out rc=0 - out="$(env "$@" 2>&1)" || rc=$? - echo "--- ${label} ---" - printf "%s\n" "${out}" | head -2 - if [ "${rc}" -eq 0 ]; then - echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1 - fi - echo "(rejected, exit ${rc})" - } - expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \ - conformance-clone --layer l2 - expect_fail "conformance-clone rejects a missing SHA" \ - conformance-clone - expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \ - CONFORMANCE_GIT_SHA=main conformance-clone - expect_fail "conformance-clone rejects an UNRENDERED placeholder" \ - CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone - expect_fail "conformance-clone rejects a HOME off the workspace volume" \ - CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \ - HOME=/etc conformance-clone - expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \ - CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \ - CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone - expect_fail "conformance-clone refuses a clone target off the workspace volume" \ - CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \ - CONFORMANCE_SRC=/etc/src conformance-clone - echo "--- rootfs is read-only (expected: a failure) ---" - if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi - echo "confirmed: / is read-only" - echo "--- HOME is writable (the workspace emptyDir) ---" - touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable" - echo "--- uid is non-zero ---" - [ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root" - ' + log "Verifying the toolchain (Containerfile target: verify)" + "${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}" log "Toolchain verified" fi +"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target conformance-runner \ + --file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}" + # --- verify the clone helper against the REAL repository (opt-in: needs network) ---- +# LOCAL ONLY: this one `run`s the image, which needs a container engine, so CI does not do it. # The negative cases above prove conformance-clone refuses bad input. This proves the # positive case: it lands the EXACT commit asked for and can prove it did. Without this, the # rev-parse assertion is only an unexercised line of shell. diff --git a/conformance/tests/l0-static/alert-rules-unit-tested/check.sh b/conformance/tests/l0-static/alert-rules-unit-tested/check.sh index ae7c593..47b6e29 100755 --- a/conformance/tests/l0-static/alert-rules-unit-tested/check.sh +++ b/conformance/tests/l0-static/alert-rules-unit-tested/check.sh @@ -250,8 +250,8 @@ check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text, # The build recipe moved from dev.sh into build-image.sh on 2026-10-29, so that `dev.sh build` # and the CI image pipeline (.gitea/workflows/images.yaml) share ONE recipe. Check the recipe. BUILDSH = ROOT / "infra/devcontainer/build-image.sh" -check(BUILDSH.is_file() and "--build-arg PROMTOOL_VERSION=" in BUILDSH.read_text(), - "infra/devcontainer/build-image.sh does not pass --build-arg PROMTOOL_VERSION — the ARG " +check(BUILDSH.is_file() and '"PROMTOOL_VERSION=$PROMTOOL_VERSION"' in BUILDSH.read_text(), + "infra/devcontainer/build-image.sh does not pass PROMTOOL_VERSION in its BUILD_ARGS — the ARG " "would be empty and the build would fetch a bogus URL") INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh" diff --git a/decisions/2026-10-29-image-pipeline.md b/decisions/2026-10-29-image-pipeline.md index 32e5a68..40c3909 100644 --- a/decisions/2026-10-29-image-pipeline.md +++ b/decisions/2026-10-29-image-pipeline.md @@ -1,7 +1,7 @@ --- date: 2026-10-29 branch: feat/image-pipeline -title: "Images are built, verified and published by CI; Renovate carries the digests into git" +title: "Images are built, verified and published by CI on rootless BuildKit; Renovate carries the digests into git" --- ## What changed @@ -11,127 +11,177 @@ the codebase, never to a one-off workstation build/push. Before this branch the package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and both `build-image.sh` scripts said "publishing is an operator action". -- **`.gitea/workflows/images.yaml`** + **`.gitea/workflows/scripts/images.sh`** (the logic). - - Pull request: build and verify only. No push, and the step references no secret at all. - - Push to `main`, or `workflow_dispatch` on `main`: build, verify, then push to - `gitea.home.keithjackson.dev/home-network/{toolbox,agent-runner,conformance-runner}`. - Tags are `sha-<12 hex>` (immutable: the driver refuses to overwrite one) and `main` (moving). - The digest is read back from the registry with a HEAD on the manifest, compared with what the - client printed, and written to the job summary. - - Only images whose inputs changed are built. A re-run on the same commit skips images that are - already published. - - agent-runner builds FROM the toolbox built in the same run (on main, FROM its just-pushed - digest). Otherwise: - - in verify mode the toolbox is built anyway as its base, because a PR holds no credential; - - in publish mode it uses `AGENT_BASE_IMAGE` from `infra/agent/versions.env`; - - if that is still the zero placeholder, the toolbox is built and published first. This is - how the first run on main bootstraps. - - An always-run step prunes the DinD image store. -- **One build recipe per image, reused by CI.** - - New `infra/devcontainer/build-image.sh`. `dev.sh build` now delegates to it. It has an - in-image verification step that the toolbox never had: every tool is on PATH, and - promtool, kubectl, chainsaw, kyverno and sops report their pins. - - `infra/agent/build-image.sh` and `conformance/incluster/build-image.sh` were made - engine-neutral, because CI runs `ENGINE=docker` against DinD: - - `podman image exists` became `image inspect`, with a pull fallback; - - `{{.Digest}}` and `--digestfile` are gone. No local digest is printed, since only the - registry's digest is meaningful; - - `probe-validation.sh` is fed over **stdin** instead of a bind mount. Under DinD a bind - source is resolved on the daemon's filesystem, which would silently mount an empty - directory; - - `probe-validation.sh` itself gained `${BASH_SOURCE[0]:-$0}` so it survives `bash -s` - under `set -u`. - - `AGENT_BASE_IMAGE` and `IMAGE_TAG` can be overridden from the environment, and - `BUILD_REVISION` adds an `org.opencontainers.image.revision` label. - - The "publishing is an operator action" text was replaced throughout. -- **Digest references.** Each now carries a `# renovate-digest: image= tag=main` marker - on the line above it: - - `platform/pm-agent/deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`), which share - one dependency, so they bump in one PR; - - `infra/agent/versions.env` `AGENT_BASE_IMAGE`, which was `home-k3s-dev:local` and is now - the published toolbox by digest (zero placeholder until the first publish); - - `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to the - zero placeholder because Renovate can only replace a digest it can match. +### The builder: rootless BuildKit — DECISION MADE (operator, on PR #81) - The admission policies are **untouched**; the `@sha256:` form is preserved. +**The problem it answers, measured on this PR's first CI run (images run 325, failed in 6 s):** a +CI job on this runner gets **no Docker daemon at all**, for two reasons, both in the deployed +versions' source: +- act_runner v0.2.11 `internal/app/cmd/daemon.go` sets `container.docker_host` to `-` ("never + mount into jobs") whenever `DOCKER_HOST` is `tcp://`, which it is here (the DinD sidecar); +- gitea/act v0.261.2 `pkg/container/docker_run.go` `sanitizeConfig` drops **every** bind, + socket included, when `valid_volumes` is empty, which it is (the runner has no config file). + +Giving jobs the DinD daemon would have made every branch push root on the node, because that +daemon is `privileged: true`. + +**The decision:** no job may reach a privileged daemon, and images are built by a **rootless +BuildKit** daemon over mTLS. The runner (`apps/wave-1/gitea-actions.yaml`, DOCKER_HOST, +valid_volumes) is **untouched**. + +- **`platform/buildkit/`** — a new hand-managed component in its own namespace `buildkit`. The + runner manifest is renderer-generated (`catalog/render.py` EXTRA_UNITS), and the builder has no + business sitting next to a privileged container. Flux reconciles it as the new Kustomization + `buildkit` (`clusters/base/buildkit.yaml`, `dependsOn: platform`). + - **Deployment:** `moby/buildkit:v0.34.1-rootless@sha256:f68752a5…0c4f89`, pinned by index + digest. The securityContext, every line measured (see *What I verified*): + - `privileged: false`; + - uid/gid 1000, `runAsNonRoot`; + - `capabilities: drop ALL, add SETUID, SETGID`; + - `allowPrivilegeEscalation: true`; + - `seccompProfile: Unconfined` and `appArmorProfile: Unconfined`; + - `readOnlyRootFilesystem: true`, with five emptyDirs. + + Other settings: + - `--oci-worker-no-process-sandbox`; + - no ServiceAccount token; + - memory limit 4Gi, which covers build steps too; + - cache emptyDir with `sizeLimit: 6Gi`, and GC capped at 4 GB in `buildkitd.toml` + (ConfigMap). + - **mTLS via cert-manager** (`certificates.yaml`): a self-signed CA used only here, the server + cert `buildkitd-tls`, and the client cert `buildkit-client-ci`. buildkitd verifies every + client against that CA. + - **Service:** ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`. + - **NetworkPolicies:** + - default-deny; + - ingress on 1234 **only from the act-runner pod** (`gitea-actions`, + `app.kubernetes.io/name=act-runner`); + - egress: DNS, world :443/:80 for downloads, and the `ingress` entity (the Gateway) for the + registry push. + - `versions.env`: the CI-side client, `buildx` v0.38.1 with the sha256 from its + `checksums.txt`, plus the address and certificate name CI dials. +- **How DinD job containers appear to Cilium (measured, read-only).** + `iptables -t nat -S POSTROUTING` in the dind container shows + `-A POSTROUTING -s 172.18.0.0/16 ! -o br-… -j MASQUERADE` (and the same for docker0). So a job's + connection leaves with the **act-runner pod's IP, and therefore that pod's Cilium identity**. + That makes "only the act-runner pod" expressible and correct, but it cannot tell one job from + another, or a job from act_runner itself. Per-job authentication is the client certificate. +- **Admission: no Kyverno exception needed.** + - Every ClusterPolicy actually applied (`platform-config/kustomization.yaml`; live: + `conformance-trigger-job-shape`, `label-cdi-worker-pods`, `pm-agent-job-shape`, + `require-auth-mode-label`, `require-default-deny-baseline`, `require-egress-class-label`, + `restrict-host-namespaces`, `secret-provenance`, `validate-default-deny-shape`) matches only + namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`, or only its own + namespace in the job-shape policies' case. `buildkit` carries no such label. + - There are no namespaced Kyverno Policies and no cluster-wide PSA default (both measured). + - The namespace is deliberately left without a PSA label, because PSS `baseline` forbids + `Unconfined` seccomp and AppArmor. + - `validate-default-deny-shape` does apply to the namespace's default-deny NetworkPolicy, and + its shape (both directions) satisfies it. +- **Unprivileged user namespaces on the nodes (measured, read-only, via Ansible `sysctl`).** On + all three Debian 13.7 nodes, kernel `7.1.8+deb13-amd64`: + - `kernel.unprivileged_userns_clone = 1`; + - `user.max_user_namespaces = 127206`; + - no `kernel.apparmor_restrict_unprivileged_userns` (an Ubuntu-only key); + - AppArmor enabled (`Y`). + + So this is a satisfied prerequisite, not an open one. +- **`infra/gitea/buildkit-client-secret.sh`** (operator action): copies `ca.crt`, `tls.crt` and + `tls.key` from the Secret `buildkit/buildkit-client-ci` into the Actions secrets `BUILDKIT_CA`, + `BUILDKIT_CLIENT_CERT` and `BUILDKIT_CLIENT_KEY`. Before writing anything it checks that: + - the cert verifies against the CA; + - the cert is a client-auth cert; + - the key matches the cert. + + It never prints a value. `--check` reports whether the stored copy is older than the current + certificate. + +### Verification moved into the Containerfiles + +`docker run` cannot exist on BuildKit, so each Containerfile now names its shipped stage after the +image (`toolbox`, `agent-runner`, `conformance-runner`) and ends in a **`verify` test stage**, +`FROM AS verify`. That stage runs every previous assertion at build time, as the image's +non-root uid (65532 for agent-runner and conformance-runner), in `RUN --network=none`. It is never +pushed. +- **Every existing assertion is preserved**, and the toolbox gained one it never had: every tool + on PATH, and five tools report their pinned versions. +- `probe-validation.sh` is `COPY`ed into the verify stage only, so it never ships. +- **Bug fixed on the way:** in the old `docker run` block, `cmd | grep -q … && echo ok` and + `[ … ] && echo` could never fail the script under `set -e` + `pipefail`. The "entrypoints refuse + arguments", "not root", "no API key baked in" and "HOME writable" checks asserted nothing. They + are real assertions now. +- **What a build step cannot reproduce, and how it is approximated** (also written in each stage's + header): + - **uid:** exact. + - **`--network none`:** exact, and each stage *proves* it. Its first check fails if DNS or a + TCP connection works. A negative control without `--network=none` fails that check. + - **`--read-only` rootfs:** not reproducible, because a RUN step always has a writable layer. + It is approximated by permission: + - `touch /should-fail` must fail; + - a `find / -xdev -writable ! -type l` inventory, run as the uid, must lie entirely under + `/workspace`, `/tmp` or `/var/tmp`. `! -type l` was needed: Fedora's `/usr/tmp` symlink + to `/var/tmp` tripped it. + + The real control remains the pods' `readOnlyRootFilesystem`, which the admission policies + enforce. + - **`--tmpfs /workspace`:** approximated by the image's own `/workspace`, owned by 65532, which + is exactly what the pod's emptyDir replaces. + + The opt-in `--verify-clone` still `run`s the image, so it remains local-only. +- **`build-image.sh` (all three):** + - verify is `build --target verify` and the image is `build --target `; + - a new `--print-build-args` prints the pins as `KEY=VALUE`, which is what CI consumes. The + arg list therefore exists once; + - the local podman path still works. +- The L0 check `alert-rules-unit-tested` now looks for `"PROMTOOL_VERSION=$PROMTOOL_VERSION"` in + the toolbox's build-arg list. + +### The pipeline (`.gitea/workflows/images.yaml` + `scripts/images.sh`) + +- **The builder.** The driver downloads the pinned `buildx` and refuses a checksum mismatch. It + writes the client cert from the `BUILDKIT_*` secrets to a 0600 temp dir that is removed on exit, + creates a `remote` builder, and runs `buildx inspect --bootstrap`. +- **It fails loudly, never skips.** It stops with a clear message if: + - the secrets are absent; + - buildkitd is unreachable. The message lists the causes in order: not deployed yet (Flux + applies it only after merge), stale certificate, NetworkPolicy. +- **Pull request:** `buildx bake -verify` for changed images. Nothing is pushed and no + registry credential is referenced. The `BUILDKIT_*` builder secrets *are* given to PR jobs, + because they must build. + - agent-runner is built FROM the toolbox built **in the same run** through a bake `target:` + context. That toolbox exists in no registry, so the toolbox is built as its base even when + its own inputs did not change. +- **Push to main / dispatch on main:** for each image, the verify target first, and only then the + image itself with `push=true`. Tags are `sha-<12 hex>` (immutable) and `main` (moving). + - The digest is read back from the registry and compared with the client's metadata. + - agent-runner builds FROM the toolbox digest just pushed, or else from the pinned + `AGENT_BASE_IMAGE` (the zero placeholder bootstraps). + - `BUILDX_NO_DEFAULT_ATTESTATIONS=1` keeps one manifest per tag, which is what admission, + Renovate and the retention script assume. +- **Unchanged from the first iteration:** change detection, publish idempotence, + `REGISTRY_PUSH_TOKEN` and its preflight, the Renovate markers, and `registry-retention.sh`. The + DinD prune step is gone, because nothing is built in DinD now. + +### Unchanged from the first iteration (still true) + +- **Digest references** carry `# renovate-digest: image= tag=main` markers: + - `platform/pm-agent/deployment.yaml` and `configmap.yaml`; + - `infra/agent/versions.env` `AGENT_BASE_IMAGE`, now the published toolbox by digest, zero + until the first publish; + - `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to zero. + + The admission policies are untouched. - **`renovate.json5`**: - - a new `custom.regex` manager for the markers; + - the `custom.regex` marker manager; - `enabledManagers` `regex` → `custom.regex`; - - a packageRule that disables the `kubernetes` manager for these three repos. Otherwise it - would also see the digest-only lines and look them up as `latest`; - - an `own-image` label; - - `ignorePaths` narrowed from `conformance/**` to its subdirectories. `ignorePaths` applies - to custom managers too, so the blanket glob hid `conformance/versions.env`. -- **`renovate.yaml`**: `renovate-version: 44.167.0` pinned, plus a `RENOVATE_HOST_RULES` docker - rule for the Gitea registry, using the same `RENOVATE_TOKEN`, which now also needs - `read:package`. See below for why the version pin is a real fix and not tidiness. -- **`infra/gitea/registry-retention.sh`**: dry run by default, `--apply` to delete. This is an - operator action. Per image it keeps: - - `main` and `latest`; - - the 10 newest `sha-` tags; - - every tag whose digest is pinned anywhere in `origin/main`; - - every digest-named version. -- **`infra/gitea/conformance-trigger-kubeconfig.sh`**: the "runner image is digest-pinned" - precondition grepped the literal line `CONFORMANCE_RUNNER_IMAGE="${…_REPO}@${…_DIGEST}"` - for 64 hex digits, so it **could never pass**. It now sources the file, evaluates the value, - and rejects the zero placeholder. -- **`conformance/tests/l0-static/alert-rules-unit-tested/check.sh`**: its "the toolbox build - passes `--build-arg PROMTOOL_VERSION`" assertion now reads `infra/devcontainer/build-image.sh` - instead of `dev.sh`. The gate caught this on the first run, because the recipe had moved. -- **Docs**: `platform/pm-agent/README.md`, `runbooks/pm-agent.md` (two rows), - `docs/plans/pm-agent-issue-to-pr.md` (follow-up 1), `conformance/incluster/README.md`, - `infra/devcontainer/README.md`, and a new "How images are built" section in - `.gitea/workflows/README.md`. + - the `kubernetes` manager disabled for these three repos; + - `ignorePaths` narrowed so `conformance/versions.env` is visible. -### BLOCKER — the runner cannot build images, and fixing that is a security decision - -**Measured on this PR's own CI run (images run 325): the job failed in 6 s** with -`failed to connect to the docker API at unix:///var/run/docker.sock ... no such file`. The job -container is given **no Docker daemon at all**, for two independent reasons, both read from the -exact versions deployed: - -- act_runner **v0.2.11** `internal/app/cmd/daemon.go`: when `DOCKER_HOST` has a scheme other than - `unix`/`npipe` — here `tcp://localhost:2376`, the DinD sidecar — it sets - `cfg.Container.DockerHost = "-"`, which means "never mount into job containers". -- gitea/act **v0.261.2** (act_runner's `go.mod` replace) `pkg/container/docker_run.go` - `sanitizeConfig`: with `valid_volumes` empty (the runner has no config file, so the default - `[]`), `hostConfig.Binds = []string{}`. **Every** bind is dropped, the daemon socket included, - and so is any `volumes:` a workflow asks for. - -I assumed the opposite while designing (from act's `getDockerDaemonSocketMountPath`, which never -gets the chance to run). That was wrong, and the PR run is what showed it. - -**Why I did not fix it in this PR.** Any fix gives workflow code the DinD daemon. That daemon is -`privileged: true`, so `docker run --privileged -v /:/host` from a job is **root on the node**. -Today a malicious workflow — for example one pushed by a prompt-injected pm-agent worker, which -holds a repo write token — runs in an unprivileged container and holds no cluster credential. -After the fix it does not. That is a change to the security boundary, and it is the operator's -call. It also cannot be tested from a PR anyway: the runner only changes after merge + Flux. - -**Options, for the operator:** - -1. **Socket to every job (smallest change):** in `apps/wave-1/gitea-actions.yaml`, - - mount a shared `emptyDir` at `/var/run` in both the `dind` sidecar and the `runner`, so - dockerd's own `unix:///var/run/docker.sock` is visible at the same path in both; - - set the runner's `DOCKER_HOST=unix:///var/run/docker.sock`, and drop `DOCKER_TLS_VERIFY` - and `DOCKER_CERT_PATH`, because TLS over a unix socket breaks the client; - - add an act_runner config file (`CONFIG_FILE`) with `container.valid_volumes: - ["/var/run/docker.sock"]` and `runner.file: /data/.runner`, so the existing registration - is kept. - - Consequence: any workflow on any branch can reach node root. -2. **Option 1 on a SECOND runner** with its own label (e.g. `image-builder`) and its own DinD, - with `images.yaml` `runs-on` that label. The gate runner stays socket-less, and image builds - stop competing with the gate for the single slot and its 7 GB node disk. **It is not a - security boundary**: Gitea cannot restrict which workflow may target a label, so any branch - could still `runs-on: image-builder`. -3. **A builder that is not a privileged daemon**, e.g. a rootless BuildKit sidecar exposed to - jobs. That bounds a job to BuildKit's rootless sandbox instead of node root. It needs the - verification steps rewritten, because they *run* the image (as uid 65532, `--read-only`), - which BuildKit alone does not do. This is a larger piece of work. - -Nothing in this PR has to change for options 1 or 2 except `runs-on` for option 2. The driver -already produced verified images with a stock docker 27 CLI (see *What I verified*). + `renovate.yaml` pins `renovate-version: 44.167.0` and adds a docker `RENOVATE_HOST_RULES` entry + using `RENOVATE_TOKEN`, which now needs `read:package`. +- **`infra/gitea/registry-retention.sh`** (dry run by default; operator action); the + `conformance-trigger-kubeconfig.sh` precondition fix; and the docs (pm-agent README, runbook, + plan, conformance README, devcontainer README, workflows README). The docs are updated again + for BuildKit, and there is a new `platform/buildkit/README.md`. ### Decisions and pushback @@ -147,7 +197,8 @@ already produced verified images with a stock docker 27 CLI (see *What I verifie - `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3. The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight - pattern. + pattern. With BuildKit the credential reaches the daemon through the buildx session (a + 0600 `config.json` in the job's temp dir), never through the cluster. - **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate: - `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` → `custom.regex`; @@ -195,89 +246,112 @@ already produced verified images with a stock docker 27 CLI (see *What I verifie ## What I verified -- **Local gate, pinned toolbox, from this worktree, committed HEAD.** Command: - `ENGINE= DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`. - The shim adds the `.git` mount. Result: see the PR body for the rc and stage list. They are - recorded there rather than here because this entry is part of the tree being gated. -- **Local podman builds through the CI driver**, run as - `MODE=verify ENGINE=podman BASE_SHA=$(git merge-base origin/main HEAD) .gitea/workflows/scripts/images.sh`. - It returned rc 0 and built and verified all three images: - - toolbox: every tool on PATH, and the five pins matched; - - agent-runner: `PM_SELFTEST=1 pm-dispatch` OK, and the stdin-fed `probe-validation.sh` - gave "20 refusal(s), 0 failure(s)"; - - conformance-runner: all seven `conformance-clone` refusals. -- **The docker-CLI code path**, i.e. what CI runs. I ran the same driver inside - `docker:27-cli` (`ENGINE=docker`, `DOCKER_BUILDKIT=0`) against podman's Docker-compatible API - socket. Exit code 0; all three images built and verified (selftest OK; probe OK, 20 refusals). - This covers the docker-specific calls (`image inspect` formats, `run -i` with stdin, - `--tmpfs`, `--read-only`, `--user`). It does **not** cover BuildKit or a real DinD daemon. -- **This PR's CI, run 325 (`images`): FAILED in 6 s**, because there is no Docker daemon in the - job container (see BLOCKER). Gate run 324 failed only on `approval-policy` (expected, - sensitive paths). Every other stage passed there, including `gitea-settings`. -- **Other driver paths, run locally:** - - agent-runner alone builds the toolbox as its base; - - an empty diff builds nothing; - - publish mode without a token dies before doing anything; - - `infra/agent/build-image.sh` with the zero placeholder dies with build-locally - instructions. -- **Renovate, with the pinned 44.167.0 image, locally:** - - `renovate-config-validator` exits 0; - - a `--platform=local` `extract` dry run finds all four marker references (custom regex manager, - `currentValue: main`, `currentDigest: sha256:000…`); - - a `lookup` dry run shows the `kubernetes`-manager duplicates as `skipReason: disabled` and the - marker deps taking the "unsupported/unversioned value main → getDigest(main)" path. That - lookup currently 404s because nothing is published yet. -- **Renovate 39.264.0 rejects main's current `renovate.json5`.** Validator output reproduced - above. -- **`registry-retention.sh` dry run against the live Gitea (read-only):** zero versions per image, - nothing to delete. The token exchange and the package listing both answered. -- **Live, read-only measurements** (kubectl and the Gitea API): - - the runner pod is on k3s-prod-1, which had **7 GB** nodefs available; - - DinD holds only the 1.54 GB act image; buildx is v0.20.1; the daemon is 27.5.1 on overlay2 - with cgroup v2; - - Gitea `/data` is 9.7G, 1% used; - - the Gateway's Envoy (Cilium) has `streamIdleTimeout: 300s`, `maxStreamDuration: 0s`, and - **no route `timeout`** on the gitea route; - - Gitea's `LIMIT_SIZE_CONTAINER` is unset (default −1 = unlimited, `modules/setting/packages.go`). +**Proved locally, against the exact pinned images:** + +- **Rootless buildkitd, configured as the manifest configures it.** + - Setup: the pinned `moby/buildkit:v0.34.1-rootless` digest, under rootless podman, with + `--cap-drop ALL --cap-add SETUID --cap-add SETGID`, seccomp unconfined and `--read-only`; + tmpfs at the same five paths; the ConfigMap's `buildkitd.toml`; mTLS on :1234. + - The image's `/etc/subuid` was overridden with `user:1:999` and `user:1001:64535`, because + rootless podman maps only 65,537 uids. rootlesskit's real `newuidmap` user-namespace path + still ran. + - It started, `buildctl debug workers` listed the OCI worker, and `debug workers -v` showed the + GC config applied (max 4.29 GB, reserved 1.07 GB). +- **The securityContext, item by item.** Each requirement was measured by taking it away: + - **default seccomp:** daemon starts, every RUN step fails → `Unconfined` is required; + - **no `SETUID`/`SETGID`:** rootlesskit dies with + `newuidmap … operation not permitted` → both capabilities are required; + - **`--read-only` rootfs:** works once the five emptyDirs exist. Its only failure was the + probe client's `~/.docker`, which is now an emptyDir; + - **minimal caps + Unconfined seccomp:** works, and a RUN step executes. +- **The full CI path, in the actual CI job image.** Setup: `ghcr.io/catthehacker/ubuntu:act-22.04` + running `.gitea/workflows/scripts/images.sh` in verify mode. It downloaded and sha256-checked + buildx v0.38.1, connected over mTLS, and bake-built all three `verify` targets from an empty + cache. rc 0, 1m32s total: + - `TOOLBOX VERIFIED` (42 s); + - `AGENT-RUNNER VERIFIED` (26 s): selftest OK, probe "20 refusal(s), 0 failure(s)", and built + FROM the in-run toolbox through the `target:` context, with the log showing + `[agent-runner-verify toolbox 1/6]` and `base=toolbox-in-this-run`; + - `CONFORMANCE-RUNNER VERIFIED` (21 s): all seven refusals; + - every stage printed `confirmed: no network`. +- **mTLS rejects what it should.** Tested with a real BuildKit API call (`buildctl debug workers`): + - no client cert → `remote error: tls: certificate required`; + - a client cert from another CA → rejected the same way; + - the valid cert → accepted. + + Through the driver: + - another CA's client cert → rc 1; + - the wrong CA for the server → rc 1; + - an unreachable host → rc 1 in about 20 s, with the explanatory message; + - missing secrets → rc 1, naming the three secrets. +- **The local podman path**, `build-image.sh` for all three (rc 0). + - Negative control: a verify stage expecting a different promtool version printed + `DRIFT promtool want 9.9.9` and failed the build. + - Negative control: the `--network=none` check fails when the flag is removed. +- **`buildkit-client-secret.sh`'s sanity checks**, run offline on local certs. They tell apart: + - a good client cert; + - a server cert (not client-auth); + - a mismatched key; + - a cert from another CA. + +**Measured live, read-only:** +- node user namespaces, Debian 13.7 and AppArmor (via an Ansible `sysctl`/`cat`); +- DinD's MASQUERADE rules; +- the live ClusterPolicies, namespaced Policies, PSA labels, and cert-manager v1.21.3 with its + ClusterIssuers; +- the act-runner pod labels. + +**Renovate (unchanged since the first iteration):** +- the validator with 44.167.0: exit 0; +- `extract` finds all four markers; +- the `kubernetes` duplicates are `skipReason: disabled`; +- 39.264.0 rejects `main`'s config. + +**Local gate, CI results:** see the PR body. They are recorded there because this entry is part of +the tree being gated. ## What I did NOT verify -- **Nothing has been pushed to the registry, by me or by CI.** The publish path has never run: - - the registry HEAD/digest read-back; +- **platform/buildkit has never run in the cluster.** It can't until it merges: agents may not + `kubectl apply`, and Flux reconciles only `main`. So this PR's own `images` job **cannot build + and is expected to fail at the builder preflight**, which is the designed loud failure. The + following are therefore all unproven in-cluster: + - kubelet admitting the pod; + - containerd honouring `appArmorProfile: Unconfined`; + - the five emptyDirs; + - cert-manager issuing the three certs; + - the CiliumNetworkPolicies. +- **AppArmor `Unconfined` is reasoned, not measured.** My test host runs SELinux, so the AppArmor + requirement comes from upstream's example plus the nodes having AppArmor enabled. If the pod + were to work under `RuntimeDefault` AppArmor, that line could be tightened; I did not test it. +- **The local userns mapping is narrower than the cluster's.** Locally it was 65,535 ids across + two ranges; in the cluster it is the image's own `100000:65536` against real host subuids. That + is the normal case, but it is the cluster's first test. +- **The NetworkPolicies are untested.** In particular: + - that `toEntities: [ingress]` admits the push to the Gateway VIP. The comment explains the + reasoning, but it is unmeasured; + - that world :443/:80 covers every download the builds make. A dnf mirror on another port + would fail a build visibly. +- **No push, anywhere.** The whole publish path is unexercised: + - `REGISTRY_PUSH_TOKEN` reaching buildkitd through the buildx session; + - `push=true`; + - the metadata digest compared with the registry read-back; + - agent-runner FROM `toolbox@sha256:`; - the immutable-tag guard; - - "already published at this commit" idempotence; - - agent-runner `FROM @sha256:`; - - `docker login` with a PAT. + - idempotence. - The first merge to main is the first test, and it needs `REGISTRY_PUSH_TOKEN` to exist first. -- **Large-layer push through the Gateway is UNVERIFIED.** Envoy has no request-body cap and no - stream duration cap here. But with no route `timeout` set, Envoy's **default 15 s upstream - timeout** applies from the end of the request to the end of the response. That covers Gitea - finalising a blob: copying the upload into package storage on Longhorn. A ~250 MB compressed - layer should finish well inside 15 s, but I could not measure it without pushing. If the first - publish shows 504s on blob PUTs, set `timeouts.request` on the gitea HTTPRoute. -- **DinD fit is estimated, not measured** (no build has run there, see the BLOCKER): - - memory: 2Gi is shared by the job container and every build container; - - disk: the toolbox, agent-runner, conformance-runner, Fedora base and build cache should - total roughly 3 GB on top of the 1.5 GB act image, against 7 GB free on the node, with - `dind-storage` having no sizeLimit. -- **No image has been built in this runner's DinD**, because it cannot be done until the BLOCKER - is resolved. Two things are therefore unmeasured until then: - - DinD memory (2Gi) and disk fit; - - BuildKit (`buildx` v0.20.1 is on the daemon) as opposed to the legacy builder I tested - with. -- **Renovate has still never completed a run here.** Three things block it: - - #80, the `configurationFile` name; - - `RENOVATE_TOKEN` gaining `read:package`; - - this branch's version pin. - - `renovatebot/github-action@v41.0.9` running a 44.x image is untested. The action only passes - the tag to `docker run`. -- **Pods cannot pull these images yet regardless.** The kubelet pull credential is PR #61 - (`pm-agent-registry` dockerconfigjson, `read:package`), still open. Until it merges, a correct - digest still ends in ImagePullBackOff (401), as before. -- **`registry-retention.sh --apply` has never run**, and its keep logic has only run against an - empty registry. -- **`git diff` change detection on Gitea `push` events relies on `github.event.before`.** A - force-push or a first push falls back to building everything, which is intended but not - observed. + So are the 15 s Envoy route-timeout risk on blob finalisation, and the Gitea PVC (10Gi, shared + with repositories) absorbing ~2 GB per toolbox publish until retention runs. +- **Node disk under real builds.** The cache emptyDir (6Gi limit, 4 GB GC cap) sits on nodes that + had 7–9 GB free. It is untested under concurrent pressure. +- **Certificate rotation is designed, not exercised.** The server cert renews 30 days early but + buildkitd only reads it at start; the client copy needs the script re-run. +- **`--oci-worker-no-process-sandbox` is an accepted risk, not a mitigated one.** A build step can + ptrace the daemon and read its TLS key and cache. Anyone with the client cert, which means every + PR job, can run build steps. +- **Renovate has still never completed a run.** It is blocked on #80, on `RENOVATE_TOKEN` gaining + `read:package`, and on this branch. The `github-action@v41.0.9` + 44.x image pairing is + untested. +- **Pods cannot pull these images until PR #61** (the kubelet pull secret) merges. +- **`registry-retention.sh --apply` has never run**; its keep logic has only run against an empty + registry. diff --git a/docs/plans/pm-agent-issue-to-pr.md b/docs/plans/pm-agent-issue-to-pr.md index 9f18152..7d994fc 100644 --- a/docs/plans/pm-agent-issue-to-pr.md +++ b/docs/plans/pm-agent-issue-to-pr.md @@ -245,8 +245,8 @@ about runtime behaviour in this document is a design claim, not a measurement. the credential decision now being local Gitea accounts with PATs, that means a `dockerconfigjson` for an account whose token carries `read:package`. Only then does the digest matter. **Superseded 2026-10-29:** the build-and-push is no longer an operator step. - `.gitea/workflows/images.yaml` builds and verifies the image with - `infra/agent/build-image.sh`, publishes it on a push to `main`, and Renovate opens the one + `.gitea/workflows/images.yaml` builds and verifies the image on rootless BuildKit + (`platform/buildkit/`), publishes it on a push to `main`, and Renovate opens the one PR that replaces both placeholder digests (`decisions/2026-10-29-image-pipeline.md`). The same two blockers sit in front of `conformance-runner`, which shares the registry path and has never been pulled. diff --git a/infra/agent/Containerfile b/infra/agent/Containerfile index 14da4d8..4b85b48 100644 --- a/infra/agent/Containerfile +++ b/infra/agent/Containerfile @@ -50,7 +50,10 @@ # default is deliberately EMPTY so a build that forgets it fails loudly instead of # resolving some other image named `agent-base`. ARG AGENT_BASE_IMAGE= -FROM ${AGENT_BASE_IMAGE} +# Stage `agent-runner` is the image that ships; stage `verify` (end of file) is a TEST stage built +# FROM it that only runs assertions and is never pushed. build-image.sh and the CI pipeline +# build `--target verify` to prove the image and `--target agent-runner` for the image itself. +FROM ${AGENT_BASE_IMAGE} AS agent-runner # Re-declared AFTER the FROM: an ARG from before the first FROM is not in scope in the # build stage, so the provenance layer below would otherwise record an empty base. @@ -208,3 +211,80 @@ WORKDIR /workspace # paths above; an ENTRYPOINT here would either be ignored or prepend something the policy # cannot see. `bash` is the interactive fallback for a human debugging the image locally. CMD ["/bin/bash"] + +# ============================================================================= +# verify — a TEST stage, never shipped. Every assertion infra/agent/build-image.sh used to make +# with `run --user 65532 --read-only --tmpfs /workspace --tmpfs /tmp --network none` now runs +# here, at build time, because CI builds on rootless BuildKit where no container runtime exists. +# +# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE, stated so nobody over-reads a green verify: +# * uid 65532 ............ exact: USER is inherited from the agent-runner stage. +# * --network none ....... exact: RUN --network=none, and the first check proves it. +# * --read-only rootfs ... NOT reproducible: a RUN step always has a writable layer. It is +# APPROXIMATED by permission: as uid 65532, every path the process can +# write is inventoried and must lie under /workspace, /tmp or /var/tmp +# (the pod mounts emptyDirs on the first two; /var/tmp is read-only +# there, and TMPDIR points into /workspace so nothing should use it). +# The pod's readOnlyRootFilesystem is enforced by admission +# (platform/kyverno/admission/pm-agent-job-shape.yaml), not by this. +# * --tmpfs /workspace ... approximated by the image's own /workspace (owned 65532), which is +# what the pod's emptyDir replaces. +# The probe script is COPYed into THIS stage only, so it never ships in the image. +# ============================================================================= +FROM agent-runner AS verify +COPY probe-validation.sh /opt/verify/probe-validation.sh +RUN --network=none /bin/bash -euo pipefail <<'VERIFY' +echo "--- no network (the build-time stand-in for run --network none) ---" +if getent hosts registry.npmjs.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then + echo "UNEXPECTED: network reachable in the verify stage"; exit 1 +fi +echo " confirmed: no network" +echo "--- id ---"; id +echo "--- HOME ---"; echo "HOME=${HOME} CLAUDE_CONFIG_DIR=${CLAUDE_CONFIG_DIR}" +echo "--- baked versions ---"; cat /etc/agent-runner-versions +echo "--- the gate toolchain the worker needs (CLAUDE.md §3) ---" +for t in git python3 bash yamllint check-jsonschema kyverno promtool chainsaw flux sops age kubectl jq diff; do + if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t"; + else printf " MISSING %s\n" "$t"; missing=1; fi +done +[ -z "${missing:-}" ] || { echo "the gate cannot run in this image"; exit 1; } +echo "--- python can import the renderer dependency ---" +python3 -c "import yaml; print('PyYAML', yaml.__version__)" +echo "--- claude ---"; claude --version +echo "--- the two entrypoints refuse arguments ---" +# Each must REFUSE. (The old docker-run version wrote `cmd | grep -q ... && echo`, which under +# `set -e` + pipefail could never fail the script; these are real assertions.) +for e in pm-worker pm-dispatch; do + out="$(/usr/local/bin/$e --anything 2>&1 || true)" + grep -q "takes no arguments" <<<"$out" || { echo "UNEXPECTED: $e accepted an argument: $out"; exit 1; } + echo " $e: rejected" +done +echo "--- the dispatcher selftest (branch vocabulary, result envelopes, Job template, persona) ---" +PM_SELFTEST=1 /usr/local/bin/pm-dispatch +echo "--- the entrypoints are root-owned and NOT writable by the runtime uid ---" +for f in /usr/local/bin/pm-worker /usr/local/bin/pm-dispatch /etc/pm-agent/worker-job-template.yaml; do + stat -c " %n %U:%G %a" "$f" + if [ -w "$f" ]; then echo "UNEXPECTED: uid $(id -u) can write $f"; exit 1; fi +done +echo " confirmed: not writable by uid $(id -u)" +echo "--- the root filesystem is not writable by the runtime uid (expected: a failure) ---" +if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi +echo " confirmed: / is not writable" +echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---" +stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \ + | grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)" +if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf ' %s\n' ${stray} | head -20; exit 1; fi +echo " confirmed: no writable path outside /workspace, /tmp, /var/tmp" +echo "--- the two writable paths really are writable ---" +touch "${HOME}/.probe" /tmp/.probe || { echo "UNEXPECTED: ${HOME} or /tmp not writable"; exit 1; } +echo " confirmed: ${HOME} and /tmp are writable" +echo "--- uid is non-zero ---" +[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; } +echo " confirmed: uid $(id -u), not root" +echo "--- no Anthropic API key is baked into the image (it would outrank the subscription token) ---" +[ -z "${ANTHROPIC_API_KEY:-}${ANTHROPIC_AUTH_TOKEN:-}" ] || { echo "UNEXPECTED: an Anthropic key is baked in"; exit 1; } +echo " confirmed: neither is set" +echo "--- pm-worker's input validation, with the interpreter and script that run in the cluster ---" +PM_WORKER_PATH=/usr/local/bin/pm-worker /bin/bash /opt/verify/probe-validation.sh +echo "AGENT-RUNNER VERIFIED" +VERIFY diff --git a/infra/agent/build-image.sh b/infra/agent/build-image.sh index e854f4e..cba017b 100755 --- a/infra/agent/build-image.sh +++ b/infra/agent/build-image.sh @@ -2,7 +2,8 @@ ############################################################################### # infra/agent/build-image.sh — build the pinned `agent-runner` image and prove it works. # -# ./build-image.sh # build + verify + print the image id +# ./build-image.sh # build the verify target, then the image +# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes) # ./build-image.sh --no-verify # build only (skip the in-image proof) # ENGINE=docker ./build-image.sh # build with docker instead of podman # AGENT_BASE_IMAGE=home-k3s-dev:local ./build-image.sh @@ -11,12 +12,12 @@ # IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this) # # THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either. -# Publishing is done by CI: .gitea/workflows/images.yaml calls this script, and on a push to -# `main` it pushes what this script built AND verified to the Gitea registry, records the -# registry-returned digest in the job summary, and Renovate then opens the pull request that -# moves platform/pm-agent/ onto that digest. So this script's job is to build, and to prove -# the toolchain and both entrypoints work INSIDE the image as the non-root uid with the -# read-only root filesystem the admission policy forces — the same proof locally and in CI. +# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on +# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and +# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest +# in the job summary. Renovate then opens the pull request that moves platform/pm-agent/ onto +# that digest. This script is the LOCAL path (podman by default); its proof — the +# Containerfile's `verify` stage — is the same one CI runs. # # VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the pins this image adds # (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds the @@ -26,11 +27,10 @@ # kept current by Renovate. An AGENT_BASE_IMAGE in the environment overrides it — CI passes # the toolbox it built in the same run, and a developer can pass `home-k3s-dev:local`. # -# ENGINE PORTABILITY. CI runs this with ENGINE=docker against a Docker-in-Docker daemon, so -# nothing below may be podman-only (`image exists`, `--digestfile`, `{{.Digest}}`), and -# nothing may BIND-MOUNT a file from this checkout into a container: under DinD the daemon -# resolves bind sources on ITS filesystem, where this checkout does not exist, and silently -# mounts an empty directory instead. Files go in over stdin. +# NO `run` STEP. Verification used to `run` the image as uid 65532 with a read-only rootfs. +# CI has no container runtime (rootless BuildKit only), so the assertions moved into the +# Containerfile's `verify` stage and this script builds that target. `--print-build-args` +# prints the pins as KEY=VALUE for the CI pipeline; AGENT_BASE_IMAGE is the caller's choice. ############################################################################### set -euo pipefail @@ -39,9 +39,11 @@ REPO_ROOT="$(cd "${HERE}/../.." && pwd)" ENGINE="${ENGINE:-podman}" VERIFY=1 +PRINT_ARGS=0 for arg in "$@"; do case "$arg" in --no-verify) VERIFY=0 ;; + --print-build-args) PRINT_ARGS=1 ;; -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;; *) echo "unknown argument: $arg" >&2; exit 2 ;; esac @@ -51,8 +53,6 @@ log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; } info() { printf ' %s\n' "$*"; } die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; } -command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)" - # --- pins: sourced, never invented ------------------------------------------- AGENT_ENV="${HERE}/versions.env" [ -f "${AGENT_ENV}" ] || die "missing ${AGENT_ENV} — the pins live there, not here" @@ -68,6 +68,17 @@ source "${AGENT_ENV}" : "${AGENT_MODEL:?AGENT_MODEL not set in ${AGENT_ENV}}" : "${AGENT_IMAGE_REPO:?AGENT_IMAGE_REPO not set in ${AGENT_ENV}}" +# THE build args, in ONE place; `--print-build-args` hands them to CI (which builds on rootless +# BuildKit and supplies its own base reference). AGENT_BASE_IMAGE is deliberately NOT in this +# list: the caller decides the base. +BUILD_ARGS=( + "NODEJS_MAJOR=${NODEJS_MAJOR}" "CLAUDE_CODE_VERSION=${CLAUDE_CODE_VERSION}" + "CLAUDE_CODE_MIN_VERSION=${CLAUDE_CODE_MIN_VERSION}" "AGENT_MODEL=${AGENT_MODEL}" +) +if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi + +command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)" + # --- the base must exist, locally or in the registry --------------------------- # Deliberately NOT built on demand. The toolbox is its own reviewed artefact with its own # pins, and silently building it here would mean this script could change the toolchain a @@ -103,89 +114,26 @@ info " model ${AGENT_MODEL}" # Provenance: CI passes the commit it is building, so a pulled image names its source tree. LABEL_ARGS=() [ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}") - -"${ENGINE}" build "${LABEL_ARGS[@]}" \ - --file "${HERE}/Containerfile" \ - --tag "${IMAGE_REF}" \ - --build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}" \ - --build-arg "NODEJS_MAJOR=${NODEJS_MAJOR}" \ - --build-arg "CLAUDE_CODE_VERSION=${CLAUDE_CODE_VERSION}" \ - --build-arg "CLAUDE_CODE_MIN_VERSION=${CLAUDE_CODE_MIN_VERSION}" \ - --build-arg "AGENT_MODEL=${AGENT_MODEL}" \ - "${HERE}" +ARG_FLAGS=(--build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}") +for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done # --- verify: the point of the whole exercise --------------------------------- -# A build that merely succeeded is not evidence this image is usable. Prove the tools -# EXECUTE, as the NON-ROOT uid, with a READ-ONLY root filesystem and only the two volumes -# the admission policy admits — the exact shape a worker Job runs in. Anything that works -# here but not there is a 3am discovery. +# A build that merely succeeded is not evidence this image is usable. The proof is the +# Containerfile's `verify` stage: as the NON-ROOT uid, with no network, the toolchain executes, +# both entrypoints refuse arguments, the dispatcher selftest passes, the entrypoints are not +# writable, nothing outside /workspace and /tmp is writable, and pm-worker's input guards +# (probe-validation.sh) refuse every hostile input. It is a build target rather than a +# `run` so the SAME proof runs here and in CI, where no container runtime exists — see the +# stage's header for what a build step can and cannot reproduce of the pod's shape. if [ "${VERIFY}" -eq 1 ]; then - log "Verifying inside the image (uid 65532, read-only rootfs, workspace+tmp only)" - "${ENGINE}" run --rm \ - --user 65532:65532 \ - --read-only \ - --tmpfs /workspace:rw,mode=1777 \ - --tmpfs /tmp:rw,mode=1777 \ - --network none \ - --entrypoint "" \ - "${IMAGE_REF}" \ - /bin/bash -euo pipefail -c ' - echo "--- id ---"; id - echo "--- HOME ---"; echo "HOME=${HOME} CLAUDE_CONFIG_DIR=${CLAUDE_CONFIG_DIR}" - echo "--- baked versions ---"; cat /etc/agent-runner-versions - echo "--- the gate toolchain the worker needs (CLAUDE.md §3) ---" - for t in git python3 bash yamllint check-jsonschema kyverno promtool chainsaw flux sops age kubectl jq diff; do - if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t"; - else printf " MISSING %s\n" "$t"; missing=1; fi - done - [ -z "${missing:-}" ] || { echo "the gate cannot run in this image"; exit 1; } - echo "--- python can import the renderer dependency ---" - python3 -c "import yaml; print(\"PyYAML\", yaml.__version__)" - echo "--- claude ---"; claude --version - echo "--- the two entrypoints refuse arguments ---" - /usr/local/bin/pm-worker --anything 2>&1 | grep -q "takes no arguments" && echo " pm-worker: rejected" - /usr/local/bin/pm-dispatch --anything 2>&1 | grep -q "takes no arguments" && echo " pm-dispatch: rejected" - echo "--- the dispatcher selftest (branch vocabulary, result envelopes, Job template, persona) ---" - PM_SELFTEST=1 /usr/local/bin/pm-dispatch - echo "--- the entrypoints are root-owned and NOT writable by the runtime uid ---" - for f in /usr/local/bin/pm-worker /usr/local/bin/pm-dispatch /etc/pm-agent/worker-job-template.yaml; do - stat -c " %n %U:%G %a" "$f" - if [ -w "$f" ]; then echo "UNEXPECTED: uid $(id -u) can write $f"; exit 1; fi - done - echo " confirmed: not writable by uid $(id -u)" - echo "--- rootfs is read-only (expected: a failure) ---" - if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi - echo " confirmed: / is read-only" - echo "--- the two writable paths really are writable ---" - touch "${HOME}/.probe" /tmp/.probe && echo " confirmed: ${HOME} and /tmp are writable" - echo "--- uid is non-zero ---" - [ "$(id -u)" -ne 0 ] && echo " confirmed: uid $(id -u), not root" - echo "--- no Anthropic API key is baked into the image (it would outrank the subscription token) ---" - [ -z "${ANTHROPIC_API_KEY:-}${ANTHROPIC_AUTH_TOKEN:-}" ] && echo " confirmed: neither is set" - ' - # The worker's input guards, exercised in the image, with the interpreter and the script - # that will actually run in the cluster. The probe is mounted read-only rather than baked - # in: it is a build-time check, not something a worker Job has any business carrying. - log "Probing pm-worker's input validation inside the image" - # - # The probe goes in over STDIN, not as a bind mount: under Docker-in-Docker (CI) a bind - # source is resolved on the daemon's filesystem, where this file does not exist, and the - # container would get an empty directory. PM_WORKER_PATH points it at the baked worker, - # since the probe's own default (a sibling of the probe file) has no meaning on stdin. - "${ENGINE}" run --rm -i \ - --user 65532:65532 \ - --read-only \ - --tmpfs /workspace:rw,mode=1777 \ - --tmpfs /tmp:rw,mode=1777 \ - --network none \ - -e PM_WORKER_PATH=/usr/local/bin/pm-worker \ - --entrypoint "" \ - "${IMAGE_REF}" \ - /bin/bash -s < "${HERE}/probe-validation.sh" - + log "Verifying (Containerfile target: verify)" + "${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}" log "Image verified" fi +"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target agent-runner \ + --file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}" + # --- result ------------------------------------------------------------------ # No digest is printed: a manifest digest does not exist until a registry has the manifest, # and the only digest anything may pin is the one the REGISTRY returns. CI captures that at diff --git a/infra/agent/probe-validation.sh b/infra/agent/probe-validation.sh index d888700..57f44ec 100755 --- a/infra/agent/probe-validation.sh +++ b/infra/agent/probe-validation.sh @@ -16,14 +16,14 @@ # scrubbed environment and fake credentials, and if a guard ever stopped firing, the case # would fail here rather than by really cloning something. # -# Run offline, no cluster, no Gitea, no model. infra/agent/build-image.sh runs it inside the -# built image as well, where the interpreter and the script are the ones that will actually -# run in the cluster. +# Run offline, no cluster, no Gitea, no model. The Containerfile's `verify` stage runs it inside +# the built image as well (as uid 65532, no network), where the interpreter and the script are +# the ones that will actually run in the cluster — locally via build-image.sh and in CI. ############################################################################### set -uo pipefail -# `:-$0` because build-image.sh feeds this script over STDIN (`bash -s`), where BASH_SOURCE is -# empty and `set -u` would abort on it; PM_WORKER_PATH is always set in that case. +# `:-$0` keeps this safe if it is ever fed over stdin (`bash -s`), where BASH_SOURCE is empty +# and `set -u` would abort on it. HERE="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)" WORKER="${PM_WORKER_PATH:-${HERE}/pm-worker}" [ -r "$WORKER" ] || { echo "cannot read ${WORKER}" >&2; exit 2; } diff --git a/infra/devcontainer/Containerfile b/infra/devcontainer/Containerfile index a46720a..469d199 100644 --- a/infra/devcontainer/Containerfile +++ b/infra/devcontainer/Containerfile @@ -8,7 +8,10 @@ # packages are retained as harmless disk-image utilities. Fedora base is pinned for # reproducibility. ARG FEDORA_VERSION=44 -FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} +# Stage `toolbox` is the image that ships. Stage `verify` (end of file) is a TEST stage: it is +# built FROM the finished image and only runs assertions, and it is never pushed. Build with +# `--target toolbox` for the image and `--target verify` to prove it (build-image.sh does both). +FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} AS toolbox ARG OPENTOFU_VERSION ARG AGE_VERSION @@ -95,3 +98,48 @@ RUN { echo "opentofu=${OPENTOFU_VERSION}"; echo "age=${AGE_VERSION}"; echo "sops WORKDIR /work CMD ["/bin/bash"] + +# ============================================================================= +# verify — a TEST stage, never shipped. It runs every assertion against the finished image AT +# BUILD TIME, so CI needs no container runtime to prove the image: the image pipeline builds +# this target on rootless BuildKit (platform/buildkit/), where `docker run` does not exist. +# `RUN --network=none` reproduces the old `run --network none` exactly, and the first check +# below proves it (it fails if any network is reachable). +# ============================================================================= +FROM toolbox AS verify +ARG PROMTOOL_VERSION +ARG KUBECTL_VERSION +ARG CHAINSAW_VERSION +ARG KYVERNO_VERSION +ARG SOPS_VERSION +RUN --network=none /bin/bash -euo pipefail <<'VERIFY' +echo "--- no network (the build-time stand-in for run --network none) ---" +if getent hosts registry.fedoraproject.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then + echo "UNEXPECTED: network reachable in the verify stage"; exit 1 +fi +echo " confirmed: no network" +echo "--- baked versions ---"; cat /etc/dev-toolchain-versions +echo "--- every tool is on PATH ---" +for t in tofu age age-keygen sops flux kubectl helm kyverno kyverno-json chainsaw yq \ + yamllint check-jsonschema promtool ansible-playbook git make jq diff python3 \ + wg qrencode; do + if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t"; + else printf " MISSING %s\n" "$t"; missing=1; fi +done +[ -z "${missing:-}" ] || { echo "toolbox is missing tools"; exit 1; } +echo "--- version-sensitive tools report their pins ---" +: "${PROMTOOL_VERSION:?}" "${KUBECTL_VERSION:?}" "${CHAINSAW_VERSION:?}" "${KYVERNO_VERSION:?}" "${SOPS_VERSION:?}" +check() { # name pin actual-output + if printf "%s" "$3" | grep -qF "$2"; then printf " ok %-10s %s\n" "$1" "$2"; + else printf " DRIFT %-10s want %s, got: %s\n" "$1" "$2" "$(printf "%s" "$3" | head -1)"; drift=1; fi +} +check promtool "${PROMTOOL_VERSION}" "$(promtool --version 2>&1)" +check kubectl "${KUBECTL_VERSION}" "$(kubectl version --client 2>&1)" +check chainsaw "${CHAINSAW_VERSION}" "$(chainsaw version 2>&1)" +check kyverno "${KYVERNO_VERSION}" "$(kyverno version 2>&1)" +check sops "${SOPS_VERSION}" "$(sops --version --disable-version-check 2>&1 || sops --version 2>&1)" +[ -z "${drift:-}" ] || { echo "a tool does not report its pinned version"; exit 1; } +echo "--- python can import what the renderer and the gate need ---" +python3 -c "import yaml; print(' PyYAML', yaml.__version__)" +echo "TOOLBOX VERIFIED" +VERIFY diff --git a/infra/devcontainer/README.md b/infra/devcontainer/README.md index 3cfbaa3..6ff9d40 100644 --- a/infra/devcontainer/README.md +++ b/infra/devcontainer/README.md @@ -36,9 +36,11 @@ cd infra/devcontainer ### The published toolbox -`./dev.sh build` delegates to [`build-image.sh`](build-image.sh), which is the **one** build -recipe: CI runs the same script (with its in-image verification — every tool executes, and the -version-sensitive ones report their `versions.env` pins) and, on a push to `main` that changes +`./dev.sh build` delegates to [`build-image.sh`](build-image.sh). The recipe is the +`Containerfile` plus that script's build-arg list, and the proof is the Containerfile's `verify` +stage (every tool on `PATH`, the version-sensitive ones report their `versions.env` pins, no +network). CI builds the same Containerfile on rootless BuildKit with the args from +`build-image.sh --print-build-args`, and, on a push to `main` that changes `Containerfile`, `versions.env` or `build-image.sh`, **publishes** the result as `gitea.home.keithjackson.dev/home-network/toolbox` (tags `sha-` and `main`) — see [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). `agent-runner` is diff --git a/infra/devcontainer/build-image.sh b/infra/devcontainer/build-image.sh index 9467a8e..07c9cc4 100755 --- a/infra/devcontainer/build-image.sh +++ b/infra/devcontainer/build-image.sh @@ -6,12 +6,14 @@ # ./build-image.sh --no-verify # build only # IMAGE=foo:bar ./build-image.sh # build under another local name # ENGINE=docker ./build-image.sh # build with docker instead of podman +# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes) # -# ONE BUILD DEFINITION, TWO CALLERS. `dev.sh build` calls this for local development, and the -# CI image pipeline (.gitea/workflows/images.yaml -> .gitea/workflows/scripts/images.sh) calls -# it to build the image it then PUBLISHES as gitea.home.keithjackson.dev/home-network/toolbox. -# Keeping the build args in exactly one place is what makes "the toolbox CI publishes" and "the -# toolbox you gate with locally" the same recipe. +# ONE BUILD DEFINITION, TWO CALLERS. `dev.sh build` calls this for local development. The CI +# image pipeline (.gitea/workflows/images.yaml -> .gitea/workflows/scripts/images.sh) builds the +# SAME Containerfile on rootless BuildKit (platform/buildkit/) — no container engine exists in a +# CI job — taking its build args from `--print-build-args`, and PUBLISHES the result as +# gitea.home.keithjackson.dev/home-network/toolbox. The Containerfile (including its `verify` +# test stage) and the arg list here are the one recipe both use. # # THIS SCRIPT NEVER PUSHES. Publishing is CI's job (images.yaml, on push to main), not a # workstation's: an image that exists only because someone ran `podman push` from a laptop has @@ -26,9 +28,11 @@ ENGINE="${ENGINE:-podman}" IMAGE="${IMAGE:-home-k3s-dev:local}" VERIFY=1 +PRINT_ARGS=0 for arg in "$@"; do case "$arg" in --no-verify) VERIFY=0 ;; + --print-build-args) PRINT_ARGS=1 ;; -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;; *) echo "unknown argument: $arg" >&2; exit 2 ;; esac @@ -37,72 +41,44 @@ done log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; } die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; } -command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)" - # shellcheck disable=SC1091 source "${HERE}/versions.env" +# THE build args, in ONE place. `--print-build-args` hands exactly this list to the CI pipeline +# (.gitea/workflows/scripts/images.sh), which builds on rootless BuildKit rather than calling +# "$ENGINE build" — so CI and a local build cannot drift apart on a pin. +BUILD_ARGS=( + "FEDORA_VERSION=$FEDORA_VERSION" "OPENTOFU_VERSION=$OPENTOFU_VERSION" "AGE_VERSION=$AGE_VERSION" + "SOPS_VERSION=$SOPS_VERSION" "FLUX_VERSION=$FLUX_VERSION" "KUBECTL_VERSION=$KUBECTL_VERSION" + "HELM_VERSION=$HELM_VERSION" "KYVERNO_VERSION=$KYVERNO_VERSION" + "KYVERNO_JSON_VERSION=$KYVERNO_JSON_VERSION" "CHAINSAW_VERSION=$CHAINSAW_VERSION" + "YAMLLINT_VERSION=$YAMLLINT_VERSION" "YQ_VERSION=$YQ_VERSION" + "CHECK_JSONSCHEMA_VERSION=$CHECK_JSONSCHEMA_VERSION" "PROMTOOL_VERSION=$PROMTOOL_VERSION" +) +if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi + +command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)" + +ARG_FLAGS=() +for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done # Provenance: CI passes the commit it is building, so a pulled image names its source tree. LABEL_ARGS=() [ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}") -log "Building ${IMAGE}" -"$ENGINE" build "${LABEL_ARGS[@]}" \ - --build-arg FEDORA_VERSION="$FEDORA_VERSION" \ - --build-arg OPENTOFU_VERSION="$OPENTOFU_VERSION" \ - --build-arg AGE_VERSION="$AGE_VERSION" \ - --build-arg SOPS_VERSION="$SOPS_VERSION" \ - --build-arg FLUX_VERSION="$FLUX_VERSION" \ - --build-arg KUBECTL_VERSION="$KUBECTL_VERSION" \ - --build-arg HELM_VERSION="$HELM_VERSION" \ - --build-arg KYVERNO_VERSION="$KYVERNO_VERSION" \ - --build-arg KYVERNO_JSON_VERSION="$KYVERNO_JSON_VERSION" \ - --build-arg CHAINSAW_VERSION="$CHAINSAW_VERSION" \ - --build-arg YAMLLINT_VERSION="$YAMLLINT_VERSION" \ - --build-arg YQ_VERSION="$YQ_VERSION" \ - --build-arg CHECK_JSONSCHEMA_VERSION="$CHECK_JSONSCHEMA_VERSION" \ - --build-arg PROMTOOL_VERSION="$PROMTOOL_VERSION" \ - -t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}" - -# --- verify ------------------------------------------------------------------ -# A build that succeeded is not evidence the toolbox can run the gate. Every tool the gate and -# the runbooks call must EXECUTE, and the version-sensitive ones must report the version -# versions.env pins — promtool above all, because ci/versions.env asserts a promtool/Prometheus -# lockstep and a drifted engine evaluates alert-rule tests differently (see versions.env). +# --- verify -------------------------------------------------------------------- +# The assertions live in the Containerfile's `verify` stage (every tool on PATH, the +# version-sensitive ones report their versions.env pins — promtool above all, because +# ci/versions.env asserts a promtool/Prometheus lockstep — and no network), so the SAME proof +# runs here under podman and in CI under rootless BuildKit. A failing assertion fails the build. if [ "${VERIFY}" -eq 1 ]; then - log "Verifying the toolchain inside ${IMAGE}" - "$ENGINE" run --rm --network none --entrypoint "" \ - -e PROMTOOL_VERSION="${PROMTOOL_VERSION}" \ - -e KUBECTL_VERSION="${KUBECTL_VERSION}" \ - -e CHAINSAW_VERSION="${CHAINSAW_VERSION}" \ - -e KYVERNO_VERSION="${KYVERNO_VERSION}" \ - -e SOPS_VERSION="${SOPS_VERSION}" \ - "$IMAGE" /bin/bash -euo pipefail -c ' - echo "--- baked versions ---"; cat /etc/dev-toolchain-versions - echo "--- every tool is on PATH ---" - for t in tofu age age-keygen sops flux kubectl helm kyverno kyverno-json chainsaw yq \ - yamllint check-jsonschema promtool ansible-playbook git make jq diff python3 \ - wg qrencode; do - if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t"; - else printf " MISSING %s\n" "$t"; missing=1; fi - done - [ -z "${missing:-}" ] || { echo "toolbox is missing tools"; exit 1; } - echo "--- version-sensitive tools report their pins ---" - check() { # name pin actual-output - if printf "%s" "$3" | grep -qF "$2"; then printf " ok %-10s %s\n" "$1" "$2"; - else printf " DRIFT %-10s want %s, got: %s\n" "$1" "$2" "$(printf "%s" "$3" | head -1)"; drift=1; fi - } - check promtool "${PROMTOOL_VERSION}" "$(promtool --version 2>&1)" - check kubectl "${KUBECTL_VERSION}" "$(kubectl version --client 2>&1)" - check chainsaw "${CHAINSAW_VERSION}" "$(chainsaw version 2>&1)" - check kyverno "${KYVERNO_VERSION}" "$(kyverno version 2>&1)" - check sops "${SOPS_VERSION}" "$(sops --version --disable-version-check 2>&1 || sops --version 2>&1)" - [ -z "${drift:-}" ] || { echo "a tool does not report its pinned version"; exit 1; } - echo "--- python can import what the renderer and the gate need ---" - python3 -c "import yaml; print(\" PyYAML\", yaml.__version__)" - ' + log "Verifying (Containerfile target: verify)" + "$ENGINE" build "${ARG_FLAGS[@]}" --target verify -f "${HERE}/Containerfile" "${HERE}" log "Toolbox verified" fi +log "Building ${IMAGE}" +"$ENGINE" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target toolbox \ + -t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}" + log "Built ${IMAGE}" printf ' image id %s\n' "$("$ENGINE" image inspect --format '{{.Id}}' "$IMAGE")" diff --git a/infra/gitea/buildkit-client-secret.sh b/infra/gitea/buildkit-client-secret.sh new file mode 100755 index 0000000..2b9f8fb --- /dev/null +++ b/infra/gitea/buildkit-client-secret.sh @@ -0,0 +1,104 @@ +#!/usr/bin/env bash +# Copy the image builder's CLIENT certificate from the cluster into the three repository Actions +# secrets the image pipeline reads: BUILDKIT_CA, BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY. +# +# ./infra/gitea/buildkit-client-secret.sh # copy and verify +# ./infra/gitea/buildkit-client-secret.sh --check # compare only; changes nothing +# +# RUNNING IT IS AN OPERATOR ACTION. It writes repository secrets. +# +# WHERE THE CERTIFICATE COMES FROM. platform/buildkit/certificates.yaml has cert-manager issue +# `buildkit-client-ci` from a CA that exists only for the builder; buildkitd admits a connection +# only with a certificate signed by that CA. CI jobs hold NO cluster credential (R-SEC-02), so they +# cannot read the Secret themselves: this script is the one bridge, run by a human who has both +# a kube context and the Gitea credential. +# +# WHEN TO RUN IT +# 1. once, after Flux has applied platform/buildkit (Kustomization `buildkit` Ready, and +# `kubectl -n buildkit get certificate buildkit-client-ci` READY=True); +# 2. after every renewal of that certificate (duration 1y, renewed 30 days early). Until then +# the image pipeline fails LOUDLY at its builder preflight — it never skips. +# `--check` tells you whether the copy is stale. +# +# NOTHING SECRET IS PRINTED. The key goes from `kubectl get secret` into the Gitea secrets API +# through shell variables and nowhere else. The script prints only names, sizes and the +# certificate's subject and expiry (public data). +# +# NEEDS: kubectl with a context for the cluster (default k3s-prod), jq, curl, openssl, base64, +# and the Gitea credential `git` already has stored for the host — the same requirements as +# infra/gitea/conformance-trigger-kubeconfig.sh, run from the same place. +set -euo pipefail +HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl +NS=buildkit; SECRET=buildkit-client-ci; KCTX="${KCTX:-k3s-prod}" +CHECK=0 +for a in "$@"; do + case "$a" in + --check) CHECK=1 ;; + -h|--help) sed -n '2,/^set -euo/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;; + *) echo "unknown argument: $a" >&2; exit 2 ;; + esac +done +die() { echo "ERROR: $*" >&2; exit 1; } +for t in kubectl jq curl openssl base64 git; do command -v "$t" >/dev/null || die "missing required tool: $t"; done + +echo "==> reading Secret ${NS}/${SECRET} (context ${KCTX}; values not shown)" +js="$(kubectl --context "$KCTX" -n "$NS" get secret "$SECRET" -o json)" \ + || die "cannot read ${NS}/${SECRET} — is platform/buildkit deployed and the Certificate Ready?" +CA="$(jq -r '.data["ca.crt"] // empty' <<<"$js" | base64 -d)" +CRT="$(jq -r '.data["tls.crt"] // empty' <<<"$js" | base64 -d)" +KEY="$(jq -r '.data["tls.key"] // empty' <<<"$js" | base64 -d)" +unset js +[ -n "$CA" ] && [ -n "$CRT" ] && [ -n "$KEY" ] || die "${NS}/${SECRET} lacks ca.crt/tls.crt/tls.key" + +# Sanity before anything is written: the leaf is a CLIENT cert issued by this CA, and the key +# belongs to it. A wrong pairing would otherwise surface only as a TLS failure in CI. +openssl verify -CAfile <(printf '%s\n' "$CA") <(printf '%s\n' "$CRT") >/dev/null 2>&1 \ + || die "tls.crt does not verify against ca.crt" +openssl x509 -noout -ext extendedKeyUsage <<<"$CRT" 2>/dev/null | grep -q "TLS Web Client Authentication" \ + || die "tls.crt is not a client-auth certificate" +[ "$(openssl x509 -noout -pubkey <<<"$CRT" | openssl sha256)" = "$(openssl pkey -pubout <<<"$KEY" 2>/dev/null | openssl sha256)" ] \ + || die "tls.key does not match tls.crt" +echo " $(openssl x509 -noout -subject <<<"$CRT") / $(openssl x509 -noout -enddate <<<"$CRT")" + +creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)" +U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"; unset creds +[ -n "$U" ] && [ -n "$P" ] || die "no stored credential for $HOST" +echo "==> authenticating to Gitea as '${U}' (secret not shown)" +api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; } +API="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets" + +if [ "$CHECK" -eq 1 ]; then + # Gitea never returns a secret's value, only its name and creation time. "Stale" therefore + # means: the secret was created before the certificate's current notBefore. + names="$(api "$API" | jq -c '[.[]? | {name, created_at}]')" + nb="$(date -u -d "$(openssl x509 -noout -startdate <<<"$CRT" | cut -d= -f2)" +%s)" + rc=0 + for n in BUILDKIT_CA BUILDKIT_CLIENT_CERT BUILDKIT_CLIENT_KEY; do + c="$(jq -r --arg n "$n" '.[] | select(.name==$n) | .created_at' <<<"$names")" + if [ -z "$c" ]; then echo " MISSING $n"; rc=1 + elif [ "$(date -u -d "$c" +%s)" -lt "$nb" ]; then echo " STALE $n (stored $c, certificate re-issued since)"; rc=1 + else echo " current $n (stored $c)"; fi + done + exit "$rc" +fi + +put() { # name value + local code + code="$(api -o /dev/null -w '%{http_code}' -X PUT -H 'Content-Type: application/json' \ + -d "$(jq -n --arg d "$2" '{data:$d}')" "${API}/$1")" + [ "$code" = 201 ] || [ "$code" = 204 ] || die "PUT $1 failed: HTTP ${code}" + echo " stored $1 (${#2} bytes, value not shown)" +} +echo "==> storing repository Actions secrets" +put BUILDKIT_CA "$CA" +put BUILDKIT_CLIENT_CERT "$CRT" +put BUILDKIT_CLIENT_KEY "$KEY" +unset CA CRT KEY + +# VERIFY BY READING BACK (names only — Gitea never returns values). curl -sS exits 0 on an HTTP +# error, which is how set-ci-token.sh once claimed a store that never happened. +names="$(api "$API" | jq -r '[.[]?.name] | join(" ")')" +for n in BUILDKIT_CA BUILDKIT_CLIENT_CERT BUILDKIT_CLIENT_KEY; do + grep -qw "$n" <<<"$names" || die "stored $n but it is NOT listed (${names}) — refusing to claim success" +done +echo "==> stored and VERIFIED present. Re-run the images workflow (workflow_dispatch) to use them." diff --git a/platform/buildkit/README.md b/platform/buildkit/README.md new file mode 100644 index 0000000..ed49634 --- /dev/null +++ b/platform/buildkit/README.md @@ -0,0 +1,74 @@ +# platform/buildkit — rootless BuildKit, the CI image builder + +The builder behind [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). It +builds `toolbox`, `agent-runner` and `conformance-runner`, and on `main` pushes them to the +Gitea registry. Flux reconciles it as the Kustomization `buildkit` +([`clusters/base/buildkit.yaml`](../../clusters/base/buildkit.yaml), `dependsOn: platform`). + +## Why it exists (operator decision on PR #81, 2026-10-29) + +CI jobs run as containers on the act runner's privileged DinD sidecar, and **cannot reach any +Docker daemon**. act_runner 0.2.11 forces `docker_host: "-"` for a `tcp://` DinD, and gitea/act +strips every bind mount when `valid_volumes` is empty. Measured on PR #81's first run. Giving jobs +that daemon would make every branch push root on the node. So the rule is: **no job may reach a +privileged daemon.** Builds go to this rootless daemon over mTLS instead. + +## Shape + +| Piece | What it is | +|---|---| +| `deployment.yaml` | `moby/buildkit:v0.34.1-rootless`, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The `securityContext` is commented line by line with how each item was measured | +| `certificates.yaml` | cert-manager: a self-signed CA used only here, plus the server cert `buildkitd-tls` and the CI client cert `buildkit-client-ci` | +| `service.yaml` | ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`, mTLS only. Never on the Gateway | +| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and the Gateway (`ingress` entity) for the registry push | +| `configmap.yaml` | `buildkitd.toml`: cache GC capped at 4 GB, because the cache is an `emptyDir` on node disk | +| `versions.env` | The CI side: pinned `buildx` (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource | + +## Admission + +**No Kyverno exception is needed.** +- Every ClusterPolicy that is actually applied (the list in + [`platform-config/kustomization.yaml`](../../platform-config/kustomization.yaml)) matches only + namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`. This namespace, like + `gitea-actions`, does not carry that label. +- The job-shape policies (`pm-agent-job-shape`, `conformance-trigger-job-shape`) match only their + own namespaces. +- There is no cluster-wide Pod Security default. This namespace is deliberately left without a + `pod-security` label, because PSS `baseline` forbids the `Unconfined` seccomp and AppArmor + profiles that rootless BuildKit needs. + +## Prerequisites (operator, in order) + +1. **Unprivileged user namespaces on the nodes.** Already true. Measured 2026-10-29 on all three + Debian 13.7 nodes (kernel 7.1.8): `kernel.unprivileged_userns_clone = 1`, + `user.max_user_namespaces = 127206`, and no Ubuntu-style AppArmor userns restriction. +2. **Merge.** Flux applies this directory. Check that the Kustomization `buildkit` is Ready and + that all three Certificates in namespace `buildkit` are READY. +3. **Copy the client certificate to CI.** Run + [`infra/gitea/buildkit-client-secret.sh`](../../infra/gitea/buildkit-client-secret.sh). + Re-run it yearly, after renewal; `--check` reports whether the copy is stale. +4. **Restart buildkitd after the server certificate renews.** It reads the certificate at start, + and cert-manager renews it 30 days before expiry, so a restart inside that window is needed. + +Until step 3 is done, the `images` workflow fails at its builder preflight with a message naming +the cause. It never skips. + +## What a build step can do + +The daemon runs with `--oci-worker-no-process-sandbox`, so build steps share its PID namespace. +buildkitd warns about this itself at startup. + +- **A build step CAN** signal or ptrace the daemon and read what it can read, including this + pod's TLS key and the build cache. +- **A build step CANNOT** reach the node. Everything runs as uid 1000 in a user namespace, + unprivileged, with no host mounts. +- **Who can run one:** anyone holding the client certificate, i.e. every CI job, including pull + requests. + +## Local equivalent + +The pinned image runs under rootless podman with the same restrictions: +`--cap-drop ALL --cap-add SETUID --cap-add SETGID --security-opt seccomp=unconfined --read-only`, +plus tmpfs at the same five paths. That is how the `securityContext` was measured. Rootless podman +maps only 65,537 uids, so the image's `/etc/subuid` (`user:100000:65536`) must be overridden with +two ranges that fit: `user:1:999` and `user:1001:64535`. diff --git a/platform/buildkit/certificates.yaml b/platform/buildkit/certificates.yaml new file mode 100644 index 0000000..ac0a280 --- /dev/null +++ b/platform/buildkit/certificates.yaml @@ -0,0 +1,100 @@ +--- +# mTLS for buildkitd, issued by cert-manager from a CA that exists ONLY for this purpose. +# +# buildkit-selfsigned (Issuer) --signs--> buildkit-ca (CA Certificate, 10y) +# buildkit-ca (Issuer) --signs--> buildkitd-tls (server, the daemon presents it) +# --signs--> buildkit-client-ci (client, CI presents it) +# +# buildkitd verifies EVERY client certificate against this CA (--tlscacert), so holding a client +# certificate signed by it is what "may build" means. Nothing else in the cluster trusts this CA, +# and it trusts nothing else. +# +# THE CLIENT CERTIFICATE REACHES CI AS ACTIONS SECRETS, not through the cluster: CI jobs hold no +# cluster credential (R-SEC-02). infra/gitea/buildkit-client-secret.sh copies ca.crt/tls.crt/ +# tls.key from the Secret below into the repository secrets BUILDKIT_CA, BUILDKIT_CLIENT_CERT and +# BUILDKIT_CLIENT_KEY without printing them. Re-run it after every renewal (see renewBefore). +# +# ROTATION. cert-manager renews both leaves `renewBefore` their expiry and rewrites the Secrets. +# * server: buildkitd reads its certificate at START, so the pod must restart inside the 30-day +# renewal window (a node reboot by kured, a Deployment rollout, or a manual restart). +# * client: the Actions secrets are a COPY; re-run the script. Until then CI fails loudly at +# its TLS preflight (.gitea/workflows/scripts/images.sh), never silently. +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: buildkit-selfsigned + namespace: buildkit +spec: + selfSigned: {} +--- +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: buildkit-ca + namespace: buildkit +spec: + isCA: true + commonName: buildkit-ca + secretName: buildkit-ca + duration: 87600h # 10 years: rotating the CA means re-issuing everything and re-copying + privateKey: + algorithm: ECDSA + size: 256 + issuerRef: + name: buildkit-selfsigned + kind: Issuer +--- +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: buildkit-ca + namespace: buildkit +spec: + ca: + secretName: buildkit-ca +--- +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: buildkitd-tls + namespace: buildkit +spec: + secretName: buildkitd-tls + commonName: buildkitd.buildkit.svc.cluster.local + dnsNames: + - buildkitd.buildkit.svc.cluster.local # what CI dials (platform/buildkit/versions.env) + - buildkitd.buildkit.svc + - buildkitd + duration: 8760h + renewBefore: 720h + usages: + - server auth + - digital signature + privateKey: + algorithm: ECDSA + size: 256 + rotationPolicy: Always + issuerRef: + name: buildkit-ca + kind: Issuer +--- +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: buildkit-client-ci + namespace: buildkit +spec: + secretName: buildkit-client-ci + commonName: gitea-actions-image-pipeline + duration: 8760h + renewBefore: 720h + usages: + - client auth + - digital signature + privateKey: + algorithm: ECDSA + size: 256 + rotationPolicy: Always + issuerRef: + name: buildkit-ca + kind: Issuer diff --git a/platform/buildkit/configmap.yaml b/platform/buildkit/configmap.yaml new file mode 100644 index 0000000..1c537a9 --- /dev/null +++ b/platform/buildkit/configmap.yaml @@ -0,0 +1,24 @@ +--- +# buildkitd.toml. Validated against the pinned image (v0.34.1-rootless): `buildctl debug workers +# -v` reports "Maximum used space: 4.29GB / Reserved space: 1.07GB" for the default policy. +# +# WHY 4GB. The cache lives in an emptyDir on the NODE's disk (deployment.yaml), and the nodes had +# 7-9 GB free when this was written. The largest image (toolbox) is ~1.4 GB uncompressed; the +# three images share their Fedora base. The emptyDir's sizeLimit sits above this cap, so GC +# trims the cache well before the kubelet would evict the pod for exceeding it. +apiVersion: v1 +kind: ConfigMap +metadata: + name: buildkitd-config + namespace: buildkit + labels: + app.kubernetes.io/name: buildkitd +data: + buildkitd.toml: | + [worker.oci] + enabled = true + gc = true + maxUsedSpace = "4GB" + reservedSpace = "1GB" + [worker.containerd] + enabled = false diff --git a/platform/buildkit/deployment.yaml b/platform/buildkit/deployment.yaml new file mode 100644 index 0000000..67d826c --- /dev/null +++ b/platform/buildkit/deployment.yaml @@ -0,0 +1,142 @@ +--- +# buildkitd, ROOTLESS. The builder behind .gitea/workflows/images.yaml; see README.md. +# +# THE SECURITY CONTEXT IS THE BOUNDARY, AND EVERY LINE OF IT WAS MEASURED (2026-10-29, the pinned +# image under podman with the same restrictions; decisions/2026-10-29-image-pipeline.md): +# * privileged: false — never. This is the whole point of the operator's decision on PR #81: +# the alternative was handing CI the privileged DinD daemon, i.e. root on the node. +# * runAsUser/Group 1000 — the image's `user`; rootlesskit maps it to uid 0 INSIDE a user +# namespace only. The nodes allow that (measured: kernel.unprivileged_userns_clone = 1, +# user.max_user_namespaces = 127206 on all three Debian 13 nodes). +# * capabilities: drop ALL, add SETUID + SETGID — the image's newuidmap/newgidmap are NOT +# setuid; they carry FILE capabilities, which need SETUID/SETGID in the bounding set. +# Measured: with drop ALL and nothing added, rootlesskit fails "newuidmap ... operation not +# permitted" and the daemon never starts. +# * allowPrivilegeEscalation: true — required for the same reason: no_new_privs (what `false` +# sets) stops a file capability from ever taking effect. It does NOT grant privileges the +# bounding set above does not contain. +# * seccompProfile: Unconfined — measured: with the default seccomp profile the daemon starts +# but every RUN step fails (the runtime cannot create the build step's namespaces/mounts). +# * appArmorProfile: Unconfined — upstream's documented requirement (buildkit +# examples/kubernetes, *.rootless.yaml); the nodes run AppArmor (enabled=Y) and containerd's +# default profile denies mount. NOT measured here: the test host runs SELinux, not AppArmor. +# * readOnlyRootFilesystem: true — measured to work with the five emptyDirs below. +# +# --oci-worker-no-process-sandbox: build steps share the daemon's PID namespace instead of getting +# their own (creating one needs an unmasked /proc, i.e. more privilege). Consequence, stated by +# buildkitd itself at startup: a build step can signal or ptrace the daemon and read what the +# daemon can read — including this pod's TLS key and the build cache. So a build step is trusted +# at the level of "may impersonate this builder", which is the level anyone holding the client +# certificate already has. It can NOT reach the node: everything still runs as uid 1000 in a user +# namespace, unprivileged, with no host mounts. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: buildkitd + namespace: buildkit + labels: + app.kubernetes.io/name: buildkitd +spec: + replicas: 1 + strategy: + type: Recreate # one builder, one cache; never two daemons fighting over a node's disk + selector: + matchLabels: + app.kubernetes.io/name: buildkitd + template: + metadata: + labels: + app.kubernetes.io/name: buildkitd + spec: + automountServiceAccountToken: false # the builder needs no Kubernetes API at all + enableServiceLinks: false + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 # makes the 0440 TLS Secret below readable by uid 1000, and nobody else + containers: + - name: buildkitd + # Pinned by digest (the multi-arch index). The kubernetes manager in renovate.json5 + # tracks platform/**, so Renovate proposes bumps; the CI client (buildx, pinned in + # versions.env) talks to any recent daemon. + image: docker.io/moby/buildkit:v0.34.1-rootless@sha256:f68752a5a94acfcdad2d07867b7075601a421638e17fc5f28efc1c814d0c4f89 + args: + - --addr + - unix:///run/user/1000/buildkit/buildkitd.sock # local only: the probes below + - --addr + - tcp://0.0.0.0:1234 # CI, mTLS only + - --tlscacert + - /certs/ca.crt # client certs are VERIFIED against this CA: no cert, no build + - --tlscert + - /certs/tls.crt + - --tlskey + - /certs/tls.key + - --config + - /etc/buildkit/buildkitd.toml + - --oci-worker-no-process-sandbox + ports: + - name: buildkit + containerPort: 1234 + protocol: TCP + readinessProbe: + exec: + command: ["buildctl", "debug", "workers"] + initialDelaySeconds: 5 + periodSeconds: 30 + livenessProbe: + exec: + command: ["buildctl", "debug", "workers"] + initialDelaySeconds: 30 + periodSeconds: 60 + timeoutSeconds: 10 + securityContext: + privileged: false + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + allowPrivilegeEscalation: true + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + add: ["SETUID", "SETGID"] + seccompProfile: + type: Unconfined + appArmorProfile: + type: Unconfined + resources: + # Build steps (dnf, npm, pip) run INSIDE this container's cgroup, so the limit covers + # the daemon and the build together. + requests: + cpu: 250m + memory: 512Mi + ephemeral-storage: 2Gi + limits: + memory: 4Gi + volumeMounts: + - { name: certs, mountPath: /certs, readOnly: true } + - { name: config, mountPath: /etc/buildkit, readOnly: true } + - { name: cache, mountPath: /home/user/.local/share/buildkit } + - { name: run, mountPath: /run/user } + - { name: tmpdir, mountPath: /home/user/.local/tmp } # the image's TMPDIR + - { name: tmp, mountPath: /tmp } + - { name: clicfg, mountPath: /home/user/.docker } # buildctl (the probes) writes here + volumes: + - name: certs + secret: + secretName: buildkitd-tls # cert-manager: ca.crt + tls.crt + tls.key + defaultMode: 0440 + - name: config + configMap: + name: buildkitd-config + - name: cache + emptyDir: + sizeLimit: 6Gi # GC caps use at 4GB (configmap.yaml); this is the backstop + - name: run + emptyDir: {} + - name: tmpdir + emptyDir: {} + - name: tmp + emptyDir: {} + - name: clicfg + emptyDir: {} diff --git a/platform/buildkit/kustomization.yaml b/platform/buildkit/kustomization.yaml new file mode 100644 index 0000000..064bf0f --- /dev/null +++ b/platform/buildkit/kustomization.yaml @@ -0,0 +1,13 @@ +--- +# platform/buildkit — rootless BuildKit, the CI image builder. Reconciled by the Flux +# Kustomization `buildkit` (clusters/base/buildkit.yaml). See README.md. +# versions.env is NOT a resource: it pins the CI-side client (buildx) and the address CI dials. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - namespace.yaml + - certificates.yaml + - configmap.yaml + - deployment.yaml + - service.yaml + - networkpolicy.yaml diff --git a/platform/buildkit/namespace.yaml b/platform/buildkit/namespace.yaml new file mode 100644 index 0000000..64ae767 --- /dev/null +++ b/platform/buildkit/namespace.yaml @@ -0,0 +1,26 @@ +--- +# buildkit — the image builder for CI (.gitea/workflows/images.yaml). Operator decision on PR #81 +# (2026-10-29): images are built by a ROOTLESS BuildKit daemon reached over mTLS, and NO CI job +# may reach a privileged daemon. See README.md for the whole design. +# +# A NAMESPACE OF ITS OWN, deliberately NOT `gitea-actions`: +# * The act-runner namespace exists to hold a PRIVILEGED DinD sidecar. Putting the builder +# beside it would put a TLS server key next to a privileged container for no benefit. +# * Its NetworkPolicies, Secrets and quota then belong to one component, and the ingress rule +# ("only the act-runner pod may connect") can be written from the outside. +# +# NO `app-namespace` LABEL, like gitea-actions: this is CI plumbing, not a user-facing app. That +# is also why NO Kyverno exception is needed — every applied ClusterPolicy +# (platform-config/kustomization.yaml) matches only namespaces labelled +# homelab.keithjackson.dev/app-namespace: "true", and none of the job-shape policies match here. +# +# NO pod-security.kubernetes.io LABEL, on purpose. Rootless BuildKit needs seccomp AND AppArmor +# `Unconfined` (measured — see deployment.yaml), which PSS `baseline` forbids; there is no +# cluster-wide PSA default here, so an unlabelled namespace admits it. It is NOT privileged and +# the deployment's securityContext is the boundary — read it before relaxing anything. +apiVersion: v1 +kind: Namespace +metadata: + name: buildkit + labels: + homelab.keithjackson.dev/ci: buildkit diff --git a/platform/buildkit/networkpolicy.yaml b/platform/buildkit/networkpolicy.yaml new file mode 100644 index 0000000..1ec331d --- /dev/null +++ b/platform/buildkit/networkpolicy.yaml @@ -0,0 +1,88 @@ +--- +# Default-deny for the namespace, then two narrow allows. +# +# WHO MAY CONNECT — AND WHY THE ACT-RUNNER POD IS THE RIGHT SOURCE (measured 2026-10-29). +# CI jobs do not run as pods. act_runner starts each job as a container on its DinD sidecar, on a +# docker bridge network INSIDE the act-runner pod's network namespace, and dockerd masquerades it: +# kubectl -n gitea-actions exec deploy/act-runner -c dind -- iptables -t nat -S POSTROUTING +# -A POSTROUTING -s 172.18.0.0/16 ! -o br-c0f13d041ad6 -j MASQUERADE +# -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE +# So a job's connection leaves with the act-runner POD's IP and therefore Cilium's identity for +# that pod (namespace gitea-actions, app.kubernetes.io/name=act-runner). That is exactly what the +# ingress rule names. It cannot distinguish one job from another, or a job from act_runner +# itself — per-job authentication is the mTLS client certificate, not this policy. What the +# policy adds is that NOTHING ELSE in the cluster can even open the port. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-deny + namespace: buildkit + labels: + homelab.keithjackson.dev/baseline: default-deny +spec: + podSelector: {} + policyTypes: + - Ingress + - Egress + ingress: [] + egress: + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 +--- +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: buildkitd-allow-ci-ingress + namespace: buildkit + labels: + app.kubernetes.io/name: buildkitd +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: buildkitd + ingress: + - fromEndpoints: + - matchLabels: + io.kubernetes.pod.namespace: gitea-actions + app.kubernetes.io/name: act-runner + toPorts: + - ports: + - { port: "1234", protocol: TCP } +--- +# WHAT A BUILD NEEDS TO REACH. Build steps download (dnf mirrors, GitHub releases, dl.k8s.io, +# npm, PyPI, Fedora/Docker registries) and the result is pushed to the Gitea registry. +# * world 443/80: the downloads. 80 because Fedora's mirror list includes plain-http mirrors +# (dnf verifies package GPG signatures, not transport). +# * ingress (the Gateway's Envoy, reserved:ingress): the push. gitea.home.keithjackson.dev +# resolves in-cluster to the Gateway VIP (split-horizon CoreDNS), and Cilium hands that +# traffic to the Gateway, whose identity is `ingress` — the same identity gitea's own +# ingress allow admits. +# Nothing else in the cluster: not the API server, not other pods. A build step is code from a +# pull request. +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: buildkitd-allow-egress + namespace: buildkit + labels: + app.kubernetes.io/name: buildkitd +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: buildkitd + egress: + - toEntities: + - world + toPorts: + - ports: + - { port: "443", protocol: TCP } + - { port: "80", protocol: TCP } + - toEntities: + - ingress diff --git a/platform/buildkit/service.yaml b/platform/buildkit/service.yaml new file mode 100644 index 0000000..705f993 --- /dev/null +++ b/platform/buildkit/service.yaml @@ -0,0 +1,19 @@ +--- +# The address CI dials: tcp://buildkitd.buildkit.svc.cluster.local:1234 (platform/buildkit/ +# versions.env), mTLS only. ClusterIP — never exposed on the Gateway. +apiVersion: v1 +kind: Service +metadata: + name: buildkitd + namespace: buildkit + labels: + app.kubernetes.io/name: buildkitd +spec: + type: ClusterIP + selector: + app.kubernetes.io/name: buildkitd + ports: + - name: buildkit + port: 1234 + targetPort: buildkit + protocol: TCP diff --git a/platform/buildkit/versions.env b/platform/buildkit/versions.env new file mode 100644 index 0000000..38b9261 --- /dev/null +++ b/platform/buildkit/versions.env @@ -0,0 +1,15 @@ +# platform/buildkit/versions.env — the CI side of the builder. Sourced by +# .gitea/workflows/scripts/images.sh. NOT a Kubernetes resource (kustomization.yaml omits it). +# +# The CLIENT. CI drives the daemon with `buildx` (remote driver) rather than bare `buildctl` +# because the image pipeline needs `bake`'s `target:` contexts: on a pull request agent-runner is +# built FROM the toolbox built in the same run, which never exists in any registry. +# Pinned with the sha256 from the release's own checksums.txt +# (https://github.com/docker/buildx/releases/download/v0.38.1/checksums.txt); the pipeline +# refuses a download that does not match. Bump BOTH lines together. +BUILDX_VERSION=0.38.1 +BUILDX_SHA256=e25d5cd675092bb5d1d03632fdc46cdf3fede7b514dc5f023a105e00d3ce5ad4 + +# The DAEMON's address and the name its certificate carries (platform/buildkit/certificates.yaml). +BUILDKIT_HOST=tcp://buildkitd.buildkit.svc.cluster.local:1234 +BUILDKIT_SERVERNAME=buildkitd.buildkit.svc.cluster.local diff --git a/platform/pm-agent/README.md b/platform/pm-agent/README.md index 5247aff..809c6b5 100644 --- a/platform/pm-agent/README.md +++ b/platform/pm-agent/README.md @@ -47,9 +47,9 @@ Both are deliberate, and both fail **visibly** rather than silently: 1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and `PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder until the first - publish. `.gitea/workflows/images.yaml` builds `agent-runner` with - `infra/agent/build-image.sh` (the same script, and the same in-image verification, you can - run locally), and on a push to `main` publishes it to + publish. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit + (`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that + `infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to `gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-` and `main`. Renovate then follows the `main` tag through the `# renovate-digest:` marker above **both** lines and moves them together in one pull request. Nobody builds-and-pushes this image from From 04eb38584ab7c1885fcfb19bd766435e284239b2 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Thu, 29 Oct 2026 22:25:12 -0500 Subject: [PATCH 6/7] fix(conformance): workload-mTLS certificates may use a private CA, if they prove they are workload-internal R-TLS-01/docs/08 govern externally reachable TLS and ADR-0006 puts workload mTLS out of scope, but tls-lifecycle-automated required ACME for every Certificate, which platform/buildkit's internal CA cannot satisfy. Clause (5b) accepts the tls-scope=workload-mtls annotation only when the certificate proves it: no Gateway serves it, internal names only, namespaced selfSigned/ca issuer backed by an exempt CA certificate. Also fixes yamllint comma spacing in the buildkitd mounts. Co-Authored-By: Claude Opus 5.5 --- .../tls-lifecycle-automated/check.sh | 70 +++++++++++++++++++ decisions/2026-10-29-image-pipeline.md | 17 +++++ platform/buildkit/certificates.yaml | 15 ++++ platform/buildkit/deployment.yaml | 14 ++-- 4 files changed, 109 insertions(+), 7 deletions(-) diff --git a/conformance/tests/l0-static/tls-lifecycle-automated/check.sh b/conformance/tests/l0-static/tls-lifecycle-automated/check.sh index 294b4b1..7f1e5e9 100755 --- a/conformance/tests/l0-static/tls-lifecycle-automated/check.sh +++ b/conformance/tests/l0-static/tls-lifecycle-automated/check.sh @@ -42,6 +42,19 @@ # whose every solver is an automated challenge type (dns01 / http01) — the # same code path serves issuance AND renewal, so a solver a human must # complete would defeat both; +# (5b) the ONE exception to (5) — WORKLOAD mTLS, added 2026-10-29 — is a Certificate +# annotated `homelab.keithjackson.dev/tls-scope: workload-mtls`. The spec's TLS +# requirements are about EXTERNALLY REACHABLE services (R-TLS-01; docs/08), and +# ADR-0006 says outright that "mTLS between workloads is a separate concern … not +# covered here". A public ACME CA cannot issue for a `*.svc.cluster.local` name or a +# client-auth certificate at all. The annotation is NOT trusted: an exempt +# Certificate must PROVE it is workload-internal — its Secret is served by NO Gateway +# listener, every DNS name / commonName is cluster-internal (a bare label or +# `*.svc[.cluster.local]`), it carries no IP/URI/email SANs, and its issuer is a +# NAMESPACED Issuer in its own namespace whose spec is `selfSigned` or a `ca` backed +# by a Secret that a tracked, likewise-exempt Certificate produces. Issuance and +# renewal stay unattended (cert-manager renews CA-issued certs itself), and (9) +# still applies. Every exemption is printed. First user: platform/buildkit. # (6) each dns01 solver's provider credential is referenced BY NAME and that # Secret is delivered SOPS-encrypted from git via a kustomization — the # operator's one-time token hand-off (docs/08 OQ-4), never a per-cert step; @@ -265,6 +278,7 @@ CERT_SECRETS: dict[tuple[str, str], str] = { # (namespace, secretName) for rel, d in CERTS } listeners_checked = 0 +GATEWAY_SECRETS: set[tuple[str, str]] = set() # every (namespace, Secret) a listener serves for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"): gw_ns = ns_of(gw) for lis in ((gw.get("spec") or {}).get("listeners") or []): @@ -280,6 +294,7 @@ for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"): for r in refs: kind = str(r.get("kind", "Secret")) key = (str(r.get("namespace") or gw_ns), str(r.get("name"))) + GATEWAY_SECRETS.add(key) check(kind == "Secret", f"{rel}: listener {lis.get('name')!r} certificateRef kind is {kind!r}; only " "Secret refs can be traced to a cert-manager Certificate") @@ -324,8 +339,61 @@ CM_NS = ns_of(cm_releases[0][1], "cert-manager") if cm_releases else "cert-manag ISSUER_INDEX = {(d.get("kind"), name_of(d)): (rel, d) for rel, d in ISSUERS} solver_creds: list[tuple[str, str, str, str]] = [] # (issuer, ns, secretName, ref field) issuers_verified: set[str] = set() + +# (5b) workload-mTLS exemption — see the header. Proven, never taken on trust. +SCOPE_ANN = "homelab.keithjackson.dev/tls-scope" +INTERNAL_NAME = re.compile(r"^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*$") +NS_ISSUERS = {(ns_of(d), name_of(d)): (rel, d) for rel, d in of_kind("Issuer", "cert-manager.io/")} +EXEMPT_SECRETS = {(ns_of(d), str((d.get("spec") or {}).get("secretName", ""))) + for _, d in CERTS + if ((d.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN) == "workload-mtls"} +exempted: list[str] = [] + +def workload_mtls_ok(rel: str, cert: dict) -> bool: + """True if `cert` claims the workload-mTLS scope AND proves it; records failures.""" + ann = ((cert.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN) + if ann is None: + return False + cns, cname = ns_of(cert), name_of(cert) + spec = cert.get("spec") or {} + ok = True + def need(cond, msg): + nonlocal ok + check(cond, f"{rel}: Certificate {cns}/{cname} claims {SCOPE_ANN}={ann!r} but {msg}") + ok = ok and bool(cond) + need(ann == "workload-mtls", "the only accepted value is 'workload-mtls'") + need((cns, str(spec.get("secretName", ""))) not in GATEWAY_SECRETS, + "its Secret is served by a Gateway listener — that is external TLS, which must be ACME (R-TLS-01)") + names = [str(n) for n in (spec.get("dnsNames") or [])] + if spec.get("commonName"): + names.append(str(spec["commonName"])) + for n in names: + internal = INTERNAL_NAME.match(n) and ("." not in n or n.endswith(".svc") or n.endswith(".svc.cluster.local")) + need(bool(internal), f"name {n!r} is not cluster-internal (bare label or *.svc[.cluster.local])") + for f in ("ipAddresses", "uris", "emailAddresses"): + need(not spec.get(f), f"it carries {f}, which a workload-internal certificate has no use for") + ref = spec.get("issuerRef") or {} + need(str(ref.get("kind", "Issuer")) == "Issuer", + "its issuer is not a NAMESPACED Issuer — a ClusterIssuer could sign for any namespace") + hit = NS_ISSUERS.get((cns, str(ref.get("name", "")))) + need(hit is not None, f"its Issuer {ref.get('name')!r} is not tracked in namespace {cns!r}") + if hit is not None: + ispec = hit[1].get("spec") or {} + kinds = sorted(ispec) + need(kinds in (["selfSigned"], ["ca"]), + f"its Issuer is {kinds} — only selfSigned or ca are accepted for workload mTLS") + if kinds == ["ca"]: + need((cns, str((ispec.get("ca") or {}).get("secretName", ""))) in EXEMPT_SECRETS, + "its CA Issuer's Secret is not produced by a tracked workload-mtls Certificate " + "in the same namespace — the CA itself would have to be minted by hand") + if ok: + exempted.append(f"{cns}/{cname}") + return ok + for rel, cert in CERTS: spec = cert.get("spec") or {} + if workload_mtls_ok(rel, cert): + continue # (5b): proven workload-internal; (9) below still applies ref = spec.get("issuerRef") or {} kind = str(ref.get("kind", "Issuer")) per_env = resolve_per_env(str(ref.get("name", ""))) @@ -379,6 +447,8 @@ for rel, cert in CERTS: solver_creds.append((iname, want_ns, n, f"{provider}.{field}")) check(len(issuers_verified) > 0, "no issuer was reachable from any Certificate — (5) proved nothing") +if exempted: + print(f" (5b) workload-mTLS certificates exempted from ACME after proving it: {', '.join(exempted)}") # (6) the solver credential is delivered, SOPS-encrypted, from git — never per-cert by hand. check(len(solver_creds) > 0, diff --git a/decisions/2026-10-29-image-pipeline.md b/decisions/2026-10-29-image-pipeline.md index 40c3909..9620e35 100644 --- a/decisions/2026-10-29-image-pipeline.md +++ b/decisions/2026-10-29-image-pipeline.md @@ -85,6 +85,23 @@ valid_volumes) is **untouched**. - AppArmor enabled (`Y`). So this is a satisfied prerequisite, not an open one. +- **The TLS conformance check needed a narrow, self-proving exception** (`conformance/**`, so this + changes what green means; flagged for the reviewer). + - The first gate run failed L0 `tls-lifecycle-automated` clause (5): "every Certificate's + issuer must be ACME". That is broader than the spec it serves: + - R-TLS-01 and docs/08 concern *externally reachable* services; + - ADR-0006 says "mTLS between workloads is a separate concern … not covered here"; + - a public CA cannot issue for a `*.svc.cluster.local` name or a client-auth certificate. + - The new clause (5b) exempts a Certificate annotated + `homelab.keithjackson.dev/tls-scope: workload-mtls` **only if it proves it**: + - no Gateway listener serves its Secret; + - every DNS name and the commonName are a bare label or `*.svc[.cluster.local]`; + - it has no IP, URI or email SANs; + - its issuer is a namespaced `selfSigned`/`ca` Issuer in its own namespace; + - a `ca` issuer's Secret is produced by a likewise-exempt tracked Certificate. + - Exemptions are printed, and clause (9), the renewal arithmetic, still applies. + - **Negative-tested:** a public DNS name, and a Gateway serving `buildkitd-tls`, each fail the + check with a specific message. - **`infra/gitea/buildkit-client-secret.sh`** (operator action): copies `ca.crt`, `tls.crt` and `tls.key` from the Secret `buildkit/buildkit-client-ci` into the Actions secrets `BUILDKIT_CA`, `BUILDKIT_CLIENT_CERT` and `BUILDKIT_CLIENT_KEY`. Before writing anything it checks that: diff --git a/platform/buildkit/certificates.yaml b/platform/buildkit/certificates.yaml index ac0a280..2f5b2b0 100644 --- a/platform/buildkit/certificates.yaml +++ b/platform/buildkit/certificates.yaml @@ -14,6 +14,15 @@ # tls.key from the Secret below into the repository secrets BUILDKIT_CA, BUILDKIT_CLIENT_CERT and # BUILDKIT_CLIENT_KEY without printing them. Re-run it after every renewal (see renewBefore). # +# WHY A PRIVATE CA AND NOT LET'S ENCRYPT. The platform's TLS rule (R-TLS-01/02, docs/08, +# ADR-0006: ACME DNS-01) is about EXTERNALLY REACHABLE services; ADR-0006 itself says "mTLS between +# workloads is a separate concern … not covered here", and a public CA cannot issue for a +# `*.svc.cluster.local` name or a client-auth certificate anyway. buildkitd is ClusterIP-only. Each +# Certificate below is therefore annotated `homelab.keithjackson.dev/tls-scope: workload-mtls`, +# which the L0 check tls-lifecycle-automated accepts ONLY if the certificate proves it: no +# Gateway serves its Secret, every name is cluster-internal, and the issuer is a namespaced +# selfSigned/CA Issuer whose CA is itself issued here. Issuance and renewal stay unattended. +# # ROTATION. cert-manager renews both leaves `renewBefore` their expiry and rewrites the Secrets. # * server: buildkitd reads its certificate at START, so the pod must restart inside the 30-day # renewal window (a node reboot by kured, a Deployment rollout, or a manual restart). @@ -32,6 +41,8 @@ kind: Certificate metadata: name: buildkit-ca namespace: buildkit + annotations: + homelab.keithjackson.dev/tls-scope: workload-mtls # see the header: R-TLS-01 is about external TLS spec: isCA: true commonName: buildkit-ca @@ -58,6 +69,8 @@ kind: Certificate metadata: name: buildkitd-tls namespace: buildkit + annotations: + homelab.keithjackson.dev/tls-scope: workload-mtls spec: secretName: buildkitd-tls commonName: buildkitd.buildkit.svc.cluster.local @@ -83,6 +96,8 @@ kind: Certificate metadata: name: buildkit-client-ci namespace: buildkit + annotations: + homelab.keithjackson.dev/tls-scope: workload-mtls spec: secretName: buildkit-client-ci commonName: gitea-actions-image-pipeline diff --git a/platform/buildkit/deployment.yaml b/platform/buildkit/deployment.yaml index 67d826c..ae057ef 100644 --- a/platform/buildkit/deployment.yaml +++ b/platform/buildkit/deployment.yaml @@ -114,13 +114,13 @@ spec: limits: memory: 4Gi volumeMounts: - - { name: certs, mountPath: /certs, readOnly: true } - - { name: config, mountPath: /etc/buildkit, readOnly: true } - - { name: cache, mountPath: /home/user/.local/share/buildkit } - - { name: run, mountPath: /run/user } - - { name: tmpdir, mountPath: /home/user/.local/tmp } # the image's TMPDIR - - { name: tmp, mountPath: /tmp } - - { name: clicfg, mountPath: /home/user/.docker } # buildctl (the probes) writes here + - { name: certs, mountPath: /certs, readOnly: true } + - { name: config, mountPath: /etc/buildkit, readOnly: true } + - { name: cache, mountPath: /home/user/.local/share/buildkit } + - { name: run, mountPath: /run/user } + - { name: tmpdir, mountPath: /home/user/.local/tmp } # the image's TMPDIR + - { name: tmp, mountPath: /tmp } + - { name: clicfg, mountPath: /home/user/.docker } # buildctl (the probes) writes here volumes: - name: certs secret: From d2c4bd5194ca56a095766c22eb11a8c1ed2f2092 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Fri, 30 Oct 2026 00:38:54 -0500 Subject: [PATCH 7/7] fix(buildkit): allow buildkitd egress to Gitea's backend so the Gateway stops 403ing the push Images run 337's push failed with "403 Forbidden: Access denied". Measured read-only: Gitea never logged the request; an unauthenticated GET /v2/ via the Gateway gets 401 from Gitea when sent from act-runner (egress unrestricted) but 403 from buildkitd; Hubble logs it as an L7 http-request DROPPED with traffic_direction EGRESS. Cilium >= 1.16.8 (CVE-2025-30162) re-checks the source pod's egress against the proxied backend identity, and buildkitd's egress only allowed world and the ingress entity. Same fix as 68690e5 for Keycloak. Gitea's ingress already admits reserved:ingress and is unchanged. Co-Authored-By: Claude Opus 5.5 --- decisions/2026-10-30-gitea-admit-buildkit.md | 81 ++++++++++++++++++++ platform/buildkit/README.md | 2 +- platform/buildkit/networkpolicy.yaml | 21 +++++ 3 files changed, 103 insertions(+), 1 deletion(-) create mode 100644 decisions/2026-10-30-gitea-admit-buildkit.md diff --git a/decisions/2026-10-30-gitea-admit-buildkit.md b/decisions/2026-10-30-gitea-admit-buildkit.md new file mode 100644 index 0000000..8e1e405 --- /dev/null +++ b/decisions/2026-10-30-gitea-admit-buildkit.md @@ -0,0 +1,81 @@ +--- +date: 2026-10-30 +branch: fix/gitea-admit-buildkit +title: The registry push 403 is Cilium refusing buildkitd's EGRESS at the Gateway — fixed in platform/buildkit, not in Gitea's ingress +--- + +## What changed + +**`platform/buildkit/networkpolicy.yaml`** — `buildkitd-allow-egress` gains one rule: +`toEndpoints {io.kubernetes.pod.namespace: gitea, app.kubernetes.io/name: gitea}` on TCP 3000, +with a comment carrying the measurement. `platform/buildkit/README.md`'s policy row says the same. + +**What did NOT change, deliberately:** `catalog/gitea.yaml` `inClusterClients` and +`apps/wave-0/gitea.yaml`. The brief proposed admitting `buildkit` there. The evidence below says +that is the wrong policy: Gitea's ingress is not what refuses the push, and adding it would only +open the direct ClusterIP path from build code to Gitea, which nothing uses. + +### Why: the brief's hypothesis was half right + +Images run 337 failed its push with `HEAD .../blobs/sha256:5855... 403 Forbidden: Access denied`. +The PM's hypothesis was that this was Cilium's Envoy denying the request, not Gitea. **That part +holds.** The hypothesis also said the denial came from Gitea's *ingress* policy, which does not +admit the `buildkit` namespace. **That part is wrong.** The denial is enforced on buildkitd's own +*egress*. Since Cilium 1.16.8/1.17.2 (CVE-2025-30162; prod runs `cilium:v1.20.1`), the Gateway's +L7 load balancer re-evaluates the source pod's egress policy against the identity of the backend +it resolves to. buildkitd's egress allowed `world` :443/:80 and the `ingress` entity. That gets +the TCP connection to Envoy, but no rule matches Gitea's backend identity, so Envoy answers +`403 Access denied` without forwarding the request. This is the failure commit `68690e5` fixed for +every SSO app's hop to Keycloak, and the fix has the same shape. Keycloak's ingress names no app +namespaces either. + +## What I verified + +All measurements are read-only, against `k3s-prod`, on 2026-10-30 around 05:27–05:37 UTC. + +- **Gitea never saw the failed push.** Its router log for the previous 12h had 316 `/v2/` lines + and **0** that were `toolbox/blobs`, PUT, POST or PATCH. It also had **0** from buildkitd's pod + IP (`10.0.0.7`). In the same window it logged the CI job's own `HEAD .../toolbox/manifests/ + sha-0dd57fbaf002 → 404` from the act-runner pod (`10.0.0.38`). Requests that came through the + Gateway reach Gitea and get logged. This one never arrived. +- **A reproduction with no credentials.** An unauthenticated `GET https://gitea.home.keithjackson.dev/v2/` + (Gateway VIP `192.168.2.20`) gave: + - from the buildkitd pod: **`403 Forbidden`**, with no matching line in Gitea's log; + - from the act-runner pod's `dind` container: **`401 Unauthorized`**, which is Gitea's normal + registry challenge, logged by Gitea as `GET /v2/ for 10.0.0.38:0, 401`. + + No token was involved, so this cannot be a package-permission problem. The bot account and the + `REGISTRY_PUSH_TOKEN` were not the cause, and the operator does not need to change them. +- **Hubble names the enforcement point.** On `cilium-55jv8`, buildkitd's node: + `buildkit/buildkitd-859d67c8f8-x44nh (ingress) -> gateway/cilium-gateway-shared:443 http-request + DROPPED (HTTP/1.1 GET https://gitea.home.keithjackson.dev/v2/)`, followed by `http-response + FORWARDED (HTTP/1.1 403 0ms)`. In the JSON form, **`traffic_direction: EGRESS`**. Envoy wrote + the 403 itself in 0 ms. +- **Gitea's ingress does not need the source namespace.** The act-runner pod's namespace + (`gitea-actions`) is **not** in Gitea's ingress allows. Its NetworkPolicy + `default-deny-ingress-allow-egress` has an `{}` egress rule, so its egress is unrestricted. Its + requests through the Gateway still reach Gitea. `cilium-dbg bpf policy get` on Gitea's endpoint + (1876 on `cilium-9rk5x`) shows all Gateway traffic admitted as `reserved:ingress 3000/TCP` + (391 MB, 1.19 M packets) and no row for `gitea-actions`. The source-identity check therefore + happens on the source's egress, not on the destination's ingress. +- **The policy gap is visible in the BPF map.** `bpf policy get 83` (buildkitd) has these egress + rows and nothing else: `reserved:world` 80/443, `reserved:ingress ANY`, and kube-dns 53. There + is no Gitea identity. +- The edited YAML parses (three documents, names unchanged). + +## What I did NOT verify + +- **That this rule turns the 403 into a 401/200. UNVERIFIED.** That needs the policy applied, which + is a Flux/operator action. My confidence is high but not proven, and it rests on two things. The + Hubble verdict is EGRESS. Commit `68690e5` made the identical change for Keycloak and recorded + "Verified live: headlamp-ns -> keycloak discovery 403 -> 200". +- **The port restriction.** The rule names `3000`, Gitea's container port, which is the upstream + port Envoy connects to. The Keycloak precedent has no port. If the 403 survives, the first thing + to test is whether L7LB enforcement uses the frontend :443 instead. I expect it does not. +- **Run 337's own flows.** The Hubble ring buffer had already rotated past them, and buildkitd had + been restarted since (the pod was 18 minutes old). The reproduction above uses the same path, the + same pod spec and the same policy, but it is not the original request. +- Envoy's access logs in the `cilium-envoy` pods. Hubble's L7 flow already shows the drop, so I + did not read them. +- The push itself, the Renovate digest flow, and whether the bot team's `Packages=Write` is enough + for a first push that creates a new package. These only become testable after this lands. diff --git a/platform/buildkit/README.md b/platform/buildkit/README.md index ed49634..4e4bf32 100644 --- a/platform/buildkit/README.md +++ b/platform/buildkit/README.md @@ -20,7 +20,7 @@ privileged daemon.** Builds go to this rootless daemon over mTLS instead. | `deployment.yaml` | `moby/buildkit:v0.34.1-rootless`, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The `securityContext` is commented line by line with how each item was measured | | `certificates.yaml` | cert-manager: a self-signed CA used only here, plus the server cert `buildkitd-tls` and the CI client cert `buildkit-client-ci` | | `service.yaml` | ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`, mTLS only. Never on the Gateway | -| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and the Gateway (`ingress` entity) for the registry push | +| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and, for the registry push, the Gateway (`ingress` entity) **plus** Gitea's backend identity on :3000 — Cilium's Gateway re-checks this pod's egress against the backend it proxies to, and without that rule answers `403 Access denied` itself (measured 2026-10-30, `decisions/2026-10-30-gitea-admit-buildkit.md`) | | `configmap.yaml` | `buildkitd.toml`: cache GC capped at 4 GB, because the cache is an `emptyDir` on node disk | | `versions.env` | The CI side: pinned `buildx` (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource | diff --git a/platform/buildkit/networkpolicy.yaml b/platform/buildkit/networkpolicy.yaml index 1ec331d..d8155c5 100644 --- a/platform/buildkit/networkpolicy.yaml +++ b/platform/buildkit/networkpolicy.yaml @@ -64,6 +64,20 @@ spec: # resolves in-cluster to the Gateway VIP (split-horizon CoreDNS), and Cilium hands that # traffic to the Gateway, whose identity is `ingress` — the same identity gitea's own # ingress allow admits. +# * gitea's BACKEND identity, :3000 only: ALSO the push, and the half that `ingress` alone does +# not cover. Cilium >= 1.16.8/1.17.2 (CVE-2025-30162; this cluster runs 1.20.1) makes the +# Gateway's Envoy re-evaluate THIS pod's egress policy against the identity of the backend it +# resolves the request to (enforce_policy_on_l7lb). `toEntities: ingress` gets the TCP +# connection to Envoy; without this rule Envoy then refuses the HTTP request itself with +# `403 Access denied`, and Gitea never sees it. Measured 2026-10-30 (images run 337 failed +# this way): Hubble on buildkitd's node logged `buildkit/buildkitd (ingress) -> +# gateway/cilium-gateway-shared:443 http-request DROPPED`, traffic_direction EGRESS, and +# Gitea's router log had no entry for it — while the same unauthenticated GET /v2/ from the +# act-runner pod (egress unrestricted) reached Gitea and got its normal 401. It is the same +# fix catalog/render.py applies for every SSO app's hop to Keycloak (commit 68690e5). +# Gitea's INGRESS needs nothing: the request arrives there as `reserved:ingress`, already +# admitted. Do not add buildkit to catalog/gitea.yaml's inClusterClients — that would open +# the ClusterIP path from build code to Gitea, which nothing here uses. # Nothing else in the cluster: not the API server, not other pods. A build step is code from a # pull request. apiVersion: cilium.io/v2 @@ -86,3 +100,10 @@ spec: - { port: "80", protocol: TCP } - toEntities: - ingress + - toEndpoints: + - matchLabels: + io.kubernetes.pod.namespace: gitea + app.kubernetes.io/name: gitea + toPorts: + - ports: + - { port: "3000", protocol: TCP }