Compare commits

..
Author SHA1 Message Date
jakman85andClaude Opus 5.5 04eb38584a fix(conformance): workload-mTLS certificates may use a private CA, if they prove they are workload-internal
gate / gate (pull_request) Successful in 1m58s
images / images (pull_request) Failing after 7s
gate / conformance (pull_request) Has been skipped
R-TLS-01/docs/08 govern externally reachable TLS and ADR-0006 puts workload mTLS out of scope, but
tls-lifecycle-automated required ACME for every Certificate, which platform/buildkit's internal CA
cannot satisfy. Clause (5b) accepts the tls-scope=workload-mtls annotation only when the
certificate proves it: no Gateway serves it, internal names only, namespaced selfSigned/ca issuer
backed by an exempt CA certificate. Also fixes yamllint comma spacing in the buildkitd mounts.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5 f65c12b395 feat(images): build on rootless BuildKit over mTLS; verification moves into Containerfile verify stages
Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched.

- platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by
  digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined,
  read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP
  :1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it).
  No Kyverno exception: every applied policy matches app-namespace=true only.
- Each Containerfile names its shipped stage and ends in a `verify` stage holding every former
  `docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for
  the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now.
- images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify
  targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes.
  Fails loudly when the builder secrets are absent or buildkitd is unreachable.
- infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5 8f35234fcc fix(images): name the measured runner blocker; the job container gets no Docker daemon
PR #81's first images run failed in 6s: act_runner 0.2.11 forces docker_host '-' for a tcp://
DOCKER_HOST, and gitea/act drops every bind when valid_volumes is empty. Giving jobs the
privileged DinD daemon is node-root for any workflow author, so it is left to the operator;
options are in the handoff entry. The driver's failure now names the cause.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5 e539109830 feat(images): build, verify and publish the three images in CI; Renovate carries the digests
Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a
workstation build/push. Nothing had ever been pushed to the Gitea registry.

- .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main
  builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as
  sha-<commit> + main, reading the digest back from the registry. Only changed images build.
- Each image's own build-image.sh does the build and in-image verification, now engine-neutral
  (DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh.
- Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex
  manager for them. The worker Job template is deliberately untracked (it would loop).
- renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the
  current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION.
- Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in
  Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85 27da5555a4 Merge pull request 'fix(renovate): the workflow still named renovate.json, so every run since #79 died with ENOENT' (#80) from fix/renovate-config-filename into main
gate / gate (push) Successful in 1m54s
gate / conformance (push) Has been skipped
Reviewed-on: #80
Reviewed-by: Keith Jackson <[email protected]>
2026-10-30 04:48:42 +00:00
jakman85andClaude Opus 5.5 da1b36eb64 fix(renovate): pin renovate-version — the action's default (39) rejects renovate.json5
gate / gate (pull_request) Successful in 1m55s
gate / conformance (pull_request) Has been skipped
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 21:49:38 -05:00
jakman85andClaude Opus 5.5 aacf361bb8 fix(renovate): the workflow still named renovate.json, so every run since #79 died with ENOENT
gate / gate (pull_request) Failing after 1m58s
gate / conformance (pull_request) Has been skipped
#79 renamed the config to renovate.json5 and left configurationFile pointing at the old
name. Runs 315-322 (2026-10-22..29) all failed in ~1s before Renovate started.

Also add a ci/lint.sh assertion that every workflow configurationFile: resolves to a real
path — valid YAML, so yamllint could not catch it, and the gate stayed green throughout.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 20:58:15 -05:00
jakman85 1301af9a4c Merge pull request 'fix(renovate): the config was INVALID, so the first manual run errored out' (#79) from fix/renovate-invalid-config into main
gate / gate (push) Successful in 1m55s
gate / conformance (push) Has been skipped
renovate / renovate (push) Failing after 9s
Reviewed-on: #79
Reviewed-by: Keith Jackson <[email protected]>
2026-10-18 14:29:34 +00:00
jakman85 42ce001d71 Merge pull request 'fix(renovate): the cron fired outside Renovate's own window, so it would never open a PR' (#77) from fix/renovate-cron-window into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #77
Reviewed-by: Keith Jackson <[email protected]>
2026-10-18 14:28:59 +00:00
jakman85 84bb0f0acd Merge pull request 'fix(pihole): make the split-horizon answer cacheable (local-ttl=60)' (#78) from fix/pihole-local-ttl into main
gate / gate (push) Successful in 1m53s
gate / conformance (push) Has been skipped
Reviewed-on: #78
Reviewed-by: Keith Jackson <[email protected]>
2026-10-18 08:19:28 +00:00
jakman85andClaude Opus 5 0d4bd7297d fix(renovate): the cron fired outside Renovate's own window, so it would never open a PR
gate / conformance (pull_request) Has been skipped
gate / gate (pull_request) Successful in 1m55s
.gitea/workflows/renovate.yaml shipped with `cron: "15 3 * * *"`. Gitea Actions
cron is UTC, and renovate.json carries schedule ["after 3am and before 6am"]
with timezone America/New_York -- and THAT inner schedule is authoritative:
Renovate re-checks it and declines to open PRs outside it.

  03:15 UTC  ->  23:15 EDT, the PREVIOUS EVENING in New York
  window     ->  03:00-06:00 local   =>  NOT INSIDE (3h45m early)

The original comment claimed the cron sat "slightly ahead" of the window. It
sat 3h45m behind it, on the previous day. Every nightly run would have started,
logged "not within schedule", opened ZERO PRs and exited SUCCESSFULLY -- a green
tick forever, indistinguishable from "nothing needed updating", hiding the ~30
pending updates PR #73 existed to surface.

Changed to `15 8 * * *`, chosen to survive DST rather than to look tidy:
  EDT (UTC-4): local 03:00-06:00 == 07:00-10:00 UTC
  EST (UTC-5): local 03:00-06:00 == 08:00-11:00 UTC
08:15 UTC is inside BOTH (04:15 EDT / 03:15 EST), verified by computation at
both extremes, so neither switchover needs an edit. A cron cannot follow DST.

Also confirmed while here: RENOVATE_TOKEN now exists as a repo Actions secret,
so the workflow's fail-hard preflight will pass.

STILL UNRESOLVED and the likelier remaining bug: renovate.json's
`enabledManagers` lists bare `regex`, where modern Renovate names the custom
manager `custom.regex`. If the bare name is ignored, every inline `# renovate:`
marker in the tree does nothing. Deliberately NOT changed -- guessing at a key
that could fail validation and take the whole run down is worse than a question
one dry run answers. See the handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 03:12:28 -05:00
jakman85andClaude Opus 5 713b0c3dba fix(renovate): the config was INVALID, so the first manual run errored out
gate / gate (pull_request) Successful in 1m55s
gate / conformance (pull_request) Has been skipped
The operator ran the Renovate workflow manually and it failed. The cause is not
the workflow: renovate.json was never a valid Renovate config. It carried five
pseudo-comment keys (_comment, _kubernetes_comment, _unmanageable_comment,
_ignorePaths_comment, _docker_comment) because JSON has no comments, and
Renovate REJECTS unknown top-level options. Proven with the official validator:

  ERROR: Found errors in configuration
    Invalid configuration option: _comment
    Invalid configuration option: _docker_comment
    Invalid configuration option: _ignorePaths_comment
    Invalid configuration option: _kubernetes_comment
    Invalid configuration option: _unmanageable_comment

This is the SECOND independent reason Renovate never worked, on top of PR #77's
cron firing outside renovate.json's own schedule window. Either alone was enough
to produce "nothing ever happens"; #73 shipped both.

Converted renovate.json -> renovate.json5. JSON5 permits real // comments and
Renovate discovers renovate.json5 natively; nothing in .gitea/ or ci/ pins the
old filename and no RENOVATE_CONFIG_FILE is set. Every word of the original
rationale is preserved as positioned comments (74 comment lines) rather than
flattened into a description string.

Verified the conversion dropped nothing: 18 non-comment keys before and after,
no missing/extra keys, no changed values, docker: {enabled: true} still present.
The validator now returns "Config validated successfully".

DISPROVED while here: `enabledManagers` containing bare `regex` was suspected of
needing to be `custom.regex` (which would silently disable every inline
`# renovate:` marker). The validator accepts `regex`; unfounded, unchanged.

Renovate has still never completed a run -- this fixes one of two blockers and
nothing in the gate catches an invalid config. See the handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 03:10:25 -05:00
45 changed files with 2922 additions and 460 deletions

No files matched your search

+45
View File
@@ -115,14 +115,59 @@ CI. That is a real coverage gap, not a phase.
| variable | `REQUIRE_CLUSTER_TESTS` | `'true'` (Phase 1+) makes unverified cluster-needing touched reqs a hard block |
| secret | `CONFORMANCE_TRIGGER_KUBECONFIG` | narrow creds to create/await the in-cluster Chainsaw Job (not cluster-admin) |
## How images are built
This repo's three images — `toolbox` (`infra/devcontainer/`), `agent-runner` (`infra/agent/`,
built `FROM` the toolbox) and `conformance-runner` (`conformance/incluster/`) — are built,
verified and published by [`images.yaml`](images.yaml), never from a workstation
(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh).
**The builder is rootless BuildKit** ([`platform/buildkit/`](../../platform/buildkit/README.md)),
reached over mTLS with a pinned `buildx` — no job here touches a Docker daemon, privileged or
otherwise (operator decision on PR #81). The client certificate reaches jobs as the Actions
secrets `BUILDKIT_CA`, `BUILDKIT_CLIENT_CERT`, `BUILDKIT_CLIENT_KEY`.
**Verification is a build target.** Each Containerfile ends in a `verify` stage that runs every
in-image assertion at build time as the non-root uid with `RUN --network=none` (tool presence and
pinned versions, `PM_SELFTEST=1 pm-dispatch`, the pm-worker probe, the `conformance-clone`
refusals, no writable path outside the workspace). The pipeline builds `<image>-verify` before it
builds, let alone pushes, the image. Build args come from each image's own
`build-image.sh --print-build-args`, which is also the local podman path — so the recipe and the
proof exist once.
| Trigger | What happens |
|---|---|
| pull request touching an image's inputs | build + verify only. Nothing is pushed and **no registry credential is referenced** |
| push to `main` touching an image's inputs | build + verify + publish to `gitea.home.keithjackson.dev/home-network/<image>` as `sha-<commit>` (immutable) and `main` (moving); the **registry-returned** digest is in the job summary |
| `workflow_dispatch` | the same as a push when run on `main` (all images by default); verify-only elsewhere |
Only images whose inputs changed are built. The publish job **never writes to git**: Renovate
follows each image's `main` tag through a `# renovate-digest:` marker next to every digest
reference (`platform/pm-agent/`, `infra/agent/versions.env`, `conformance/versions.env`) and
opens the bump PR, which goes through the gate like anything else. A PR opened with the Actions
token would not trigger the required gate, which is why it is Renovate's job and not this one's.
Credential: the repository secret **`REGISTRY_PUSH_TOKEN`** (scope `write:package`, owned by an
account with package write on the org). The automatic Actions token cannot push in Gitea 1.24.3
— the reasoning, with source citations, is in the header of `images.yaml`. Retention:
[`infra/gitea/registry-retention.sh`](../../infra/gitea/registry-retention.sh), an operator
action, keeps every digest git pins plus the newest tags.
Cost: the job shares the single act_runner slot with the gate, but the build itself runs in the
`buildkitd` pod, not in DinD, so the runner's disk and memory are untouched. Until
`platform/buildkit` is deployed and its client certificate copied
(`infra/gitea/buildkit-client-secret.sh`), the job fails loudly at its builder preflight.
## Files
```
.gitea/workflows/
merge-gate.yaml the pipeline (lint → l0 → l1 → conformance → decision)
images.yaml build/verify/publish this repo's container images (see above)
scripts/
install-tools.sh installs the pinned tools per stage (from ci/versions.env)
decide.sh the R-TEST-03 merge decision over job results
images.sh the image pipeline's driver (what to build, base, push, digest)
README.md this file
```
+185
View File
@@ -0,0 +1,185 @@
# Container images — built, verified and published by CI, never from a workstation.
#
# ═══════════════════════════════════════════════════════════════════════════════════════════════
# WHY THIS FILE EXISTS (operator rule, 2026-10-29)
# ═══════════════════════════════════════════════════════════════════════════════════════════════
# "Container image creation, hosting and lifecycle must be part of CI/CD and the codebase, never a
# one-off build/push from a workstation." Before this workflow, three images existed only as
# workstation builds and NOTHING had ever been pushed to the Gitea registry: the Home-Network org's
# package list was empty, platform/pm-agent/ pinned an all-zero digest and sat in ImagePullBackOff,
# and conformance/versions.env carried an empty digest. Both build scripts said "publishing is an
# operator action" — a step with no log, no recorded inputs and no verification anyone could read.
#
# WHAT IT DOES — the logic is in scripts/images.sh; this file only wires triggers and secrets.
# pull_request -> build + VERIFY only. No push, and no registry credential is referenced.
# push to main -> build + verify + PUBLISH to gitea.home.keithjackson.dev/home-network/<image>
# dispatch -> same as a push when run on main; verify-only on any other ref.
# Images: toolbox (infra/devcontainer), agent-runner (infra/agent, FROM toolbox) and
# conformance-runner (conformance/incluster). Only images whose inputs changed are built.
#
# THE DIGEST LIFECYCLE — this workflow NEVER writes to git.
# 1. publish pushes `sha-<commit>` (immutable) and `main` (moving), and prints the digest the
# REGISTRY returns in the job summary.
# 2. Renovate (renovate.yaml) follows the `main` tag of each image through the
# `# renovate-digest:` markers next to every digest reference, and opens the bump PR.
# 3. That PR goes through the gate like any other change, and Flux deploys it.
# The bump PR is Renovate's and not this job's because a PR opened with the Actions token does not
# trigger workflows, so it could never pass the REQUIRED gate check; Renovate's token is a real
# user's.
#
# THE PUSH CREDENTIAL — `REGISTRY_PUSH_TOKEN`, a repository Actions secret, referenced BY NAME.
# The automatic Actions token CANNOT be used. Gitea 1.24.3 source, verified:
# * services/auth/basic.go (Basic.Verify) turns a task token into the pseudo-user
# `gitea-actions` (ActionsUserID -2) and sets NO token scope;
# * routers/api/packages/container/container.go (Authenticate, the /v2/token exchange every
# docker client uses) then requires read:package / write:package / all scope and answers 401;
# * services/context/package.go (determineAccessMode) has `// TODO: ActionUser permission check`
# and no branch for that user, so even past login it would get at most READ on a public org.
# So: a personal access token with scope `write:package`, minted by an account that has package
# WRITE on the Home-Network org (org owner, or a team with the Packages unit at write). A dedicated
# bot account in a packages-only team is the least-privilege choice. The username sent with it is
# not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER`
# is optional and only labels the login.
#
# THE BUILDER — rootless BuildKit (operator decision on PR #81, 2026-10-29).
# The act runner's job containers get NO Docker daemon (act_runner 0.2.11 forces docker_host "-"
# for a tcp:// DinD, and gitea/act strips every bind when valid_volumes is empty — measured on
# this PR's first run), and handing them the privileged DinD daemon would be root on the node for
# anyone who can push a branch. So images are built by platform/buildkit/: a ROOTLESS buildkitd
# (not privileged; uid 1000 in a user namespace), reached over mTLS at
# buildkitd.buildkit.svc.cluster.local:1234 with a pinned `buildx` (remote driver). Its client
# certificate reaches jobs as three Actions secrets — BUILDKIT_CA, BUILDKIT_CLIENT_CERT,
# BUILDKIT_CLIENT_KEY — copied from cert-manager by infra/gitea/buildkit-client-secret.sh.
# Pull requests get the client certificate (they must build) but never a registry credential.
#
# UNTIL platform/buildkit IS DEPLOYED (Flux applies it only after merge) THIS JOB FAILS, LOUDLY,
# at its builder preflight. A skipped Gitea job reports `success`, so it is red on purpose.
#
# VERIFICATION is each Containerfile's `verify` stage, built before anything is pushed.
#
name: images
on:
pull_request:
branches: [main]
paths:
- infra/devcontainer/Containerfile
- infra/devcontainer/versions.env
- infra/devcontainer/build-image.sh
- infra/agent/**
- conformance/incluster/Containerfile
- conformance/incluster/conformance-clone
- conformance/incluster/build-image.sh
- .gitea/workflows/images.yaml
- .gitea/workflows/scripts/images.sh
- platform/buildkit/versions.env
push:
branches: [main]
paths:
- infra/devcontainer/Containerfile
- infra/devcontainer/versions.env
- infra/devcontainer/build-image.sh
- infra/agent/**
- conformance/incluster/Containerfile
- conformance/incluster/conformance-clone
- conformance/incluster/build-image.sh
- .gitea/workflows/images.yaml
- .gitea/workflows/scripts/images.sh
- platform/buildkit/versions.env
workflow_dispatch:
inputs:
images:
description: "Which images: 'all', or a space-separated subset of: toolbox agent-runner conformance-runner"
required: false
default: "all"
concurrency:
# Never cancel a publish half-way: a pushed `sha-` tag with no matching `main` move is
# recoverable (a rerun skips what is already published) but pointless to provoke.
group: images-${{ github.ref }}
cancel-in-progress: false
jobs:
images:
runs-on: ubuntu-latest
# Measured on the PR that introduced this file — see its decisions entry. Generous because a
# cold DinD store downloads the Fedora base and every pinned tool on each run.
timeout-minutes: 75
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0 # the per-image change detection diffs against the base commit
# ── pull requests: build + verify, NO credential ───────────────────────────────────────
- name: Build and verify (pull request — nothing is pushed)
if: github.event_name == 'pull_request'
env:
MODE: verify
IMAGES: auto
PR_BASE_REF: ${{ github.base_ref }}
# The BUILDER's client certificate (platform/buildkit). Not a registry credential:
# a pull request can build, never push.
BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
run: |
set -euo pipefail
BASE_SHA="$(git merge-base "origin/${PR_BASE_REF}" HEAD)"
export BASE_SHA
./.gitea/workflows/scripts/images.sh
# ── non-main dispatch: verify only ──────────────────────────────────────────────────────
- name: Build and verify (dispatch off main — nothing is pushed)
if: github.event_name == 'workflow_dispatch' && github.ref != 'refs/heads/main'
env:
MODE: verify
IMAGES: ${{ inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images }}
BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
run: ./.gitea/workflows/scripts/images.sh
# ── main: publish ───────────────────────────────────────────────────────────────────────
- name: Preflight - the push token must exist, and its absence must be LOUD
if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
env:
REGISTRY_PUSH_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
run: |
set -euo pipefail
if [ -z "${REGISTRY_PUSH_TOKEN:-}" ]; then
echo "::error::REGISTRY_PUSH_TOKEN is not set on this repository." >&2
cat >&2 <<'MSG'
Images cannot be published without a registry credential, and this step fails rather
than skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would
make "nothing was published" look exactly like "published". That is how this repo went
a month with images that existed only on one workstation.
WHY NOT THE BUILT-IN ACTIONS TOKEN: Gitea 1.24.3 gives it no package scope at the
registry's /v2/token exchange (routers/api/packages/container/container.go,
Authenticate) — see the header of .gitea/workflows/images.yaml.
TO FIX: as an account with package WRITE on the Home-Network org (ideally a dedicated
bot in a packages-only team), create an access token with scope `write:package` and
nothing else, then add it to this repository as the Actions secret
REGISTRY_PUSH_TOKEN. Optionally set the Actions variable REGISTRY_PUSH_USER to that
account's name. Do not commit the token anywhere. Then re-run this workflow
(workflow_dispatch on main, images: all).
MSG
exit 1
fi
echo "REGISTRY_PUSH_TOKEN is present (value not printed)."
- name: Build, verify and publish (main)
if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
env:
MODE: publish
IMAGES: ${{ github.event_name == 'workflow_dispatch' && (inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images) || 'auto' }}
BASE_SHA: ${{ github.event.before }}
REGISTRY_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
REGISTRY_USER: ${{ vars.REGISTRY_PUSH_USER }}
BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
run: ./.gitea/workflows/scripts/images.sh
+44 -11
View File
@@ -60,8 +60,9 @@
# entry's "What I did NOT verify" — that gap is stated, not papered over.
#
# OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE:
# `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository` and
# `read:user`. Set it as an Actions secret on this repository. Nothing in this repo mints,
# `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository`,
# `read:user` and (since 2026-10-29) `read:package`. Set it as an Actions secret on this
# repository. Nothing in this repo mints,
# stores or references its value; it is consumed by name only.
# Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a
# visible red is the correct state for "automation configured but not credentialed".
@@ -74,12 +75,26 @@ name: renovate
on:
schedule:
# 03:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` in
# America/New_York, and that inner schedule is the authoritative one — Renovate re-checks it and
# will decline to open PRs outside it. The two are deliberately both present: this cron decides
# when the process RUNS, renovate.json decides when it may ACT. Keeping the cron slightly ahead
# of the repo window means a run is always in flight when the window opens.
- cron: "15 3 * * *"
# 08:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` with
# `"timezone": "America/New_York"`, and THAT inner schedule is authoritative — Renovate re-checks
# it and DECLINES TO OPEN PRs outside it. So this cron must fire INSIDE the local window, not
# merely near it.
#
# WAS `15 3 * * *`, WHICH NEVER WOULD HAVE WORKED (fixed 2026-10-18, before the first scheduled
# run). Gitea Actions cron is UTC, so 03:15 UTC is 23:15 the PREVIOUS EVENING in New York --
# 3h45m BEFORE the window opens, not "slightly ahead" of it as the original comment claimed.
# Every nightly run would have started, logged "not within schedule", opened zero PRs and
# exited SUCCESSFULLY. A green tick forever, and no updates ever: the exact "failure mode is a
# green tick" shape this repo keeps getting burned by, and indistinguishable from "nothing
# needed updating".
#
# THE WINDOW IN UTC MOVES WITH US DST, and a cron cannot follow it:
# EDT (UTC-4, Mar-Nov): local 03:00-06:00 == 07:00-10:00 UTC
# EST (UTC-5, Nov-Mar): local 03:00-06:00 == 08:00-11:00 UTC
# 08:15 UTC is the only quarter-hour comfortably inside BOTH (04:15 EDT / 03:15 EST), so the
# schedule keeps working across both switchovers without an edit. Do not "tidy" this back to an
# early-morning-UTC value without redoing this arithmetic.
- cron: "15 8 * * *"
workflow_dispatch:
inputs:
dry_run:
@@ -125,8 +140,9 @@ jobs:
seven files claimed "Renovate maintains this pin".
TO FIX: create a Gitea access token for the bot account with scopes
`write:repository` and `read:user`, then add it to this repository as the Actions
secret RENOVATE_TOKEN. Do not commit it anywhere.
`write:repository`, `read:user` and `read:package` (the last one lets Renovate read
this repo's own images in the Gitea registry), then add it to this repository as the
Actions secret RENOVATE_TOKEN. Do not commit it anywhere.
MSG
exit 1
fi
@@ -138,8 +154,17 @@ jobs:
# Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers
# includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`.
# This is the one dependency in the repo whose updater is its own updater.
configurationFile: renovate.json
# MUST match the file's real name. #79 renamed it to renovate.json5 and left this line
# saying `renovate.json`; every scheduled run from 2026-10-18 then died in ~1s with
# ENOENT before Renovate started, and nothing in the gate noticed. ci/lint.sh now
# asserts that every `configurationFile:` here names a file that exists.
configurationFile: renovate.json5
token: ${{ secrets.RENOVATE_TOKEN }}
# PINNED 2026-10-29. Unset, this action version runs Renovate `39` (action.yml
# `renovate-version` default at v41.0.9), i.e. 39.264.0, and renovate.json5 uses
# `managerFilePatterns` (>= 40.2.0): under 39.x validation fails with five errors and
# the run stops at CONFIG_VALIDATION. #79 validated against `latest`, not this.
renovate-version: 44.167.0
env:
# Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
# header said were "set at runtime by the operator's renovate runner" — this is that
@@ -156,6 +181,14 @@ jobs:
RENOVATE_REQUIRE_CONFIG: "required"
RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }}
LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }}
# ADDED 2026-10-29: registry read access for this repo's OWN images (toolbox,
# agent-runner, conformance-runner — published by .gitea/workflows/images.yaml and
# followed through `# renovate-digest:` markers). The Gitea registry answers 401 to
# anonymous reads, and Renovate scopes the platform token to hostType `gitea`, which a
# docker lookup never matches — so without this rule every digest lookup fails. Same
# token; it therefore needs scope `read:package` in addition to the ones listed in the
# preflight above. Gitea ignores the username for token auth (services/auth/basic.go).
RENOVATE_HOST_RULES: '[{"hostType":"docker","matchHost":"gitea.home.keithjackson.dev","username":"renovate","password":"${{ secrets.RENOVATE_TOKEN }}"}]'
- name: Summary
if: always()
+373
View File
@@ -0,0 +1,373 @@
#!/usr/bin/env bash
###############################################################################
# .gitea/workflows/scripts/images.sh — the image pipeline's driver (images.yaml calls this).
#
# Builds, verifies and (on main) PUBLISHES the three images this repo ships:
#
# toolbox infra/devcontainer/ the pinned dev/gate toolchain
# agent-runner infra/agent/ FROM toolbox; the PM dispatcher + worker
# conformance-runner conformance/incluster/ the in-cluster L2/L3 Chainsaw runner
#
# THE BUILDER (operator decision on PR #81): a ROOTLESS BuildKit daemon, platform/buildkit/,
# reached over mTLS. No CI job may reach a privileged daemon, so there is no `docker build` and no
# `docker run` here: this script drives buildkitd with a pinned `buildx` (remote driver) and
# nothing else. The client certificate arrives as Actions secrets (BUILDKIT_CA,
# BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY; infra/gitea/buildkit-client-secret.sh puts them
# there). If the daemon is unreachable this FAILS LOUDLY — it never skips.
#
# VERIFICATION IS A BUILD TARGET. Each Containerfile ends in a `verify` stage that runs every
# in-image assertion at build time (as the non-root uid, with RUN --network=none); this script
# builds `<image>-verify` and only then the image. Build args come from each image's own
# `build-image.sh --print-build-args`, the same list the local podman path uses.
#
# MODES (env MODE):
# verify build the verify targets, never push, hold no registry credential. Pull requests.
# publish verify, then build + push to ${REGISTRY}/${NAMESPACE}/<image> with tags
# `sha-<12 hex>` (immutable: never re-pushed) and `main` (moving; Renovate follows
# it). The digest recorded is the one the REGISTRY returns, read back after the push.
#
# WHAT GETS BUILT (env IMAGES): `auto` (default) diffs BASE_SHA..HEAD_SHA against each image's
# inputs; otherwise a space-separated list. A change to the pipeline itself (images.yaml, this
# file, platform/buildkit/versions.env) builds everything in verify mode, and nothing extra in
# publish mode: re-publishing unchanged inputs would only mint new digests and churn Renovate PRs.
#
# THE agent-runner BASE:
# * verify mode -> FROM the toolbox built in this run, linked with a bake `target:` context —
# a pull request holds no registry credential and that toolbox exists in no registry. If the
# toolbox's inputs did not change it is built anyway, as the base.
# * publish mode, toolbox published in this run -> FROM the digest the registry just returned.
# * publish mode otherwise -> AGENT_BASE_IMAGE from infra/agent/versions.env (the published
# toolbox digest Renovate keeps current). If that is still the all-zero placeholder the
# toolbox has never been published, so it is built and published first (the bootstrap).
#
# LOOP-FREEDOM. Renovate turns each publish into a digest-bump PR. Those bumps land in
# platform/pm-agent/ (no image input), conformance/versions.env (no image input) and
# infra/agent/versions.env (agent-runner input -> ONE more agent-runner build, whose own bump
# lands in platform/pm-agent/ and stops). The worker Job template is deliberately NOT tracked:
# it is baked INTO agent-runner, so bumping it would rebuild agent-runner, mint a new digest,
# bump it again, forever. pm-dispatch overwrites that field at runtime anyway.
###############################################################################
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
cd "${REPO_ROOT}"
# shellcheck disable=SC1091
BK_HOST_OVERRIDE="${BUILDKIT_HOST:-}"; BK_NAME_OVERRIDE="${BUILDKIT_SERVERNAME:-}"
. ./platform/buildkit/versions.env
BUILDKIT_HOST="${BK_HOST_OVERRIDE:-${BUILDKIT_HOST}}"
BUILDKIT_SERVERNAME="${BK_NAME_OVERRIDE:-${BUILDKIT_SERVERNAME}}"
MODE="${MODE:-verify}"
IMAGES="${IMAGES:-auto}"
REGISTRY="${REGISTRY:-gitea.home.keithjackson.dev}"
NAMESPACE="${NAMESPACE:-home-network}"
HEAD_SHA="${HEAD_SHA:-$(git rev-parse HEAD)}"
BASE_SHA="${BASE_SHA:-}"
SUMMARY="${GITHUB_STEP_SUMMARY:-/dev/null}"
ZERO_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000"
ALL_IMAGES=(toolbox agent-runner conformance-runner)
SHA_TAG="sha-${HEAD_SHA:0:12}"
MOVING_TAG="main"
log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
info() { printf ' %s\n' "$*"; }
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; printf -- '- **ERROR**: %s\n' "$*" >>"${SUMMARY}"; exit 1; }
yn() { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; }
case "${MODE}" in verify|publish) ;; *) die "MODE must be verify or publish, got '${MODE}'" ;; esac
for t in git jq curl sha256sum; do command -v "$t" >/dev/null || die "the job image lacks '$t'"; done
# --- per-image facts ------------------------------------------------------------------------
# Inputs: a change under any of these rebuilds that image. conformance-runner takes its pins from
# infra/devcontainer/versions.env. conformance/versions.env is NOT an input (it only RECORDS the
# published digest), and must not become one: Renovate writes it after every publish -> a loop.
inputs() {
case "$1" in
toolbox) echo "infra/devcontainer/Containerfile infra/devcontainer/versions.env infra/devcontainer/build-image.sh" ;;
agent-runner) echo "infra/agent/" ;;
conformance-runner) echo "conformance/incluster/Containerfile conformance/incluster/conformance-clone conformance/incluster/build-image.sh infra/devcontainer/versions.env" ;;
esac
}
context_of() { case "$1" in toolbox) echo infra/devcontainer ;; agent-runner) echo infra/agent ;; conformance-runner) echo conformance/incluster ;; esac; }
PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh platform/buildkit/versions.env"
declare -A WANT=() BUILT=() DIGEST=() SECS=() NOTE=()
changed_any() { # <paths...> -> 0 if BASE_SHA..HEAD_SHA touches any
[ -n "$(git diff --name-only "${BASE_SHA}" "${HEAD_SHA}" -- "$@")" ]
}
select_images() {
local i
if [ "${IMAGES}" != "auto" ]; then
for i in ${IMAGES}; do
case " ${ALL_IMAGES[*]} " in *" ${i} "*) WANT[$i]=1; NOTE[$i]="requested" ;; *) die "unknown image '${i}'" ;; esac
done
return
fi
if [ -z "${BASE_SHA}" ] || [[ "${BASE_SHA}" =~ ^0+$ ]] || ! git cat-file -e "${BASE_SHA}^{commit}" 2>/dev/null; then
info "no usable diff base ('${BASE_SHA}'): building everything"
for i in "${ALL_IMAGES[@]}"; do WANT[$i]=1; NOTE[$i]="no diff base"; done
return
fi
info "diff base ${BASE_SHA} .. ${HEAD_SHA}"
for i in "${ALL_IMAGES[@]}"; do
# shellcheck disable=SC2046
if changed_any $(inputs "$i"); then WANT[$i]=1; NOTE[$i]="inputs changed"; fi
done
# shellcheck disable=SC2086
if [ "${MODE}" = verify ] && changed_any ${PIPELINE_FILES}; then
for i in "${ALL_IMAGES[@]}"; do [ -n "${WANT[$i]:-}" ] || { WANT[$i]=1; NOTE[$i]="pipeline changed"; }; done
fi
}
# --- the builder: pinned buildx + mTLS to rootless buildkitd ---------------------------------
WORK="$(mktemp -d)"
trap 'rm -rf "${WORK}"' EXIT # client key + registry auth live only here, only this long
export BUILDX_CONFIG="${WORK}/buildx" DOCKER_CONFIG="${WORK}/docker"
# Single-manifest images: the admission policies, Renovate and registry-retention.sh all reason
# about one manifest per tag. (Provenance attestations would turn every push into an index.)
export BUILDX_NO_DEFAULT_ATTESTATIONS=1
mkdir -p "${BUILDX_CONFIG}" "${DOCKER_CONFIG}" "${WORK}/certs"
echo '{}' > "${DOCKER_CONFIG}/config.json"
setup_builder() {
log "Builder: rootless buildkitd at ${BUILDKIT_HOST} (mTLS)"
BUILDX="${BUILDX_BIN:-}"
if [ -z "${BUILDX}" ]; then
BUILDX="${WORK}/buildx-bin"
curl -fsSL --retry 3 -o "${BUILDX}" \
"https://github.com/docker/buildx/releases/download/v${BUILDX_VERSION}/buildx-v${BUILDX_VERSION}.linux-amd64" \
|| die "could not download buildx v${BUILDX_VERSION}"
echo "${BUILDX_SHA256} ${BUILDX}" | sha256sum -c --quiet - \
|| die "buildx v${BUILDX_VERSION} does not match the sha256 pinned in platform/buildkit/versions.env — refusing to run it"
chmod +x "${BUILDX}"
fi
info "client: $("${BUILDX}" version)"
local missing=()
[ -n "${BUILDKIT_CA:-}" ] || missing+=(BUILDKIT_CA)
[ -n "${BUILDKIT_CLIENT_CERT:-}" ] || missing+=(BUILDKIT_CLIENT_CERT)
[ -n "${BUILDKIT_CLIENT_KEY:-}" ] || missing+=(BUILDKIT_CLIENT_KEY)
[ "${#missing[@]}" -eq 0 ] || die "the builder's client certificate is missing: ${missing[*]} not set.
These Actions secrets are copied from the cert-manager Secret buildkit/buildkit-client-ci by
infra/gitea/buildkit-client-secret.sh (an operator action, after platform/buildkit is
deployed). Without them nothing can be built — this fails rather than skipping ON PURPOSE."
umask 077
printf '%s\n' "${BUILDKIT_CA}" > "${WORK}/certs/ca.pem"
printf '%s\n' "${BUILDKIT_CLIENT_CERT}" > "${WORK}/certs/cert.pem"
printf '%s\n' "${BUILDKIT_CLIENT_KEY}" > "${WORK}/certs/key.pem"
umask 022
"${BUILDX}" create --name ci-remote --driver remote \
--driver-opt "cacert=${WORK}/certs/ca.pem,cert=${WORK}/certs/cert.pem,key=${WORK}/certs/key.pem,servername=${BUILDKIT_SERVERNAME}" \
"${BUILDKIT_HOST}" >/dev/null
if ! out="$(timeout 90 "${BUILDX}" inspect --bootstrap --builder ci-remote 2>&1)"; then
printf '%s\n' "${out}" >&2
die "buildkitd is UNREACHABLE at ${BUILDKIT_HOST} — nothing can be built, and this job fails
rather than skipping. In order of likelihood:
1. platform/buildkit is not deployed yet. Flux applies it only after it merges to main
(Kustomization 'buildkit'); a pull request that ADDS it cannot reach it.
2. the BUILDKIT_* secrets are stale (the client cert was renewed) or from another CA:
re-run infra/gitea/buildkit-client-secret.sh.
3. the NetworkPolicy (platform/buildkit/networkpolicy.yaml) does not admit this runner."
fi
printf '%s\n' "${out}" | grep -E '^(Name|Driver|Status|BuildKit version|Platforms):' | sed 's/^/ /' || true
}
# KEY=VALUE lines from an image's build-image.sh --print-build-args -> a JSON object.
build_args_json() {
local ctx; ctx="$(context_of "$1")"
"./${ctx}/build-image.sh" --print-build-args | jq -Rn '[inputs | select(length>0) | capture("^(?<k>[^=]+)=(?<v>.*)$")] | map({(.k): .v}) | add'
}
# bake_run <bakefile> <targets...>
bake_run() {
local f="$1"; shift
"${BUILDX}" bake --builder ci-remote --progress plain -f "$f" \
--metadata-file "${WORK}/meta.json" "$@"
}
# Write a bake file with a verify target and a ship target for one image.
# $1 image, $2 agent base mode: "" | "link" | <image ref>
write_bake() {
local img="$1" base="${2:-}" ctx args f
ctx="$(context_of "$img")"
args="$(build_args_json "$img")"
f="${WORK}/bake-${img}.json"
local contexts='{}'
if [ "${img}" = agent-runner ]; then
if [ "${base}" = link ]; then
# FROM ${AGENT_BASE_IMAGE} resolves to the named context below, i.e. the toolbox target
# of this same bake file — built here, never pushed anywhere.
args="$(jq -c '. + {AGENT_BASE_IMAGE: "toolbox-in-this-run"}' <<<"${args}")"
contexts='{"toolbox-in-this-run": "target:toolbox"}'
else
args="$(jq -c --arg b "${base}" '. + {AGENT_BASE_IMAGE: $b}' <<<"${args}")"
fi
fi
jq -n --arg img "$img" --arg ctx "$ctx" --argjson args "$args" --argjson contexts "$contexts" \
--arg tbargs "$( [ "${base}" = link ] && build_args_json toolbox || echo '{}')" \
--arg rev "${HEAD_SHA}" --arg repo "${REGISTRY}/${NAMESPACE}/${img}" \
--arg sha "${SHA_TAG}" --arg moving "${MOVING_TAG}" --arg mode "${MODE}" '
{target: (
{
($img + "-verify"): {context: $ctx, dockerfile: "Containerfile", target: "verify",
args: $args, contexts: $contexts, output: ["type=cacheonly"]},
($img): ({context: $ctx, dockerfile: "Containerfile", target: $img, args: $args,
contexts: $contexts,
labels: {"org.opencontainers.image.revision": $rev}}
+ (if $mode == "publish"
then {tags: [($repo + ":" + $sha), ($repo + ":" + $moving)], output: ["type=image,push=true"]}
else {output: ["type=cacheonly"]} end))
}
+ (if ($contexts | has("toolbox-in-this-run"))
then {toolbox: {context: "infra/devcontainer", dockerfile: "Containerfile", target: "toolbox",
args: ($tbargs | fromjson), output: ["type=cacheonly"]}}
else {} end)
)}' > "$f"
echo "$f"
}
# --- registry helpers (publish mode only) ----------------------------------------------------
REG_TOKEN=""
registry_token() {
[ -n "${REG_TOKEN}" ] && return 0
REG_TOKEN="$(curl -fsS --config <(printf 'user = "%s:%s"\n' "${REGISTRY_USER}" "${REGISTRY_TOKEN}") \
"https://${REGISTRY}/v2/token?service=container_registry&scope=*" | jq -r '.token // empty')" \
|| die "could not obtain a registry bearer token from https://${REGISTRY}/v2/token"
[ -n "${REG_TOKEN}" ] || die "registry returned no bearer token"
}
# Prints the manifest digest the REGISTRY serves for <image>:<tag>; nothing if it is absent (404).
# Any other answer is fatal: "cannot tell" must never be read as "not published", or the
# immutable-tag guard would be bypassed by an auth hiccup.
registry_digest() {
registry_token
local hdrs code digest
hdrs="$(curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "${REG_TOKEN}") \
-H 'Accept: application/vnd.oci.image.manifest.v1+json' \
-H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
-H 'Accept: application/vnd.oci.image.index.v1+json' \
-H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
"https://${REGISTRY}/v2/${NAMESPACE}/$1/manifests/$2" | tr -d '\r')" \
|| die "registry unreachable while reading $1:$2"
code="$(awk 'NR==1{print $2}' <<<"${hdrs}")"
digest="$(awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' <<<"${hdrs}" | tail -1)"
case "${code}" in
200) [[ "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]] || die "registry answered 200 for $1:$2 with no usable digest"
echo "${digest}" ;;
404) ;;
*) die "registry answered HTTP ${code} for $1:$2 — cannot tell whether it exists" ;;
esac
}
# build_one <image> [agent base] — verify, then (publish) build + push + read back.
build_one() {
local img="$1" base="${2:-}" t0 f pushed got existing
t0=$(date +%s)
f="$(write_bake "$img" "$base")"
log "VERIFY ${img} (Containerfile target: verify, on rootless BuildKit)"
bake_run "$f" "${img}-verify"
BUILT[$img]=1
if [ "${MODE}" = publish ]; then
existing="$(registry_digest "${img}" "${SHA_TAG}")"
# Immutable tag: never overwrite. A rebuild is not byte-reproducible, so re-pushing would
# silently change what `sha-<commit>` means after something may already have pinned it.
[ -z "${existing}" ] || die "${REGISTRY}/${NAMESPACE}/${img}:${SHA_TAG} already exists (${existing}); tags are immutable"
log "BUILD + PUSH ${img} -> ${REGISTRY}/${NAMESPACE}/${img}:{${SHA_TAG},${MOVING_TAG}}"
bake_run "$f" "${img}"
pushed="$(jq -r --arg t "$img" '.[$t]["containerimage.digest"] // empty' "${WORK}/meta.json")"
# The AUTHORITATIVE digest is what the registry serves, read back — not what the client
# reported. They should agree; if they do not, nothing downstream may trust either.
got="$(registry_digest "${img}" "${SHA_TAG}")"
[ -n "${got}" ] || die "pushed ${img}:${SHA_TAG} but the registry does not serve it"
[ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] || die "${img}: client reported ${pushed} but the registry serves ${got}"
[ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \
|| die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})"
DIGEST[$img]="${got}"
info "${img}: registry digest ${got}"
fi
SECS[$img]=$(( $(date +%s) - t0 ))
info "${img}: done in ${SECS[$img]}s"
}
# --- main ---------------------------------------------------------------------------------
log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}"
select_images
if [ "${MODE}" = publish ]; then
: "${REGISTRY_TOKEN:?REGISTRY_TOKEN is required in publish mode}"
REGISTRY_USER="${REGISTRY_USER:-ci-image-publisher}"
# buildx (and through it buildkitd, which does the push) reads registry auth from this file.
jq -n --arg r "${REGISTRY}" --arg a "$(printf '%s:%s' "${REGISTRY_USER}" "${REGISTRY_TOKEN}" | base64 -w0)" \
'{auths: {($r): {auth: $a}}}' > "${DOCKER_CONFIG}/config.json"
chmod 600 "${DOCKER_CONFIG}/config.json"
registry_token
# Publish idempotence: an image already published at THIS commit is not rebuilt. That is what
# makes a re-run after a half-finished publish safe — the immutable tag is never pushed twice.
for i in "${ALL_IMAGES[@]}"; do
[ -n "${WANT[$i]:-}" ] || continue
d="$(registry_digest "${i}" "${SHA_TAG}")"
if [ -n "${d}" ]; then
info "${i}: already published at ${SHA_TAG} (${d}); not rebuilding"
unset "WANT[$i]"; DIGEST[$i]="${d}"; NOTE[$i]="already published at this commit"
fi
done
fi
# Decide the agent-runner base (see header).
AGENT_BASE=""
if [ -n "${WANT[agent-runner]:-}" ]; then
PINNED_BASE="$(. ./infra/agent/versions.env; echo "${AGENT_BASE_IMAGE}")"
if [ "${MODE}" = verify ]; then
AGENT_BASE=link
[ -n "${WANT[toolbox]:-}" ] || { WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)"; }
elif [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then
case "${PINNED_BASE}" in
*@"${ZERO_DIGEST}") WANT[toolbox]=1; NOTE[toolbox]="never published; required as agent-runner's base" ;;
*@sha256:*) AGENT_BASE="${PINNED_BASE}" ;;
*) die "infra/agent/versions.env AGENT_BASE_IMAGE='${PINNED_BASE}' is not digest-pinned; CI will not build on a floating base" ;;
esac
fi
fi
for i in "${ALL_IMAGES[@]}"; do info "plan: ${i}: $(yn "${WANT[$i]:-}" BUILD skip) (${NOTE[$i]:-inputs unchanged})"; done
if [ -z "${WANT[toolbox]:-}${WANT[agent-runner]:-}${WANT[conformance-runner]:-}" ]; then
info "nothing to build"
else
setup_builder
fi
if [ -n "${WANT[toolbox]:-}" ]; then build_one toolbox; fi
if [ -n "${WANT[agent-runner]:-}" ]; then
if [ -z "${AGENT_BASE}" ]; then
AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}"
fi
info "agent-runner base: $( [ "${AGENT_BASE}" = link ] && echo "the toolbox built in this run (bake target: context)" || echo "${AGENT_BASE}")"
build_one agent-runner "${AGENT_BASE}"
fi
if [ -n "${WANT[conformance-runner]:-}" ]; then build_one conformance-runner; fi
# --- summary ------------------------------------------------------------------------------
{
echo "## Images (${MODE})"
echo
echo "commit \`${HEAD_SHA}\` — builder: rootless BuildKit \`${BUILDKIT_HOST}\` (mTLS)"
echo
echo "| image | verify target built | seconds | published digest (registry-returned) | note |"
echo "|---|---|---|---|---|"
for i in "${ALL_IMAGES[@]}"; do
d="${DIGEST[$i]:-}"
echo "| \`${i}\` | $(yn "${BUILT[$i]:-}" yes no) | ${SECS[$i]:--} | $(yn "${d}" "\`${REGISTRY}/${NAMESPACE}/${i}@${d}\`" -) | ${NOTE[$i]:-inputs unchanged} |"
done
echo
if [ "${MODE}" = publish ]; then
echo "Tags pushed: \`${SHA_TAG}\` (immutable) and \`${MOVING_TAG}\` (moving)."
echo "Nothing here writes to git. Renovate opens the digest-bump PR on its next run."
else
echo "Verify mode: nothing was pushed and no registry credential was present."
fi
} >>"${SUMMARY}"
log "done"
+15
View File
@@ -27,3 +27,18 @@ fi
yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${files[@]}"
ok "yaml/lint passed (${#files[@]} files)"
# A workflow's `configurationFile:` must name a file that exists. Renovate's did not after #79
# renamed renovate.json -> renovate.json5, and every scheduled run failed with ENOENT for 11 days
# while the gate stayed green. Valid YAML, so yamllint cannot see it.
missing=0
while IFS=: read -r wf _ value; do
value="${value#*configurationFile:}"
value="$(echo "$value" | sed -E 's/#.*//; s/^[[:space:]"'\'']+//; s/[[:space:]"'\'']+$//')"
if [ -n "$value" ] && [ ! -e "$value" ]; then
err "${wf}: configurationFile '${value}' does not exist in the repo"
missing=1
fi
done < <(grep -nE '^[[:space:]]*configurationFile:' .gitea/workflows/*.y*ml 2>/dev/null || true)
[ "$missing" -eq 0 ] || exit 1
ok "workflow configurationFile references resolve"
+21
View File
@@ -0,0 +1,21 @@
---
# Flux Kustomization -> ./platform/buildkit — rootless BuildKit, the image builder CI uses
# (.gitea/workflows/images.yaml). Its own Kustomization so a stuck Certificate or a failing
# builder can never stall any other tier (Flux health-checks Certificates under wait:true).
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: buildkit
namespace: flux-system
spec:
interval: 10m
retryInterval: 2m
timeout: 5m
prune: true
wait: true
dependsOn:
- name: platform # cert-manager CRDs (Issuer/Certificate) come from its HelmRelease
sourceRef:
kind: GitRepository
name: flux-system
path: ./platform/buildkit
+1
View File
@@ -98,4 +98,5 @@ resources:
# deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon.
# See clusters/base/kured.yaml and platform/kured/helmrelease.yaml.
- kured.yaml # Flux Kustomization -> ./platform/kured (dependsOn platform)
- buildkit.yaml # Flux Kustomization -> ./platform/buildkit (dependsOn platform)
- cluster-vars.yaml # overlay knob surface (base defaults)
+87 -1
View File
@@ -98,7 +98,10 @@ RUN set -eux; \
# --- runtime -----------------------------------------------------------------
ARG FEDORA_VERSION=44
FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION}
# Stage `conformance-runner` is the image that ships; stage `verify` (end of file) is a TEST stage
# built FROM it that only runs assertions and is never pushed. build-image.sh and the CI
# pipeline build `--target verify` to prove it and `--target conformance-runner` for the image.
FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION} AS conformance-runner
ARG CHAINSAW_VERSION=
ARG KUBECTL_VERSION=
@@ -198,3 +201,86 @@ WORKDIR /workspace
# prepend something the policy cannot see. `bash` is the interactive fallback for a human
# debugging the image locally.
CMD ["/bin/bash"]
# =============================================================================
# verify — a TEST stage, never shipped. Every assertion conformance/incluster/build-image.sh used
# to make with `run --user 65532 --read-only --tmpfs /workspace` now runs here, at build time,
# because CI builds on rootless BuildKit where no container runtime exists.
#
# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE:
# * uid 65532 ............ exact (USER inherited from the conformance-runner stage).
# * no network ........... exact: RUN --network=none, and the first check proves it.
# * --read-only rootfs ... NOT reproducible (a RUN step always has a writable layer).
# Approximated by permission: everything uid 65532 can write must lie
# under /workspace, /tmp or /var/tmp. In the Job only /workspace (and
# /results) are writable — the image points HOME and TMPDIR there — and
# readOnlyRootFilesystem is enforced by
# platform/kyverno/admission/conformance-trigger-job-shape.yaml.
# =============================================================================
FROM conformance-runner AS verify
RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
echo "--- no network (the build-time stand-in for the Job's isolation) ---"
if getent hosts gitea.home.keithjackson.dev >/dev/null 2>&1 || timeout 3 bash -c '</dev/tcp/1.1.1.1/443' 2>/dev/null; then
echo "UNEXPECTED: network reachable in the verify stage"; exit 1
fi
echo "confirmed: no network"
echo "--- id ---"; id
echo "--- HOME ---"; echo "HOME=${HOME}"
echo "--- bash ---"; bash --version | head -1
echo "--- python3 ---"; python3 --version
echo "--- git ---"; git --version
echo "--- kubectl ---"; kubectl version --client
echo "--- chainsaw ---"; chainsaw version
echo "--- baked versions ---"; cat /etc/conformance-runner-versions
echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
if [ -w /usr/local/bin/conformance-clone ]; then
echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
fi
echo "confirmed: not writable by uid $(id -u)"
# Each of these MUST fail. Assert the non-zero exit as well as the message: a check that only
# greps stderr would pass a build where the guard had stopped rejecting.
expect_fail() {
local label="$1"; shift
local out rc=0
out="$(env "$@" 2>&1)" || rc=$?
echo "--- ${label} ---"
printf "%s\n" "${out}" | head -2
if [ "${rc}" -eq 0 ]; then
echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
fi
echo "(rejected, exit ${rc})"
}
expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
conformance-clone --layer l2
expect_fail "conformance-clone rejects a missing SHA" \
conformance-clone
expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
CONFORMANCE_GIT_SHA=main conformance-clone
expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
expect_fail "conformance-clone rejects a HOME off the workspace volume" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
HOME=/etc conformance-clone
expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
expect_fail "conformance-clone refuses a clone target off the workspace volume" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
CONFORMANCE_SRC=/etc/src conformance-clone
echo "--- the root filesystem is not writable by the runner (expected: a failure) ---"
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi
echo "confirmed: / is not writable"
echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---"
stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \
| grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)"
if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf ' %s\n' ${stray} | head -20; exit 1; fi
echo "confirmed: no writable path outside /workspace, /tmp, /var/tmp"
echo "--- HOME is writable (the workspace emptyDir) ---"
touch "${HOME}/.probe" || { echo "UNEXPECTED: ${HOME} not writable"; exit 1; }
echo "confirmed: ${HOME} is writable"
echo "--- uid is non-zero ---"
[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; }
echo "confirmed: running as uid $(id -u), not root"
echo "CONFORMANCE-RUNNER VERIFIED"
VERIFY
+9 -4
View File
@@ -61,10 +61,15 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
> digest-pinned to the conformance-runner image, `command` exactly
> `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
> `env`/`envFrom` were the same shape of hole and are closed in the same policy.
- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
prints the exact `podman push --digestfile …` command for the operator instead. See
`conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
- `build-image.sh` — builds the above and **verifies the toolchain by running it**: the
Containerfile's `verify` stage executes every tool and every `conformance-clone` refusal as
uid 65532 with no network, at build time. It **never pushes**, and neither does anyone from a
workstation: CI (`.gitea/workflows/images.yaml`) builds the same Containerfile and `verify`
stage on rootless BuildKit (`platform/buildkit/`), and on a push to `main` publishes the
verified image under `sha-<commit>` and `main`. Renovate then moves
`CONFORMANCE_RUNNER_IMAGE_DIGEST` in `conformance/versions.env` onto the registry-returned
digest in a pull request (that file is a sensitive path, so the bump needs an
administrator's `/approve-override`).
`--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
default verify stays offline so an air-gapped build is fully checked, not half-checked).
+51 -129
View File
@@ -1,43 +1,49 @@
#!/usr/bin/env bash
###############################################################################
# conformance/incluster/build-image.sh — build the pinned `conformance-runner`
# image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
# image and prove it works. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
#
# ./build-image.sh # build + verify + print the digest and push command
# ./build-image.sh # build the verify target, then the image
# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes)
# ./build-image.sh --no-verify # build only (skip the in-image tool proof)
# ./build-image.sh --verify-clone # ALSO really clone this repo with conformance-clone.
# # Opt-in because it needs the Gitea host reachable; the
# # default verify is offline so an air-gapped build is
# # still fully checked rather than half-checked.
# ENGINE=docker ./build-image.sh # build with docker instead of podman
# IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this)
#
# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is
# not an agent's to hold, and the pushed digest is what a fail-closed admission policy
# then trusts. So this builds locally, proves the toolchain works INSIDE the image as the
# non-root uid the policy demands, and prints the exact `podman push` line for a human.
# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on
# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and
# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest
# in the job summary; Renovate then opens the pull request that moves conformance/versions.env
# onto that digest. The proof that the image works is the Containerfile's `verify` stage, which
# this script (podman locally) and CI both build.
#
# VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
# single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
# pins the toolbox gate runs with. Nothing in this file invents a version.
#
# THE DIGEST CAVEAT — read it before pasting a digest into anything.
# The digest printed here is the LOCAL manifest digest. It is normally identical to the
# registry digest (podman pushes the manifest it already has rather than recompressing),
# but that is a property of the push, not a guarantee. So the push command below uses
# `--digestfile`: the digest the registry actually returns is the authoritative one, and
# that is the one that goes into conformance/versions.env.
# NO DIGEST IS PRINTED. A manifest digest does not exist until a registry holds the manifest,
# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at
# push time (.gitea/workflows/scripts/images.sh) and Renovate carries it into
# conformance/versions.env.
###############################################################################
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
ENGINE="${ENGINE:-podman}"
TAG_OVERRIDE="${IMAGE_TAG:-}"
VERIFY=1
VERIFY_CLONE=0
PRINT_ARGS=0
for arg in "$@"; do
case "$arg" in
--no-verify) VERIFY=0 ;;
--print-build-args) PRINT_ARGS=1 ;;
--verify-clone) VERIFY_CLONE=1 ;;
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown argument: $arg" >&2; exit 2 ;;
@@ -48,8 +54,6 @@ log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
info() { printf ' %s\n' "$*"; }
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
# --- pins: sourced, never invented -------------------------------------------
TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
[ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
@@ -59,14 +63,21 @@ source "${TOOLCHAIN_ENV}"
: "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
: "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
# THE build args, in ONE place; `--print-build-args` hands them to the CI pipeline, which builds
# this Containerfile on rootless BuildKit.
BUILD_ARGS=("CHAINSAW_VERSION=${CHAINSAW_VERSION}" "KUBECTL_VERSION=${KUBECTL_VERSION}" "FEDORA_VERSION=${FEDORA_VERSION}")
if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
# --- image name: fixed by the admission policy -------------------------------
# platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
# is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here.
IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
# The tag is build convenience only (a human-readable handle for the push). Admission
# ignores it: only the digest is ever referenced from a Job.
IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}"
# The tag is a local handle only. CI re-tags what it publishes (git SHA + `main`), and
# admission ignores tags entirely: only the digest is ever referenced from a Job.
IMAGE_TAG="${TAG_OVERRIDE:-chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}}"
IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}"
log "Building ${IMAGE_REF}"
@@ -76,86 +87,31 @@ info " CHAINSAW ${CHAINSAW_VERSION}"
info " KUBECTL ${KUBECTL_VERSION}"
info " FEDORA ${FEDORA_VERSION}"
"${ENGINE}" build \
--file "${HERE}/Containerfile" \
--tag "${IMAGE_REF}" \
--build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
--build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \
--build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \
"${HERE}"
# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
LABEL_ARGS=()
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
ARG_FLAGS=()
for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
# --- verify: the point of the whole exercise ---------------------------------
# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the
# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root
# filesystem — the shape the admission policy forces. A pass here is the only evidence
# this image is usable; a build that merely succeeded is not.
# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So the proof is that
# the tools EXECUTE, as the NON-ROOT uid, with no network, that conformance-clone is not writable
# by that uid and refuses every hostile input, and that nothing outside /workspace is writable.
# That proof is the Containerfile's `verify` stage, built here and in CI alike (CI has no
# container runtime — rootless BuildKit only). See the stage's header for what a build step can
# and cannot reproduce of the Job's shape. A build that merely succeeded is not evidence.
if [ "${VERIFY}" -eq 1 ]; then
log "Verifying the toolchain inside the image (non-root, read-only rootfs)"
"${ENGINE}" run --rm \
--user 65532:65532 \
--read-only \
--tmpfs /workspace:rw,mode=1777 \
--entrypoint "" \
"${IMAGE_REF}" \
/bin/bash -euo pipefail -c '
echo "--- id ---"; id
echo "--- HOME ---"; echo "HOME=${HOME}"
echo "--- bash ---"; bash --version | head -1
echo "--- python3 ---"; python3 --version
echo "--- git ---"; git --version
echo "--- kubectl ---"; kubectl version --client
echo "--- chainsaw ---"; chainsaw version
echo "--- baked versions ---"; cat /etc/conformance-runner-versions
echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
if [ -w /usr/local/bin/conformance-clone ]; then
echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
fi
echo "confirmed: not writable by uid $(id -u)"
# Each of these MUST fail. Assert the non-zero exit as well as the message: a check
# that only greps stderr would pass a build where the guard had stopped rejecting.
# (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the
# expected failure — without it the verify dies at the first negative case.)
expect_fail() {
local label="$1"; shift
local out rc=0
out="$(env "$@" 2>&1)" || rc=$?
echo "--- ${label} ---"
printf "%s\n" "${out}" | head -2
if [ "${rc}" -eq 0 ]; then
echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
fi
echo "(rejected, exit ${rc})"
}
expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
conformance-clone --layer l2
expect_fail "conformance-clone rejects a missing SHA" \
conformance-clone
expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
CONFORMANCE_GIT_SHA=main conformance-clone
expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
expect_fail "conformance-clone rejects a HOME off the workspace volume" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
HOME=/etc conformance-clone
expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
expect_fail "conformance-clone refuses a clone target off the workspace volume" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
CONFORMANCE_SRC=/etc/src conformance-clone
echo "--- rootfs is read-only (expected: a failure) ---"
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
echo "confirmed: / is read-only"
echo "--- HOME is writable (the workspace emptyDir) ---"
touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable"
echo "--- uid is non-zero ---"
[ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root"
'
log "Verifying the toolchain (Containerfile target: verify)"
"${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}"
log "Toolchain verified"
fi
"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target conformance-runner \
--file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}"
# --- verify the clone helper against the REAL repository (opt-in: needs network) ----
# LOCAL ONLY: this one `run`s the image, which needs a container engine, so CI does not do it.
# The negative cases above prove conformance-clone refuses bad input. This proves the
# positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
# rev-parse assertion is only an unexercised line of shell.
@@ -184,46 +140,12 @@ if [ "${VERIFY_CLONE}" -eq 1 ]; then
log "conformance-clone verified against the real repository"
fi
# --- digest ------------------------------------------------------------------
DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
# --- result ------------------------------------------------------------------
IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
log "Built ${IMAGE_REF}"
info "local manifest digest ${DIGEST}"
info "image id ${IMAGE_ID}"
info "size $(( SIZE / 1024 / 1024 )) MiB"
cat <<EOF
================================================================================
NOT PUSHED. Publishing is an operator action — this script holds no registry
credential and will never acquire one.
OPERATOR: run these two commands, then paste the digest from the digestfile into
conformance/versions.env (CONFORMANCE_RUNNER_IMAGE) via a PR:
${ENGINE} login gitea.home.keithjackson.dev
${ENGINE} push --digestfile /tmp/conformance-runner.digest \\
${IMAGE_REF}
Then:
cat /tmp/conformance-runner.digest
# -> sha256:.... THIS is the authoritative digest, not the local one above.
# conformance/versions.env
CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)"
WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml
is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not
${IMAGE_REPO}@sha256:<digest>. A tag reference is denied,
and so is any other repository path.
TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard
\`${IMAGE_REPO}@sha256:*\` to be replaced with the
LITERAL digest once the image exists, closing the "anyone who can push to this path" gap.
That is a separate, human-reviewed PR against the security boundary.
================================================================================
EOF
info "image id ${IMAGE_ID}"
info "size $(( SIZE / 1024 / 1024 )) MiB"
info "Not pushed. CI publishes this image from main (.gitea/workflows/images.yaml) and"
info "Renovate moves conformance/versions.env onto the registry-returned digest."
@@ -29,7 +29,8 @@
# THEN (1) promtool is a PINNED member of the toolchain: PROMTOOL_VERSION is declared in BOTH
# infra/devcontainer/versions.env and ci/versions.env and the two AGREE (the repo's
# lockstep rule), the Containerfile takes it as a build ARG, installs it from the
# pinned prometheus release and records it for provenance, dev.sh passes it, and the
# pinned prometheus release and records it for provenance, build-image.sh (the one
# recipe `dev.sh build` and CI both use) passes it, and the
# CI runner installs it for the l0 stage;
# (2) the promtool ACTUALLY IN USE reports exactly that pinned version — a unit test is
# evidence only if the engine matches the one prod evaluates rules with;
@@ -246,10 +247,12 @@ check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text,
"infra/devcontainer/Containerfile does not record promtool in /etc/dev-toolchain-versions — "
"that provenance line is how an operator tells a stale image from a current one")
DEVSH = ROOT / "infra/devcontainer/dev.sh"
check(DEVSH.is_file() and "--build-arg PROMTOOL_VERSION=" in DEVSH.read_text(),
"infra/devcontainer/dev.sh does not pass --build-arg PROMTOOL_VERSION — the ARG would be "
"empty and the build would fetch a bogus URL")
# The build recipe moved from dev.sh into build-image.sh on 2026-10-29, so that `dev.sh build`
# and the CI image pipeline (.gitea/workflows/images.yaml) share ONE recipe. Check the recipe.
BUILDSH = ROOT / "infra/devcontainer/build-image.sh"
check(BUILDSH.is_file() and '"PROMTOOL_VERSION=$PROMTOOL_VERSION"' in BUILDSH.read_text(),
"infra/devcontainer/build-image.sh does not pass PROMTOOL_VERSION in its BUILD_ARGS — the ARG "
"would be empty and the build would fetch a bogus URL")
INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh"
if INSTALL.is_file():
@@ -42,6 +42,19 @@
# whose every solver is an automated challenge type (dns01 / http01) — the
# same code path serves issuance AND renewal, so a solver a human must
# complete would defeat both;
# (5b) the ONE exception to (5) — WORKLOAD mTLS, added 2026-10-29 — is a Certificate
# annotated `homelab.keithjackson.dev/tls-scope: workload-mtls`. The spec's TLS
# requirements are about EXTERNALLY REACHABLE services (R-TLS-01; docs/08), and
# ADR-0006 says outright that "mTLS between workloads is a separate concern … not
# covered here". A public ACME CA cannot issue for a `*.svc.cluster.local` name or a
# client-auth certificate at all. The annotation is NOT trusted: an exempt
# Certificate must PROVE it is workload-internal — its Secret is served by NO Gateway
# listener, every DNS name / commonName is cluster-internal (a bare label or
# `*.svc[.cluster.local]`), it carries no IP/URI/email SANs, and its issuer is a
# NAMESPACED Issuer in its own namespace whose spec is `selfSigned` or a `ca` backed
# by a Secret that a tracked, likewise-exempt Certificate produces. Issuance and
# renewal stay unattended (cert-manager renews CA-issued certs itself), and (9)
# still applies. Every exemption is printed. First user: platform/buildkit.
# (6) each dns01 solver's provider credential is referenced BY NAME and that
# Secret is delivered SOPS-encrypted from git via a kustomization — the
# operator's one-time token hand-off (docs/08 OQ-4), never a per-cert step;
@@ -265,6 +278,7 @@ CERT_SECRETS: dict[tuple[str, str], str] = { # (namespace, secretName)
for rel, d in CERTS
}
listeners_checked = 0
GATEWAY_SECRETS: set[tuple[str, str]] = set() # every (namespace, Secret) a listener serves
for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
gw_ns = ns_of(gw)
for lis in ((gw.get("spec") or {}).get("listeners") or []):
@@ -280,6 +294,7 @@ for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
for r in refs:
kind = str(r.get("kind", "Secret"))
key = (str(r.get("namespace") or gw_ns), str(r.get("name")))
GATEWAY_SECRETS.add(key)
check(kind == "Secret",
f"{rel}: listener {lis.get('name')!r} certificateRef kind is {kind!r}; only "
"Secret refs can be traced to a cert-manager Certificate")
@@ -324,8 +339,61 @@ CM_NS = ns_of(cm_releases[0][1], "cert-manager") if cm_releases else "cert-manag
ISSUER_INDEX = {(d.get("kind"), name_of(d)): (rel, d) for rel, d in ISSUERS}
solver_creds: list[tuple[str, str, str, str]] = [] # (issuer, ns, secretName, ref field)
issuers_verified: set[str] = set()
# (5b) workload-mTLS exemption — see the header. Proven, never taken on trust.
SCOPE_ANN = "homelab.keithjackson.dev/tls-scope"
INTERNAL_NAME = re.compile(r"^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*$")
NS_ISSUERS = {(ns_of(d), name_of(d)): (rel, d) for rel, d in of_kind("Issuer", "cert-manager.io/")}
EXEMPT_SECRETS = {(ns_of(d), str((d.get("spec") or {}).get("secretName", "")))
for _, d in CERTS
if ((d.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN) == "workload-mtls"}
exempted: list[str] = []
def workload_mtls_ok(rel: str, cert: dict) -> bool:
"""True if `cert` claims the workload-mTLS scope AND proves it; records failures."""
ann = ((cert.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN)
if ann is None:
return False
cns, cname = ns_of(cert), name_of(cert)
spec = cert.get("spec") or {}
ok = True
def need(cond, msg):
nonlocal ok
check(cond, f"{rel}: Certificate {cns}/{cname} claims {SCOPE_ANN}={ann!r} but {msg}")
ok = ok and bool(cond)
need(ann == "workload-mtls", "the only accepted value is 'workload-mtls'")
need((cns, str(spec.get("secretName", ""))) not in GATEWAY_SECRETS,
"its Secret is served by a Gateway listener — that is external TLS, which must be ACME (R-TLS-01)")
names = [str(n) for n in (spec.get("dnsNames") or [])]
if spec.get("commonName"):
names.append(str(spec["commonName"]))
for n in names:
internal = INTERNAL_NAME.match(n) and ("." not in n or n.endswith(".svc") or n.endswith(".svc.cluster.local"))
need(bool(internal), f"name {n!r} is not cluster-internal (bare label or *.svc[.cluster.local])")
for f in ("ipAddresses", "uris", "emailAddresses"):
need(not spec.get(f), f"it carries {f}, which a workload-internal certificate has no use for")
ref = spec.get("issuerRef") or {}
need(str(ref.get("kind", "Issuer")) == "Issuer",
"its issuer is not a NAMESPACED Issuer — a ClusterIssuer could sign for any namespace")
hit = NS_ISSUERS.get((cns, str(ref.get("name", ""))))
need(hit is not None, f"its Issuer {ref.get('name')!r} is not tracked in namespace {cns!r}")
if hit is not None:
ispec = hit[1].get("spec") or {}
kinds = sorted(ispec)
need(kinds in (["selfSigned"], ["ca"]),
f"its Issuer is {kinds} — only selfSigned or ca are accepted for workload mTLS")
if kinds == ["ca"]:
need((cns, str((ispec.get("ca") or {}).get("secretName", ""))) in EXEMPT_SECRETS,
"its CA Issuer's Secret is not produced by a tracked workload-mtls Certificate "
"in the same namespace — the CA itself would have to be minted by hand")
if ok:
exempted.append(f"{cns}/{cname}")
return ok
for rel, cert in CERTS:
spec = cert.get("spec") or {}
if workload_mtls_ok(rel, cert):
continue # (5b): proven workload-internal; (9) below still applies
ref = spec.get("issuerRef") or {}
kind = str(ref.get("kind", "Issuer"))
per_env = resolve_per_env(str(ref.get("name", "")))
@@ -379,6 +447,8 @@ for rel, cert in CERTS:
solver_creds.append((iname, want_ns, n, f"{provider}.{field}"))
check(len(issuers_verified) > 0, "no issuer was reachable from any Certificate — (5) proved nothing")
if exempted:
print(f" (5b) workload-mTLS certificates exempted from ACME after proving it: {', '.join(exempted)}")
# (6) the solver credential is delivered, SOPS-encrypted, from git — never per-cert by hand.
check(len(solver_creds) > 0,
+18 -21
View File
@@ -40,32 +40,29 @@ PYTHON_MIN="3.9"
# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*
# Any other path, and any TAG reference, is DENIED at admission. Do not "tidy" these values.
CONFORMANCE_RUNNER_IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
# Build handle only. Admission never sees the tag; a Job must reference the digest.
# Local build handle only. Admission never sees the tag; a Job must reference the digest.
# CI publishes under `sha-<commit>` and `main` instead (.gitea/workflows/images.yaml).
CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
#
# ############################################################################
# # DIGEST DELIBERATELY EMPTY — THE IMAGE HAS NOT BEEN PUBLISHED YET. #
# # THE DIGEST IS WRITTEN BY RENOVATE, NOT BY HAND. #
# ############################################################################
# The image has been BUILT and its toolchain verified locally (non-root uid 65532,
# read-only rootfs: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4, chainsaw 0.2.12,
# and `./run.sh --layer l2 --no-cluster` completing with rc 0). It has NOT been pushed:
# pushing needs a registry credential, and publishing the artifact a fail-closed security
# boundary then trusts is an OPERATOR action, not an agent's.
# CI builds and verifies the image (non-root uid 65532, read-only rootfs, the
# conformance-clone refusals) and, on a push to main, publishes it and prints the digest the
# REGISTRY returned (.gitea/workflows/images.yaml). Renovate follows the image's `main` tag
# through the marker below and opens the pull request that moves this value. Nobody pushes
# this image from a workstation, and nobody pastes a digest here.
#
# OPERATOR, to complete the pin:
# 1. ./conformance/incluster/build-image.sh # rebuild, verify, print the commands
# 2. podman login gitea.home.keithjackson.dev
# 3. podman push --digestfile /tmp/conformance-runner.digest \
# "${CONFORMANCE_RUNNER_IMAGE_REPO}:${CONFORMANCE_RUNNER_IMAGE_TAG}"
# 4. Set CONFORMANCE_RUNNER_IMAGE_DIGEST below to `cat /tmp/conformance-runner.digest`
# — the digest the REGISTRY returned, not the local manifest digest the build printed.
# 5. Land steps 4 in a PR (this file is GitOps input, not a live edit).
#
# UNTIL THEN: this is empty on purpose, NOT a placeholder to be guessed at. An empty value
# makes any consumer that renders the Job fail visibly; a fake or stale digest would instead
# produce an ImagePullBackOff that looks like a cluster fault. Do not invent one.
CONFORMANCE_RUNNER_IMAGE_DIGEST=""
CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_DIGEST:+${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}}"
# ALL-ZERO UNTIL THE FIRST PUBLISH. Before 2026-10-29 this was EMPTY, so that a consumer
# rendering the Job would fail visibly. It is now the same impossible all-zero digest
# platform/pm-agent/ uses, because Renovate can only replace a digest that is there to match.
# A consumer MUST treat the all-zero value as "not published" (no consumer exists yet: the
# L2/L3 trigger is unwired, docs/plans/l2-l3-conformance-ci-wiring.md). Note also that
# chainsaw-job.yaml's @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ takes the 64 hex digits WITHOUT the
# `sha256:` prefix this value carries — the trigger must strip it.
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/conformance-runner tag=main
CONFORMANCE_RUNNER_IMAGE_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000"
CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}"
#
# AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for
# its `@sha256:*` wildcard to be replaced with the LITERAL digest once the image exists —
@@ -0,0 +1,78 @@
---
date: 2026-10-18
branch: fix/renovate-cron-window
title: "The Renovate cron fired outside Renovate's own schedule, so it would never have opened a PR"
---
## What changed
`.gitea/workflows/renovate.yaml` shipped (PR #73) with `cron: "15 3 * * *"`. **That schedule could
never have produced a single PR.**
Gitea Actions cron is **UTC**. `renovate.json` carries `"schedule": ["after 3am and before 6am"]`
with `"timezone": "America/New_York"`, and that inner schedule is **authoritative** — Renovate
re-checks it and declines to open PRs outside it. So:
```
cron 03:15 UTC -> 23:15 EDT, the PREVIOUS EVENING in New York
window -> 03:00-06:00 local
inside? -> NO (3h45m before the window opens)
```
The original comment claimed the cron was "slightly ahead of the repo window ... so a run is always
in flight when the window opens". It is 3h45m *behind*, on the previous day. Every nightly run would
have started, logged "not within schedule", opened **zero** PRs and exited **successfully** — a
green tick forever, indistinguishable from "nothing needed updating". That is the same
"failure-mode-is-a-green-tick" shape this repo keeps hitting, and it would have hidden the ~30
pending updates PR #73 was written to surface.
Changed to `cron: "15 8 * * *"`, chosen to survive DST rather than to be tidy. The window's UTC
position moves:
| | local window | in UTC |
|---|---|---|
| EDT (UTC-4, Mar–Nov) | 03:00–06:00 | 07:00–10:00 |
| EST (UTC-5, Nov–Mar) | 03:00–06:00 | 08:00–11:00 |
`08:15 UTC` is inside **both** (04:15 EDT / 03:15 EST), so no edit is needed at either switchover.
A cron cannot follow DST, so this is the only way to get it right once.
## What I verified
* **The defect, by arithmetic rather than assertion:** `datetime(2026,10,18,03,15, UTC)` converted to
`America/New_York` via `zoneinfo` is `2026-10-17 23:15 EDT`; `3 <= hour < 6` is **False**.
* **The fix, at both DST extremes:** `08:15 UTC` → `04:15 EDT` in July and `03:15 EST` in December,
both inside `03:00–06:00`. Computed, not reasoned.
* **The inner schedule really is authoritative and really is set:** `renovate.json` on `main` carries
`timezone = America/New_York` and `schedule = ['after 3am and before 6am']`.
* **`RENOVATE_TOKEN` now exists** as a repository Actions secret (listed by name via the Gitea API
alongside `SETTINGS_API_TOKEN`), so the workflow's fail-hard preflight will pass. It did not exist
when PR #73 was authored.
* The edited workflow parses as YAML.
## What I did NOT verify
* **Renovate has still never executed.** This fixes *when* it runs, not *whether it works*. No run
has happened, so nothing here proves it extracts dependencies, authenticates to Gitea, or opens a
usable PR.
* **`enabledManagers` may silently disable the custom managers — UNRESOLVED and the likelier
remaining bug.** `renovate.json` lists `enabledManagers: [..., 'regex', ...]` and defines
`customManagers` (correct modern key; `regexManagers` is absent, which is right). But modern
Renovate names the custom manager **`custom.regex`** in `enabledManagers`, not bare `regex`. If
the bare name is invalid or ignored, every inline `# renovate:` marker in the tree does nothing —
including the `external-snapshotter` `newTag` pin PR #73 flagged as probably inert. **I did not
change it**, because guessing at a config key that could fail validation and take the whole run
down is worse than leaving a question that one dry run answers definitively. The `dry_run: true`
dispatch log settles it: look for a per-manager dependency-extraction count, and for any config
validation warning naming `regex`.
* **Whether a dry run inside vs outside the window differs usefully.** Renovate honours its schedule
even in dry-run, so a manual dispatch outside 03:00–06:00 local will also log "not within
schedule". That makes the first dispatch's output less informative than it looks; to see what
Renovate *would* do, either dispatch inside the local window or override the schedule for that
run. Not tested either way.
* **No alert fires if the schedule stops firing.** PR #73 recorded this gap and it is still open: a
Renovate workflow that silently stops running (disabled Actions, expired token, deleted secret)
produces no signal at all. Closing it needs a check on last-successful-run age, which nothing
does.
* No conformance test asserts the cron falls inside `renovate.json`'s window, so this exact bug can
return. A static check comparing the two is feasible and is not written.
@@ -0,0 +1,78 @@
---
date: 2026-10-18
branch: fix/renovate-invalid-config
title: "renovate.json was invalid, so the first manual run errored out"
---
## What changed
The operator ran the Renovate workflow manually and it **failed**. The cause is not the workflow:
**`renovate.json` was never a valid Renovate config.**
It carried five pseudo-comment keys — `_comment`, `_kubernetes_comment`, `_unmanageable_comment`,
`_ignorePaths_comment`, `_docker_comment` — because JSON has no comments. Renovate **rejects unknown
top-level options**, so the entire config failed validation. Proven with the official validator
rather than inferred:
```
$ podman run --rm -v "$PWD/renovate.json:/tmp/renovate.json:ro,Z" \
ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json
ERROR: Found errors in configuration
Invalid configuration option: _comment
Invalid configuration option: _docker_comment
Invalid configuration option: _ignorePaths_comment
Invalid configuration option: _kubernetes_comment
Invalid configuration option: _unmanageable_comment
```
This is the second independent reason Renovate had never worked, on top of PR #77's cron firing
outside `renovate.json`'s own schedule window. Either alone was sufficient to produce "nothing ever
happens"; #73 shipped both.
**Converted `renovate.json` → `renovate.json5`.** JSON5 permits real `//` comments, and Renovate
discovers `renovate.json5` natively — nothing in `.gitea/` or `ci/` pins the old filename (only
prose inside comments), and no `RENOVATE_CONFIG_FILE` is set. Every word of the original rationale
is preserved as comments positioned next to what it explains, which a flattened `description`
string would have lost.
A header on the new file records the failure, the validator command, and the instruction to run it
before committing — this class of bug is invisible to the repo's own gate.
## What I verified
* **The exact failure**, from the official validator, quoted above. Not read off a log, not inferred
from the workflow.
* **The fix validates**: the same validator against `renovate.json5` returns
`INFO: Config validated successfully against 1 file(s)`.
* **Nothing was dropped in the conversion**, checked programmatically against `git show
HEAD:renovate.json`: 18 non-comment keys before, 18 after, **no missing keys, no extra keys, and
no key whose value changed**. `docker: {enabled: true}` confirmed still present — the validator
prints a normalisation diff that mentions it, which is its own migration suggestion, not a
removal.
* **74 comment lines survive** in the new file, so the rationale is intact.
* **`RENOVATE_TOKEN` exists** as a repository Actions secret (listed by name via the Gitea API), so
the workflow's fail-hard preflight is satisfied.
* **A suspicion was DISPROVED:** `enabledManagers` containing bare `regex` was thought to need to be
`custom.regex`, which would have silently disabled every inline `# renovate:` marker. The
validator accepts `regex`, so that concern was unfounded and nothing here changes it.
* Nothing functionally references `renovate.json` by name — `grep -rn "renovate\.json" .gitea/ ci/
conformance/` returns only comment prose.
## What I did NOT verify
* **Renovate has still never completed a run.** This removes one of two blockers; PR #77 removes the
other. Neither is merged, so **no successful Renovate run has been observed** and I cannot say the
bot works — only that these two specific defects are fixed.
* **I did not re-run the workflow after the fix.** The honest test is a `workflow_dispatch` with
`dry_run: true` once both this and #77 are merged. Expect ~30 dependencies behind, rate-limited to
`prHourlyLimit: 2` / `prConcurrentLimit: 5`.
* **Whether the managers actually extract anything.** A valid config that matches no files is still
a no-op. The dry-run log's per-manager dependency counts are the only thing that proves the
`managerFilePatterns` are right, and I have not seen them.
* **Whether `renovate.json5` is honoured by the pinned Renovate version in the workflow.** JSON5
discovery is long-standing and the validator accepted the file, but the workflow's image tag was
not cross-checked against the version that added it.
* **Nothing in the gate catches an invalid Renovate config.** This bug would have recurred silently.
A one-line L0 check shelling out to `renovate-config-validator` would close it; it is not written
here, and `conformance/**` is a sensitive path.
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
+374
View File
@@ -0,0 +1,374 @@
---
date: 2026-10-29
branch: feat/image-pipeline
title: "Images are built, verified and published by CI on rootless BuildKit; Renovate carries the digests into git"
---
## What changed
Operator rule (2026-10-29): container image creation, hosting and lifecycle belong to CI/CD and
the codebase, never to a one-off workstation build/push. Before this branch the Home-Network org's
package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and
both `build-image.sh` scripts said "publishing is an operator action".
### The builder: rootless BuildKit — DECISION MADE (operator, on PR #81)
**The problem it answers, measured on this PR's first CI run (images run 325, failed in 6 s):** a
CI job on this runner gets **no Docker daemon at all**, for two reasons, both in the deployed
versions' source:
- act_runner v0.2.11 `internal/app/cmd/daemon.go` sets `container.docker_host` to `-` ("never
mount into jobs") whenever `DOCKER_HOST` is `tcp://`, which it is here (the DinD sidecar);
- gitea/act v0.261.2 `pkg/container/docker_run.go` `sanitizeConfig` drops **every** bind,
socket included, when `valid_volumes` is empty, which it is (the runner has no config file).
Giving jobs the DinD daemon would have made every branch push root on the node, because that
daemon is `privileged: true`.
**The decision:** no job may reach a privileged daemon, and images are built by a **rootless
BuildKit** daemon over mTLS. The runner (`apps/wave-1/gitea-actions.yaml`, DOCKER_HOST,
valid_volumes) is **untouched**.
- **`platform/buildkit/`** — a new hand-managed component in its own namespace `buildkit`. The
runner manifest is renderer-generated (`catalog/render.py` EXTRA_UNITS), and the builder has no
business sitting next to a privileged container. Flux reconciles it as the new Kustomization
`buildkit` (`clusters/base/buildkit.yaml`, `dependsOn: platform`).
- **Deployment:** `moby/buildkit:v0.34.1-rootless@sha256:f68752a5…0c4f89`, pinned by index
digest. The securityContext, every line measured (see *What I verified*):
- `privileged: false`;
- uid/gid 1000, `runAsNonRoot`;
- `capabilities: drop ALL, add SETUID, SETGID`;
- `allowPrivilegeEscalation: true`;
- `seccompProfile: Unconfined` and `appArmorProfile: Unconfined`;
- `readOnlyRootFilesystem: true`, with five emptyDirs.
Other settings:
- `--oci-worker-no-process-sandbox`;
- no ServiceAccount token;
- memory limit 4Gi, which covers build steps too;
- cache emptyDir with `sizeLimit: 6Gi`, and GC capped at 4 GB in `buildkitd.toml`
(ConfigMap).
- **mTLS via cert-manager** (`certificates.yaml`): a self-signed CA used only here, the server
cert `buildkitd-tls`, and the client cert `buildkit-client-ci`. buildkitd verifies every
client against that CA.
- **Service:** ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`.
- **NetworkPolicies:**
- default-deny;
- ingress on 1234 **only from the act-runner pod** (`gitea-actions`,
`app.kubernetes.io/name=act-runner`);
- egress: DNS, world :443/:80 for downloads, and the `ingress` entity (the Gateway) for the
registry push.
- `versions.env`: the CI-side client, `buildx` v0.38.1 with the sha256 from its
`checksums.txt`, plus the address and certificate name CI dials.
- **How DinD job containers appear to Cilium (measured, read-only).**
`iptables -t nat -S POSTROUTING` in the dind container shows
`-A POSTROUTING -s 172.18.0.0/16 ! -o br-… -j MASQUERADE` (and the same for docker0). So a job's
connection leaves with the **act-runner pod's IP, and therefore that pod's Cilium identity**.
That makes "only the act-runner pod" expressible and correct, but it cannot tell one job from
another, or a job from act_runner itself. Per-job authentication is the client certificate.
- **Admission: no Kyverno exception needed.**
- Every ClusterPolicy actually applied (`platform-config/kustomization.yaml`; live:
`conformance-trigger-job-shape`, `label-cdi-worker-pods`, `pm-agent-job-shape`,
`require-auth-mode-label`, `require-default-deny-baseline`, `require-egress-class-label`,
`restrict-host-namespaces`, `secret-provenance`, `validate-default-deny-shape`) matches only
namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`, or only its own
namespace in the job-shape policies' case. `buildkit` carries no such label.
- There are no namespaced Kyverno Policies and no cluster-wide PSA default (both measured).
- The namespace is deliberately left without a PSA label, because PSS `baseline` forbids
`Unconfined` seccomp and AppArmor.
- `validate-default-deny-shape` does apply to the namespace's default-deny NetworkPolicy, and
its shape (both directions) satisfies it.
- **Unprivileged user namespaces on the nodes (measured, read-only, via Ansible `sysctl`).** On
all three Debian 13.7 nodes, kernel `7.1.8+deb13-amd64`:
- `kernel.unprivileged_userns_clone = 1`;
- `user.max_user_namespaces = 127206`;
- no `kernel.apparmor_restrict_unprivileged_userns` (an Ubuntu-only key);
- AppArmor enabled (`Y`).
So this is a satisfied prerequisite, not an open one.
- **The TLS conformance check needed a narrow, self-proving exception** (`conformance/**`, so this
changes what green means; flagged for the reviewer).
- The first gate run failed L0 `tls-lifecycle-automated` clause (5): "every Certificate's
issuer must be ACME". That is broader than the spec it serves:
- R-TLS-01 and docs/08 concern *externally reachable* services;
- ADR-0006 says "mTLS between workloads is a separate concern … not covered here";
- a public CA cannot issue for a `*.svc.cluster.local` name or a client-auth certificate.
- The new clause (5b) exempts a Certificate annotated
`homelab.keithjackson.dev/tls-scope: workload-mtls` **only if it proves it**:
- no Gateway listener serves its Secret;
- every DNS name and the commonName are a bare label or `*.svc[.cluster.local]`;
- it has no IP, URI or email SANs;
- its issuer is a namespaced `selfSigned`/`ca` Issuer in its own namespace;
- a `ca` issuer's Secret is produced by a likewise-exempt tracked Certificate.
- Exemptions are printed, and clause (9), the renewal arithmetic, still applies.
- **Negative-tested:** a public DNS name, and a Gateway serving `buildkitd-tls`, each fail the
check with a specific message.
- **`infra/gitea/buildkit-client-secret.sh`** (operator action): copies `ca.crt`, `tls.crt` and
`tls.key` from the Secret `buildkit/buildkit-client-ci` into the Actions secrets `BUILDKIT_CA`,
`BUILDKIT_CLIENT_CERT` and `BUILDKIT_CLIENT_KEY`. Before writing anything it checks that:
- the cert verifies against the CA;
- the cert is a client-auth cert;
- the key matches the cert.
It never prints a value. `--check` reports whether the stored copy is older than the current
certificate.
### Verification moved into the Containerfiles
`docker run` cannot exist on BuildKit, so each Containerfile now names its shipped stage after the
image (`toolbox`, `agent-runner`, `conformance-runner`) and ends in a **`verify` test stage**,
`FROM <image> AS verify`. That stage runs every previous assertion at build time, as the image's
non-root uid (65532 for agent-runner and conformance-runner), in `RUN --network=none`. It is never
pushed.
- **Every existing assertion is preserved**, and the toolbox gained one it never had: every tool
on PATH, and five tools report their pinned versions.
- `probe-validation.sh` is `COPY`ed into the verify stage only, so it never ships.
- **Bug fixed on the way:** in the old `docker run` block, `cmd | grep -q … && echo ok` and
`[ … ] && echo` could never fail the script under `set -e` + `pipefail`. The "entrypoints refuse
arguments", "not root", "no API key baked in" and "HOME writable" checks asserted nothing. They
are real assertions now.
- **What a build step cannot reproduce, and how it is approximated** (also written in each stage's
header):
- **uid:** exact.
- **`--network none`:** exact, and each stage *proves* it. Its first check fails if DNS or a
TCP connection works. A negative control without `--network=none` fails that check.
- **`--read-only` rootfs:** not reproducible, because a RUN step always has a writable layer.
It is approximated by permission:
- `touch /should-fail` must fail;
- a `find / -xdev -writable ! -type l` inventory, run as the uid, must lie entirely under
`/workspace`, `/tmp` or `/var/tmp`. `! -type l` was needed: Fedora's `/usr/tmp` symlink
to `/var/tmp` tripped it.
The real control remains the pods' `readOnlyRootFilesystem`, which the admission policies
enforce.
- **`--tmpfs /workspace`:** approximated by the image's own `/workspace`, owned by 65532, which
is exactly what the pod's emptyDir replaces.
The opt-in `--verify-clone` still `run`s the image, so it remains local-only.
- **`build-image.sh` (all three):**
- verify is `build --target verify` and the image is `build --target <image>`;
- a new `--print-build-args` prints the pins as `KEY=VALUE`, which is what CI consumes. The
arg list therefore exists once;
- the local podman path still works.
- The L0 check `alert-rules-unit-tested` now looks for `"PROMTOOL_VERSION=$PROMTOOL_VERSION"` in
the toolbox's build-arg list.
### The pipeline (`.gitea/workflows/images.yaml` + `scripts/images.sh`)
- **The builder.** The driver downloads the pinned `buildx` and refuses a checksum mismatch. It
writes the client cert from the `BUILDKIT_*` secrets to a 0600 temp dir that is removed on exit,
creates a `remote` builder, and runs `buildx inspect --bootstrap`.
- **It fails loudly, never skips.** It stops with a clear message if:
- the secrets are absent;
- buildkitd is unreachable. The message lists the causes in order: not deployed yet (Flux
applies it only after merge), stale certificate, NetworkPolicy.
- **Pull request:** `buildx bake <image>-verify` for changed images. Nothing is pushed and no
registry credential is referenced. The `BUILDKIT_*` builder secrets *are* given to PR jobs,
because they must build.
- agent-runner is built FROM the toolbox built **in the same run** through a bake `target:`
context. That toolbox exists in no registry, so the toolbox is built as its base even when
its own inputs did not change.
- **Push to main / dispatch on main:** for each image, the verify target first, and only then the
image itself with `push=true`. Tags are `sha-<12 hex>` (immutable) and `main` (moving).
- The digest is read back from the registry and compared with the client's metadata.
- agent-runner builds FROM the toolbox digest just pushed, or else from the pinned
`AGENT_BASE_IMAGE` (the zero placeholder bootstraps).
- `BUILDX_NO_DEFAULT_ATTESTATIONS=1` keeps one manifest per tag, which is what admission,
Renovate and the retention script assume.
- **Unchanged from the first iteration:** change detection, publish idempotence,
`REGISTRY_PUSH_TOKEN` and its preflight, the Renovate markers, and `registry-retention.sh`. The
DinD prune step is gone, because nothing is built in DinD now.
### Unchanged from the first iteration (still true)
- **Digest references** carry `# renovate-digest: image=<repo> tag=main` markers:
- `platform/pm-agent/deployment.yaml` and `configmap.yaml`;
- `infra/agent/versions.env` `AGENT_BASE_IMAGE`, now the published toolbox by digest, zero
until the first publish;
- `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to zero.
The admission policies are untouched.
- **`renovate.json5`**:
- the `custom.regex` marker manager;
- `enabledManagers` `regex` → `custom.regex`;
- the `kubernetes` manager disabled for these three repos;
- `ignorePaths` narrowed so `conformance/versions.env` is visible.
`renovate.yaml` pins `renovate-version: 44.167.0` and adds a docker `RENOVATE_HOST_RULES` entry
using `RENOVATE_TOKEN`, which now needs `read:package`.
- **`infra/gitea/registry-retention.sh`** (dry run by default; operator action); the
`conformance-trigger-kubeconfig.sh` precondition fix; and the docs (pm-agent README, runbook,
plan, conformance README, devcontainer README, workflows README). The docs are updated again
for BuildKit, and there is a new `platform/buildkit/README.md`.
### Decisions and pushback
- **The push credential is a PAT secret, `REGISTRY_PUSH_TOKEN`; the Actions token cannot push.**
Checked against Gitea **v1.24.3** source:
- `services/auth/basic.go` `Basic.Verify` turns a task token into the pseudo-user
`gitea-actions` (ID −2) and sets no token scope;
- `routers/api/packages/container/container.go` `Authenticate` (the `/v2/token` exchange every
docker client performs) requires `read:package`/`write:package`/`all` scope, so it answers
401. The Actions token cannot even *pull*;
- `services/context/package.go` `determineAccessMode` has `// TODO: ActionUser permission check`
and no branch for that user. Even past login it would get at most read on a public org;
- `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3.
The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight
pattern. With BuildKit the credential reaches the daemon through the buildx session (a
0600 `config.json` in the job's temp dir), never through the cluster.
- **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate:
- `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` →
`custom.regex`;
- that migration runs on every repository run (`lib/workers/repository/init/merge.ts` →
`migrateAndValidate`), not only in the validator;
- `lib/modules/manager/index.ts` `getEnabledManagersList` accepts either spelling;
- `custom.regex` exists since 37.51.1 (renovatebot/renovate#24079).
- **The open question hid a bigger bug: the Renovate config on main is invalid for the Renovate
the workflow actually runs.**
- `renovatebot/[email protected]` defaults `renovate-version` to `39`
(`action.yml`, `src/docker.ts`), which is 39.264.0.
- The validator used in #79 was `ghcr.io/renovatebot/renovate` with no tag, i.e. `latest`,
44.167.0.
- **Measured:** `renovate:39.264.0 renovate-config-validator` on main's `renovate.json5` gives
`ERROR: Found errors in configuration` with five errors, all `managerFilePatterns`.
`fileMatch` was renamed in 40.2.0.
- So once #80 fixes the filename, every run would still die at `CONFIG_VALIDATION`.
- Fixed by pinning `renovate-version: 44.167.0`. This branch's config validates clean on it.
The only `--strict` complaint is a pending migration of the pre-existing top-level
`"docker": {"enabled": true}` block, which I left alone.
- **Unworkable as briefed: tracking the worker Job template's digest.**
- `infra/agent/worker-job-template.yaml` is `COPY`ed into agent-runner. A Renovate bump there
changes agent-runner's inputs, CI publishes a new digest, Renovate bumps it again, and so on
forever.
- `pm-dispatch` overwrites that field from `PM_WORKER_IMAGE` at runtime
(`pm-dispatch.py:649`), so nothing is lost by leaving it all-zero. It is commented as
deliberately untracked.
- For the same loop reason, `conformance/versions.env` is **not** an input of the
conformance-runner build, and a pipeline-only change does not republish on main.
- **Changed from the brief: retention is a script, not a Gitea cleanup rule.**
- Gitea 1.24.3 has **no API** for cleanup rules (none in `routers/api/v1/api.go`; web
session+CSRF forms only, in `routers/web/web.go`). So a rule cannot be created, or
drift-asserted, as code. **`ci/gitea-settings.sh` was therefore not extended.** A cheaper
proxy (count versions per image) would need `read:package` on `SETTINGS_API_TOKEN`, which
it lacks.
- A built-in rule is also **unsafe here**. Gitea stores a manifest as a file of the TAG
version it was pushed under (`routers/api/packages/container/manifest.go`
`createPackageAndVersion`), and resolves a by-digest pull from those files
(`models/packages/container/search.go`). Deleting the tag a deployed digest was pushed under
breaks the `@sha256:` pull, and the rule's "newest N" knows nothing about what git pins. The
script keeps every digest git pins.
- **Auto-merge.** Digest bumps of these images auto-merge on a green gate (the existing
`digest` rule). The exception is the conformance-runner bump: `conformance/versions.env` is a
sensitive path, so it waits for an administrator's `/approve-override` every time.
## What I verified
**Proved locally, against the exact pinned images:**
- **Rootless buildkitd, configured as the manifest configures it.**
- Setup: the pinned `moby/buildkit:v0.34.1-rootless` digest, under rootless podman, with
`--cap-drop ALL --cap-add SETUID --cap-add SETGID`, seccomp unconfined and `--read-only`;
tmpfs at the same five paths; the ConfigMap's `buildkitd.toml`; mTLS on :1234.
- The image's `/etc/subuid` was overridden with `user:1:999` and `user:1001:64535`, because
rootless podman maps only 65,537 uids. rootlesskit's real `newuidmap` user-namespace path
still ran.
- It started, `buildctl debug workers` listed the OCI worker, and `debug workers -v` showed the
GC config applied (max 4.29 GB, reserved 1.07 GB).
- **The securityContext, item by item.** Each requirement was measured by taking it away:
- **default seccomp:** daemon starts, every RUN step fails → `Unconfined` is required;
- **no `SETUID`/`SETGID`:** rootlesskit dies with
`newuidmap … operation not permitted` → both capabilities are required;
- **`--read-only` rootfs:** works once the five emptyDirs exist. Its only failure was the
probe client's `~/.docker`, which is now an emptyDir;
- **minimal caps + Unconfined seccomp:** works, and a RUN step executes.
- **The full CI path, in the actual CI job image.** Setup: `ghcr.io/catthehacker/ubuntu:act-22.04`
running `.gitea/workflows/scripts/images.sh` in verify mode. It downloaded and sha256-checked
buildx v0.38.1, connected over mTLS, and bake-built all three `verify` targets from an empty
cache. rc 0, 1m32s total:
- `TOOLBOX VERIFIED` (42 s);
- `AGENT-RUNNER VERIFIED` (26 s): selftest OK, probe "20 refusal(s), 0 failure(s)", and built
FROM the in-run toolbox through the `target:` context, with the log showing
`[agent-runner-verify toolbox 1/6]` and `base=toolbox-in-this-run`;
- `CONFORMANCE-RUNNER VERIFIED` (21 s): all seven refusals;
- every stage printed `confirmed: no network`.
- **mTLS rejects what it should.** Tested with a real BuildKit API call (`buildctl debug workers`):
- no client cert → `remote error: tls: certificate required`;
- a client cert from another CA → rejected the same way;
- the valid cert → accepted.
Through the driver:
- another CA's client cert → rc 1;
- the wrong CA for the server → rc 1;
- an unreachable host → rc 1 in about 20 s, with the explanatory message;
- missing secrets → rc 1, naming the three secrets.
- **The local podman path**, `build-image.sh` for all three (rc 0).
- Negative control: a verify stage expecting a different promtool version printed
`DRIFT promtool want 9.9.9` and failed the build.
- Negative control: the `--network=none` check fails when the flag is removed.
- **`buildkit-client-secret.sh`'s sanity checks**, run offline on local certs. They tell apart:
- a good client cert;
- a server cert (not client-auth);
- a mismatched key;
- a cert from another CA.
**Measured live, read-only:**
- node user namespaces, Debian 13.7 and AppArmor (via an Ansible `sysctl`/`cat`);
- DinD's MASQUERADE rules;
- the live ClusterPolicies, namespaced Policies, PSA labels, and cert-manager v1.21.3 with its
ClusterIssuers;
- the act-runner pod labels.
**Renovate (unchanged since the first iteration):**
- the validator with 44.167.0: exit 0;
- `extract` finds all four markers;
- the `kubernetes` duplicates are `skipReason: disabled`;
- 39.264.0 rejects `main`'s config.
**Local gate, CI results:** see the PR body. They are recorded there because this entry is part of
the tree being gated.
## What I did NOT verify
- **platform/buildkit has never run in the cluster.** It can't until it merges: agents may not
`kubectl apply`, and Flux reconciles only `main`. So this PR's own `images` job **cannot build
and is expected to fail at the builder preflight**, which is the designed loud failure. The
following are therefore all unproven in-cluster:
- kubelet admitting the pod;
- containerd honouring `appArmorProfile: Unconfined`;
- the five emptyDirs;
- cert-manager issuing the three certs;
- the CiliumNetworkPolicies.
- **AppArmor `Unconfined` is reasoned, not measured.** My test host runs SELinux, so the AppArmor
requirement comes from upstream's example plus the nodes having AppArmor enabled. If the pod
were to work under `RuntimeDefault` AppArmor, that line could be tightened; I did not test it.
- **The local userns mapping is narrower than the cluster's.** Locally it was 65,535 ids across
two ranges; in the cluster it is the image's own `100000:65536` against real host subuids. That
is the normal case, but it is the cluster's first test.
- **The NetworkPolicies are untested.** In particular:
- that `toEntities: [ingress]` admits the push to the Gateway VIP. The comment explains the
reasoning, but it is unmeasured;
- that world :443/:80 covers every download the builds make. A dnf mirror on another port
would fail a build visibly.
- **No push, anywhere.** The whole publish path is unexercised:
- `REGISTRY_PUSH_TOKEN` reaching buildkitd through the buildx session;
- `push=true`;
- the metadata digest compared with the registry read-back;
- agent-runner FROM `toolbox@sha256:`;
- the immutable-tag guard;
- idempotence.
So are the 15 s Envoy route-timeout risk on blob finalisation, and the Gitea PVC (10Gi, shared
with repositories) absorbing ~2 GB per toolbox publish until retention runs.
- **Node disk under real builds.** The cache emptyDir (6Gi limit, 4 GB GC cap) sits on nodes that
had 7–9 GB free. It is untested under concurrent pressure.
- **Certificate rotation is designed, not exercised.** The server cert renews 30 days early but
buildkitd only reads it at start; the client copy needs the script re-run.
- **`--oci-worker-no-process-sandbox` is an accepted risk, not a mitigated one.** A build step can
ptrace the daemon and read its TLS key and cache. Anyone with the client cert, which means every
PR job, can run build steps.
- **Renovate has still never completed a run.** It is blocked on #80, on `RENOVATE_TOKEN` gaining
`read:package`, and on this branch. The `[email protected]` + 44.x image pairing is
untested.
- **Pods cannot pull these images until PR #61** (the kubelet pull secret) merges.
- **`registry-retention.sh --apply` has never run**; its keep logic has only run against an empty
registry.
@@ -0,0 +1,57 @@
---
date: 2026-10-29
branch: fix/renovate-config-filename
title: Renovate has failed every run since #79 — the workflow still named renovate.json
---
## What changed
**`.gitea/workflows/renovate.yaml`** — `configurationFile: renovate.json` → `renovate.json5`.
PR #79 (`fix/renovate-invalid-config`) renamed the config to `renovate.json5` and did not update
the one line in the workflow that names it. Measured 2026-10-29 from the Gitea Actions log of run
322 (task 513, schedule, 08:15 UTC):
Error: ENOENT: no such file or directory, stat '/workspace/Home-Network/home-network-k3s-impl/renovate.json'
Runs 315–322 (2026-10-22 → 10-29, every one listed) are all `failure` on the `schedule` event,
each failing in about one second. Renovate never started. `RENOVATE_TOKEN` **does** exist as a
repository Actions secret (listed by name via the API; value not read), so the token preflight
passes and this filename is the only blocker observed.
**`ci/lint.sh`** — a new assertion after yamllint: every non-comment `configurationFile:` key in
`.gitea/workflows/*.y*ml` must name a path that exists. This is valid YAML, so yamllint cannot see
it, and the gate stayed green through 11 days of failures.
**Second blocker, same PR:** the workflow did not pin a Renovate version, so
`renovatebot/[email protected]` ran its default, Renovate **39** (`action.yml`:
`renovate-version` `default: '39'`, read at that tag). `renovate.json5` uses `managerFilePatterns`,
which needs >= 40.2.0. Under 39.264.0 the validator rejects the config with five errors. This was
found and reproduced by the image-pipeline worker (PR #81). #79 had validated against `latest`. Pinned
`renovate-version: 44.167.0`, the version the config validates clean against per that worker.
## What I verified
- The root cause, from the run log quoted above, and that `origin/main` has `renovate.json5` and
no `renovate.json` (`git ls-tree`).
- The new guard, both ways, run standalone: on this branch it prints
`workflow configurationFile references resolve`, rc 0. With the line reverted to `renovate.json`
it prints `configurationFile 'renovate.json' does not exist in the repo`, rc 1. My first version
also matched its own explanatory comment in the workflow and failed. That is why the grep is
anchored to non-comment lines.
- Toolbox gate: result and rc are reported in the PR, not predicted here.
## What I did NOT verify
- **The 39.x rejection and the 44.167.0 pass were not reproduced by me.** I confirmed only the
action's default (`'39'`). The validation results are the PR #81 worker's.
- **Renovate has still never completed a run.** This removes the observed blocker. Others may
follow it. The test is a `workflow_dispatch` with `dry_run: true` after merge, and its log is
the first evidence that Renovate authenticates, extracts dependencies, and honours `.json5`.
- **The `enabledManagers: ["regex"]` vs `custom.regex` question** (#73, #77) is still open. The
same dry-run log answers it: look for the regex/custom manager's dependency-extraction count.
- **No alert for "Renovate keeps failing".** The guard prevents this specific cause from recurring.
Nothing notices a scheduled workflow that is red every day. That gap stays open.
- **Sensitive path.** `.gitea/**` and `ci/**` are both in `approval-policy.json`. CI will block
on `approval-policy` until the operator posts `/approve-override` as a review. The local gate
reports that stage as N/A.
+6 -4
View File
@@ -244,10 +244,12 @@ about runtime behaviour in this document is a design claim, not a measurement.
(ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with
the credential decision now being local Gitea accounts with PATs, that means a
`dockerconfigjson` for an account whose token carries `read:package`. Only then does the
digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both
placeholder digests. Expect the build itself to need iteration too: the nodejs module
stream and the npm install have never run against this base. The same two blockers sit in
front of `conformance-runner`, which shares the registry path and has never been pulled.
digest matter. **Superseded 2026-10-29:** the build-and-push is no longer an operator step.
`.gitea/workflows/images.yaml` builds and verifies the image on rootless BuildKit
(`platform/buildkit/`), publishes it on a push to `main`, and Renovate opens the one
PR that replaces both placeholder digests (`decisions/2026-10-29-image-pipeline.md`). The
same two blockers sit in front of `conformance-runner`, which shares the registry path and
has never been pulled.
2. **Create the credentials** per `pm-agent-credentials.sops.yaml.example`, and **verify the
bot's permission is `write`, not `admin`** — that measurement is the control that keeps
the agent out of its own approval gate.
+81 -1
View File
@@ -50,7 +50,10 @@
# default is deliberately EMPTY so a build that forgets it fails loudly instead of
# resolving some other image named `agent-base`.
ARG AGENT_BASE_IMAGE=
FROM ${AGENT_BASE_IMAGE}
# Stage `agent-runner` is the image that ships; stage `verify` (end of file) is a TEST stage built
# FROM it that only runs assertions and is never pushed. build-image.sh and the CI pipeline
# build `--target verify` to prove the image and `--target agent-runner` for the image itself.
FROM ${AGENT_BASE_IMAGE} AS agent-runner
# Re-declared AFTER the FROM: an ARG from before the first FROM is not in scope in the
# build stage, so the provenance layer below would otherwise record an empty base.
@@ -208,3 +211,80 @@ WORKDIR /workspace
# paths above; an ENTRYPOINT here would either be ignored or prepend something the policy
# cannot see. `bash` is the interactive fallback for a human debugging the image locally.
CMD ["/bin/bash"]
# =============================================================================
# verify — a TEST stage, never shipped. Every assertion infra/agent/build-image.sh used to make
# with `run --user 65532 --read-only --tmpfs /workspace --tmpfs /tmp --network none` now runs
# here, at build time, because CI builds on rootless BuildKit where no container runtime exists.
#
# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE, stated so nobody over-reads a green verify:
# * uid 65532 ............ exact: USER is inherited from the agent-runner stage.
# * --network none ....... exact: RUN --network=none, and the first check proves it.
# * --read-only rootfs ... NOT reproducible: a RUN step always has a writable layer. It is
# APPROXIMATED by permission: as uid 65532, every path the process can
# write is inventoried and must lie under /workspace, /tmp or /var/tmp
# (the pod mounts emptyDirs on the first two; /var/tmp is read-only
# there, and TMPDIR points into /workspace so nothing should use it).
# The pod's readOnlyRootFilesystem is enforced by admission
# (platform/kyverno/admission/pm-agent-job-shape.yaml), not by this.
# * --tmpfs /workspace ... approximated by the image's own /workspace (owned 65532), which is
# what the pod's emptyDir replaces.
# The probe script is COPYed into THIS stage only, so it never ships in the image.
# =============================================================================
FROM agent-runner AS verify
COPY probe-validation.sh /opt/verify/probe-validation.sh
RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
echo "--- no network (the build-time stand-in for run --network none) ---"
if getent hosts registry.npmjs.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then
echo "UNEXPECTED: network reachable in the verify stage"; exit 1
fi
echo " confirmed: no network"
echo "--- id ---"; id
echo "--- HOME ---"; echo "HOME=${HOME} CLAUDE_CONFIG_DIR=${CLAUDE_CONFIG_DIR}"
echo "--- baked versions ---"; cat /etc/agent-runner-versions
echo "--- the gate toolchain the worker needs (CLAUDE.md §3) ---"
for t in git python3 bash yamllint check-jsonschema kyverno promtool chainsaw flux sops age kubectl jq diff; do
if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t";
else printf " MISSING %s\n" "$t"; missing=1; fi
done
[ -z "${missing:-}" ] || { echo "the gate cannot run in this image"; exit 1; }
echo "--- python can import the renderer dependency ---"
python3 -c "import yaml; print('PyYAML', yaml.__version__)"
echo "--- claude ---"; claude --version
echo "--- the two entrypoints refuse arguments ---"
# Each must REFUSE. (The old docker-run version wrote `cmd | grep -q ... && echo`, which under
# `set -e` + pipefail could never fail the script; these are real assertions.)
for e in pm-worker pm-dispatch; do
out="$(/usr/local/bin/$e --anything 2>&1 || true)"
grep -q "takes no arguments" <<<"$out" || { echo "UNEXPECTED: $e accepted an argument: $out"; exit 1; }
echo " $e: rejected"
done
echo "--- the dispatcher selftest (branch vocabulary, result envelopes, Job template, persona) ---"
PM_SELFTEST=1 /usr/local/bin/pm-dispatch
echo "--- the entrypoints are root-owned and NOT writable by the runtime uid ---"
for f in /usr/local/bin/pm-worker /usr/local/bin/pm-dispatch /etc/pm-agent/worker-job-template.yaml; do
stat -c " %n %U:%G %a" "$f"
if [ -w "$f" ]; then echo "UNEXPECTED: uid $(id -u) can write $f"; exit 1; fi
done
echo " confirmed: not writable by uid $(id -u)"
echo "--- the root filesystem is not writable by the runtime uid (expected: a failure) ---"
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi
echo " confirmed: / is not writable"
echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---"
stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \
| grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)"
if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf ' %s\n' ${stray} | head -20; exit 1; fi
echo " confirmed: no writable path outside /workspace, /tmp, /var/tmp"
echo "--- the two writable paths really are writable ---"
touch "${HOME}/.probe" /tmp/.probe || { echo "UNEXPECTED: ${HOME} or /tmp not writable"; exit 1; }
echo " confirmed: ${HOME} and /tmp are writable"
echo "--- uid is non-zero ---"
[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; }
echo " confirmed: uid $(id -u), not root"
echo "--- no Anthropic API key is baked into the image (it would outrank the subscription token) ---"
[ -z "${ANTHROPIC_API_KEY:-}${ANTHROPIC_AUTH_TOKEN:-}" ] || { echo "UNEXPECTED: an Anthropic key is baked in"; exit 1; }
echo " confirmed: neither is set"
echo "--- pm-worker's input validation, with the interpreter and script that run in the cluster ---"
PM_WORKER_PATH=/usr/local/bin/pm-worker /bin/bash /opt/verify/probe-validation.sh
echo "AGENT-RUNNER VERIFIED"
VERIFY
+86 -140
View File
@@ -1,26 +1,36 @@
#!/usr/bin/env bash
###############################################################################
# infra/agent/build-image.sh — build the pinned `agent-runner` image and print its digest.
# infra/agent/build-image.sh — build the pinned `agent-runner` image and prove it works.
#
# ./build-image.sh # build + verify + print the digest and push command
# ./build-image.sh # build the verify target, then the image
# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes)
# ./build-image.sh --no-verify # build only (skip the in-image proof)
# ENGINE=docker ./build-image.sh # build with docker instead of podman
# AGENT_BASE_IMAGE=home-k3s-dev:local ./build-image.sh
# # build on a LOCAL toolbox instead of the pinned one
# # (`infra/devcontainer/dev.sh build` makes that tag)
# IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this)
#
# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is not
# an agent's to hold, and the pushed digest is what a fail-closed admission policy then
# trusts. So this builds locally, proves the toolchain and both entrypoints work INSIDE the
# image as the non-root uid with the read-only root filesystem the policy forces, and prints
# the exact push line for a human.
# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on
# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and
# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest
# in the job summary. Renovate then opens the pull request that moves platform/pm-agent/ onto
# that digest. This script is the LOCAL path (podman by default); its proof — the
# Containerfile's `verify` stage — is the same one CI runs.
#
# VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the four pins this image
# adds (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds
# the platform toolchain the base already carries. Nothing in this file invents a version.
# VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the pins this image adds
# (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds the
# platform toolchain the base already carries. Nothing in this file invents a version.
#
# THE DIGEST CAVEAT — read it before pasting a digest into anything. The digest printed here
# is the LOCAL manifest digest. It is normally identical to the registry digest, but that is
# a property of the push rather than a guarantee, so the push command below uses
# `--digestfile`: the digest the REGISTRY returns is the authoritative one, and that is the
# one that goes into platform/pm-agent/configmap.yaml.
# THE BASE. AGENT_BASE_IMAGE in versions.env is the PUBLISHED toolbox, pinned by digest and
# kept current by Renovate. An AGENT_BASE_IMAGE in the environment overrides it — CI passes
# the toolbox it built in the same run, and a developer can pass `home-k3s-dev:local`.
#
# NO `run` STEP. Verification used to `run` the image as uid 65532 with a read-only rootfs.
# CI has no container runtime (rootless BuildKit only), so the assertions moved into the
# Containerfile's `verify` stage and this script builds that target. `--print-build-args`
# prints the pins as KEY=VALUE for the CI pipeline; AGENT_BASE_IMAGE is the caller's choice.
###############################################################################
set -euo pipefail
@@ -29,9 +39,11 @@ REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
ENGINE="${ENGINE:-podman}"
VERIFY=1
PRINT_ARGS=0
for arg in "$@"; do
case "$arg" in
--no-verify) VERIFY=0 ;;
--print-build-args) PRINT_ARGS=1 ;;
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown argument: $arg" >&2; exit 2 ;;
esac
@@ -41,13 +53,14 @@ log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
info() { printf ' %s\n' "$*"; }
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
# --- pins: sourced, never invented -------------------------------------------
AGENT_ENV="${HERE}/versions.env"
[ -f "${AGENT_ENV}" ] || die "missing ${AGENT_ENV} — the pins live there, not here"
BASE_OVERRIDE="${AGENT_BASE_IMAGE:-}"
TAG_OVERRIDE="${IMAGE_TAG:-}"
# shellcheck disable=SC1090
source "${AGENT_ENV}"
[ -z "${BASE_OVERRIDE}" ] || AGENT_BASE_IMAGE="${BASE_OVERRIDE}"
: "${AGENT_BASE_IMAGE:?AGENT_BASE_IMAGE not set in ${AGENT_ENV}}"
: "${NODEJS_MAJOR:?NODEJS_MAJOR not set in ${AGENT_ENV}}"
: "${CLAUDE_CODE_VERSION:?CLAUDE_CODE_VERSION not set in ${AGENT_ENV}}"
@@ -55,16 +68,39 @@ source "${AGENT_ENV}"
: "${AGENT_MODEL:?AGENT_MODEL not set in ${AGENT_ENV}}"
: "${AGENT_IMAGE_REPO:?AGENT_IMAGE_REPO not set in ${AGENT_ENV}}"
# --- the base must already exist ---------------------------------------------
# Deliberately NOT built on demand. The toolbox build is its own reviewed artefact with its
# own pins, and silently building it here would mean this script could change the toolchain
# a worker gates with as a side effect of rebuilding the agent image.
if ! "${ENGINE}" image exists "${AGENT_BASE_IMAGE}" 2>/dev/null; then
die "base image '${AGENT_BASE_IMAGE}' is not present. Build the pinned toolbox first:
${REPO_ROOT}/infra/devcontainer/dev.sh build"
# THE build args, in ONE place; `--print-build-args` hands them to CI (which builds on rootless
# BuildKit and supplies its own base reference). AGENT_BASE_IMAGE is deliberately NOT in this
# list: the caller decides the base.
BUILD_ARGS=(
"NODEJS_MAJOR=${NODEJS_MAJOR}" "CLAUDE_CODE_VERSION=${CLAUDE_CODE_VERSION}"
"CLAUDE_CODE_MIN_VERSION=${CLAUDE_CODE_MIN_VERSION}" "AGENT_MODEL=${AGENT_MODEL}"
)
if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
# --- the base must exist, locally or in the registry ---------------------------
# Deliberately NOT built on demand. The toolbox is its own reviewed artefact with its own
# pins, and silently building it here would mean this script could change the toolchain a
# worker gates with as a side effect of rebuilding the agent image.
case "${AGENT_BASE_IMAGE}" in
*@sha256:0000000000000000000000000000000000000000000000000000000000000000)
die "AGENT_BASE_IMAGE is still the all-zero placeholder in ${AGENT_ENV}: the toolbox has
not been published yet. Either let CI publish it (.gitea/workflows/images.yaml) and
merge Renovate's digest bump, or build on a local toolbox:
${REPO_ROOT}/infra/devcontainer/dev.sh build
AGENT_BASE_IMAGE=home-k3s-dev:local $0" ;;
esac
if ! "${ENGINE}" image inspect "${AGENT_BASE_IMAGE}" >/dev/null 2>&1; then
info "base ${AGENT_BASE_IMAGE} is not local; pulling it"
"${ENGINE}" pull "${AGENT_BASE_IMAGE}" >/dev/null \
|| die "base image '${AGENT_BASE_IMAGE}' is neither local nor pullable (registry login?).
To build on a local toolbox instead:
${REPO_ROOT}/infra/devcontainer/dev.sh build
AGENT_BASE_IMAGE=home-k3s-dev:local $0"
fi
IMAGE_TAG="claude${CLAUDE_CODE_VERSION}-node${NODEJS_MAJOR}"
IMAGE_TAG="${TAG_OVERRIDE:-claude${CLAUDE_CODE_VERSION}-node${NODEJS_MAJOR}}"
IMAGE_REF="${AGENT_IMAGE_REPO}:${IMAGE_TAG}"
log "Building ${IMAGE_REF}"
@@ -75,126 +111,36 @@ info " node ${NODEJS_MAJOR}"
info " claude ${CLAUDE_CODE_VERSION} (floor ${CLAUDE_CODE_MIN_VERSION})"
info " model ${AGENT_MODEL}"
"${ENGINE}" build \
--file "${HERE}/Containerfile" \
--tag "${IMAGE_REF}" \
--build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}" \
--build-arg "NODEJS_MAJOR=${NODEJS_MAJOR}" \
--build-arg "CLAUDE_CODE_VERSION=${CLAUDE_CODE_VERSION}" \
--build-arg "CLAUDE_CODE_MIN_VERSION=${CLAUDE_CODE_MIN_VERSION}" \
--build-arg "AGENT_MODEL=${AGENT_MODEL}" \
"${HERE}"
# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
LABEL_ARGS=()
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
ARG_FLAGS=(--build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}")
for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
# --- verify: the point of the whole exercise ---------------------------------
# A build that merely succeeded is not evidence this image is usable. Prove the tools
# EXECUTE, as the NON-ROOT uid, with a READ-ONLY root filesystem and only the two volumes
# the admission policy admits — the exact shape a worker Job runs in. Anything that works
# here but not there is a 3am discovery.
# A build that merely succeeded is not evidence this image is usable. The proof is the
# Containerfile's `verify` stage: as the NON-ROOT uid, with no network, the toolchain executes,
# both entrypoints refuse arguments, the dispatcher selftest passes, the entrypoints are not
# writable, nothing outside /workspace and /tmp is writable, and pm-worker's input guards
# (probe-validation.sh) refuse every hostile input. It is a build target rather than a
# `run` so the SAME proof runs here and in CI, where no container runtime exists — see the
# stage's header for what a build step can and cannot reproduce of the pod's shape.
if [ "${VERIFY}" -eq 1 ]; then
log "Verifying inside the image (uid 65532, read-only rootfs, workspace+tmp only)"
"${ENGINE}" run --rm \
--user 65532:65532 \
--read-only \
--tmpfs /workspace:rw,mode=1777 \
--tmpfs /tmp:rw,mode=1777 \
--network none \
--entrypoint "" \
"${IMAGE_REF}" \
/bin/bash -euo pipefail -c '
echo "--- id ---"; id
echo "--- HOME ---"; echo "HOME=${HOME} CLAUDE_CONFIG_DIR=${CLAUDE_CONFIG_DIR}"
echo "--- baked versions ---"; cat /etc/agent-runner-versions
echo "--- the gate toolchain the worker needs (CLAUDE.md §3) ---"
for t in git python3 bash yamllint check-jsonschema kyverno promtool chainsaw flux sops age kubectl jq diff; do
if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t";
else printf " MISSING %s\n" "$t"; missing=1; fi
done
[ -z "${missing:-}" ] || { echo "the gate cannot run in this image"; exit 1; }
echo "--- python can import the renderer dependency ---"
python3 -c "import yaml; print(\"PyYAML\", yaml.__version__)"
echo "--- claude ---"; claude --version
echo "--- the two entrypoints refuse arguments ---"
/usr/local/bin/pm-worker --anything 2>&1 | grep -q "takes no arguments" && echo " pm-worker: rejected"
/usr/local/bin/pm-dispatch --anything 2>&1 | grep -q "takes no arguments" && echo " pm-dispatch: rejected"
echo "--- the dispatcher selftest (branch vocabulary, result envelopes, Job template, persona) ---"
PM_SELFTEST=1 /usr/local/bin/pm-dispatch
echo "--- the entrypoints are root-owned and NOT writable by the runtime uid ---"
for f in /usr/local/bin/pm-worker /usr/local/bin/pm-dispatch /etc/pm-agent/worker-job-template.yaml; do
stat -c " %n %U:%G %a" "$f"
if [ -w "$f" ]; then echo "UNEXPECTED: uid $(id -u) can write $f"; exit 1; fi
done
echo " confirmed: not writable by uid $(id -u)"
echo "--- rootfs is read-only (expected: a failure) ---"
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
echo " confirmed: / is read-only"
echo "--- the two writable paths really are writable ---"
touch "${HOME}/.probe" /tmp/.probe && echo " confirmed: ${HOME} and /tmp are writable"
echo "--- uid is non-zero ---"
[ "$(id -u)" -ne 0 ] && echo " confirmed: uid $(id -u), not root"
echo "--- no Anthropic API key is baked into the image (it would outrank the subscription token) ---"
[ -z "${ANTHROPIC_API_KEY:-}${ANTHROPIC_AUTH_TOKEN:-}" ] && echo " confirmed: neither is set"
'
# The worker's input guards, exercised in the image, with the interpreter and the script
# that will actually run in the cluster. The probe is mounted read-only rather than baked
# in: it is a build-time check, not something a worker Job has any business carrying.
log "Probing pm-worker's input validation inside the image"
"${ENGINE}" run --rm \
--user 65532:65532 \
--read-only \
--tmpfs /workspace:rw,mode=1777 \
--tmpfs /tmp:rw,mode=1777 \
--network none \
--volume "${HERE}/probe-validation.sh:/probe-validation.sh:ro,Z" \
--entrypoint "" \
"${IMAGE_REF}" \
/bin/bash /probe-validation.sh
log "Verifying (Containerfile target: verify)"
"${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}"
log "Image verified"
fi
# --- digest ------------------------------------------------------------------
DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target agent-runner \
--file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}"
# --- result ------------------------------------------------------------------
# No digest is printed: a manifest digest does not exist until a registry has the manifest,
# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at
# push time and prints it in the job summary (.gitea/workflows/scripts/images.sh).
SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
log "Built ${IMAGE_REF}"
info "local manifest digest ${DIGEST}"
info "size $(( SIZE / 1024 / 1024 )) MiB"
cat <<EOF
================================================================================
NOT PUSHED. Publishing is an operator action — this script holds no registry
credential and will never acquire one.
OPERATOR: run these, then put the digest from the digestfile into
platform/pm-agent/configmap.yaml (PM_WORKER_IMAGE) via a pull request:
${ENGINE} login gitea.home.keithjackson.dev
${ENGINE} push --digestfile /tmp/agent-runner.digest ${IMAGE_REF}
cat /tmp/agent-runner.digest
# -> sha256:.... THIS is the authoritative digest, not the local one above.
# platform/pm-agent/configmap.yaml
PM_WORKER_IMAGE: "${AGENT_IMAGE_REPO}@\$(cat /tmp/agent-runner.digest)"
WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/pm-agent-job-shape.yaml is a
fail-closed Enforce policy. It DENIES any worker Job whose image is not
${AGENT_IMAGE_REPO}@sha256:<digest> — a tag reference is denied,
and so is any other repository path. The dispatcher also refuses to start if
PM_WORKER_IMAGE carries no digest, so the failure names its own cause.
THE SAME IMAGE RUNS THE DISPATCHER. platform/pm-agent/deployment.yaml pins it too, so a
new digest is a two-line change in one pull request: the Deployment's image and
PM_WORKER_IMAGE. They must be the same digest — the dispatcher and its workers share the
committed Job template and the env-name contract, and a mismatched pair would disagree
about it.
TIGHTEN THE BOUNDARY AFTERWARDS: the policy admits the wildcard
\`${AGENT_IMAGE_REPO}@sha256:*\`, so anyone who can push to that registry path can publish
a different digest. Replacing the wildcard with the LITERAL digest closes that gap; it is a
one-line change, reviewed in a pull request, which is exactly where a change to the
security boundary belongs.
================================================================================
EOF
info "image id $("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
info "size $(( SIZE / 1024 / 1024 )) MiB"
info "Not pushed. CI publishes this image from main (.gitea/workflows/images.yaml); see"
info "platform/pm-agent/README.md for how a new digest reaches the cluster."
+6 -4
View File
@@ -16,13 +16,15 @@
# scrubbed environment and fake credentials, and if a guard ever stopped firing, the case
# would fail here rather than by really cloning something.
#
# Run offline, no cluster, no Gitea, no model. infra/agent/build-image.sh runs it inside the
# built image as well, where the interpreter and the script are the ones that will actually
# run in the cluster.
# Run offline, no cluster, no Gitea, no model. The Containerfile's `verify` stage runs it inside
# the built image as well (as uid 65532, no network), where the interpreter and the script are
# the ones that will actually run in the cluster — locally via build-image.sh and in CI.
###############################################################################
set -uo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# `:-$0` keeps this safe if it is ever fed over stdin (`bash -s`), where BASH_SOURCE is empty
# and `set -u` would abort on it.
HERE="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)"
WORKER="${PM_WORKER_PATH:-${HERE}/pm-worker}"
[ -r "$WORKER" ] || { echo "cannot read ${WORKER}" >&2; exit 2; }
+10 -4
View File
@@ -9,10 +9,16 @@
# truth and, the first time the two drifted, the worker's "green" would stop meaning the
# gate's green. So this file pins ONLY what the toolbox does not carry.
# The toolbox image the runner is layered on. Built by `infra/devcontainer/dev.sh build`
# (tag `home-k3s-dev:local`); infra/agent/build-image.sh asserts it exists rather than
# silently building a different base.
AGENT_BASE_IMAGE=home-k3s-dev:local
# The toolbox image the runner is layered on: the PUBLISHED toolbox, by digest. CI builds and
# publishes it (.gitea/workflows/images.yaml) and Renovate moves this line onto each new digest
# of its `main` tag via the marker below, in a pull request — nobody edits it by hand.
#
# ALL-ZERO UNTIL THE FIRST PUBLISH. The first run of the image pipeline on main sees this
# placeholder, builds and publishes the toolbox first, and builds agent-runner FROM that fresh
# digest. For a LOCAL build on a local toolbox, override it:
# infra/devcontainer/dev.sh build && AGENT_BASE_IMAGE=home-k3s-dev:local infra/agent/build-image.sh
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/toolbox tag=main
AGENT_BASE_IMAGE=gitea.home.keithjackson.dev/home-network/toolbox@sha256:0000000000000000000000000000000000000000000000000000000000000000
# Node.js — the runtime for the Claude Code CLI, installed from the Fedora base's module
# stream. A major version, not a patch: the CLI supports the active LTS line and the
+6
View File
@@ -80,6 +80,12 @@ spec:
# Replaced by the dispatcher with the digest-pinned image. A tag here would be
# DENIED by admission; the dispatcher refuses a non-digest PM_WORKER_IMAGE at
# startup so the error names the real cause.
#
# DELIBERATELY ALL-ZERO, AND DELIBERATELY NOT TRACKED BY RENOVATE. This file is
# baked INTO agent-runner, so it cannot carry agent-runner's own digest: bumping it
# would change the image's inputs, CI would publish a new digest, Renovate would bump
# it again — forever. pm-dispatch overwrites this field from PM_WORKER_IMAGE
# (platform/pm-agent/configmap.yaml, which IS tracked) on every Job it creates.
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000
imagePullPolicy: IfNotPresent
# The pinned entrypoint, with NO args. Args on a pinned command are
+49 -1
View File
@@ -8,7 +8,10 @@
# packages are retained as harmless disk-image utilities. Fedora base is pinned for
# reproducibility.
ARG FEDORA_VERSION=44
FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION}
# Stage `toolbox` is the image that ships. Stage `verify` (end of file) is a TEST stage: it is
# built FROM the finished image and only runs assertions, and it is never pushed. Build with
# `--target toolbox` for the image and `--target verify` to prove it (build-image.sh does both).
FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} AS toolbox
ARG OPENTOFU_VERSION
ARG AGE_VERSION
@@ -95,3 +98,48 @@ RUN { echo "opentofu=${OPENTOFU_VERSION}"; echo "age=${AGE_VERSION}"; echo "sops
WORKDIR /work
CMD ["/bin/bash"]
# =============================================================================
# verify — a TEST stage, never shipped. It runs every assertion against the finished image AT
# BUILD TIME, so CI needs no container runtime to prove the image: the image pipeline builds
# this target on rootless BuildKit (platform/buildkit/), where `docker run` does not exist.
# `RUN --network=none` reproduces the old `run --network none` exactly, and the first check
# below proves it (it fails if any network is reachable).
# =============================================================================
FROM toolbox AS verify
ARG PROMTOOL_VERSION
ARG KUBECTL_VERSION
ARG CHAINSAW_VERSION
ARG KYVERNO_VERSION
ARG SOPS_VERSION
RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
echo "--- no network (the build-time stand-in for run --network none) ---"
if getent hosts registry.fedoraproject.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then
echo "UNEXPECTED: network reachable in the verify stage"; exit 1
fi
echo " confirmed: no network"
echo "--- baked versions ---"; cat /etc/dev-toolchain-versions
echo "--- every tool is on PATH ---"
for t in tofu age age-keygen sops flux kubectl helm kyverno kyverno-json chainsaw yq \
yamllint check-jsonschema promtool ansible-playbook git make jq diff python3 \
wg qrencode; do
if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t";
else printf " MISSING %s\n" "$t"; missing=1; fi
done
[ -z "${missing:-}" ] || { echo "toolbox is missing tools"; exit 1; }
echo "--- version-sensitive tools report their pins ---"
: "${PROMTOOL_VERSION:?}" "${KUBECTL_VERSION:?}" "${CHAINSAW_VERSION:?}" "${KYVERNO_VERSION:?}" "${SOPS_VERSION:?}"
check() { # name pin actual-output
if printf "%s" "$3" | grep -qF "$2"; then printf " ok %-10s %s\n" "$1" "$2";
else printf " DRIFT %-10s want %s, got: %s\n" "$1" "$2" "$(printf "%s" "$3" | head -1)"; drift=1; fi
}
check promtool "${PROMTOOL_VERSION}" "$(promtool --version 2>&1)"
check kubectl "${KUBECTL_VERSION}" "$(kubectl version --client 2>&1)"
check chainsaw "${CHAINSAW_VERSION}" "$(chainsaw version 2>&1)"
check kyverno "${KYVERNO_VERSION}" "$(kyverno version 2>&1)"
check sops "${SOPS_VERSION}" "$(sops --version --disable-version-check 2>&1 || sops --version 2>&1)"
[ -z "${drift:-}" ] || { echo "a tool does not report its pinned version"; exit 1; }
echo "--- python can import what the renderer and the gate need ---"
python3 -c "import yaml; print(' PyYAML', yaml.__version__)"
echo "TOOLBOX VERIFIED"
VERIFY
+18
View File
@@ -34,6 +34,24 @@ cd infra/devcontainer
```
`ENGINE=docker ./dev.sh` to use docker instead of rootless podman.
### The published toolbox
`./dev.sh build` delegates to [`build-image.sh`](build-image.sh). The recipe is the
`Containerfile` plus that script's build-arg list, and the proof is the Containerfile's `verify`
stage (every tool on `PATH`, the version-sensitive ones report their `versions.env` pins, no
network). CI builds the same Containerfile on rootless BuildKit with the args from
`build-image.sh --print-build-args`, and, on a push to `main` that changes
`Containerfile`, `versions.env` or `build-image.sh`, **publishes** the result as
`gitea.home.keithjackson.dev/home-network/toolbox` (tags `sha-<commit>` and `main`) —
see [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). `agent-runner` is
built `FROM` that published digest (`infra/agent/versions.env`, kept current by Renovate).
Nobody pushes this image from a workstation. To use the published one locally instead of
building, after `podman login gitea.home.keithjackson.dev` (a token with `read:package`):
```bash
IMAGE=gitea.home.keithjackson.dev/home-network/toolbox:main ./dev.sh run make -C ci gate
```
## Choosing a cluster — `CLUSTER=dev|prod`
```bash
+84
View File
@@ -0,0 +1,84 @@
#!/usr/bin/env bash
###############################################################################
# infra/devcontainer/build-image.sh — build the pinned dev `toolbox` image and prove it works.
#
# ./build-image.sh # build + verify (tag: home-k3s-dev:local)
# ./build-image.sh --no-verify # build only
# IMAGE=foo:bar ./build-image.sh # build under another local name
# ENGINE=docker ./build-image.sh # build with docker instead of podman
# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes)
#
# ONE BUILD DEFINITION, TWO CALLERS. `dev.sh build` calls this for local development. The CI
# image pipeline (.gitea/workflows/images.yaml -> .gitea/workflows/scripts/images.sh) builds the
# SAME Containerfile on rootless BuildKit (platform/buildkit/) — no container engine exists in a
# CI job — taking its build args from `--print-build-args`, and PUBLISHES the result as
# gitea.home.keithjackson.dev/home-network/toolbox. The Containerfile (including its `verify`
# test stage) and the arg list here are the one recipe both use.
#
# THIS SCRIPT NEVER PUSHES. Publishing is CI's job (images.yaml, on push to main), not a
# workstation's: an image that exists only because someone ran `podman push` from a laptop has
# no build log, no recorded inputs and no verification anyone else can read.
#
# VERSIONS ARE NOT DEFINED HERE. infra/devcontainer/versions.env is the single source of truth.
###############################################################################
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENGINE="${ENGINE:-podman}"
IMAGE="${IMAGE:-home-k3s-dev:local}"
VERIFY=1
PRINT_ARGS=0
for arg in "$@"; do
case "$arg" in
--no-verify) VERIFY=0 ;;
--print-build-args) PRINT_ARGS=1 ;;
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown argument: $arg" >&2; exit 2 ;;
esac
done
log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
# shellcheck disable=SC1091
source "${HERE}/versions.env"
# THE build args, in ONE place. `--print-build-args` hands exactly this list to the CI pipeline
# (.gitea/workflows/scripts/images.sh), which builds on rootless BuildKit rather than calling
# "$ENGINE build" — so CI and a local build cannot drift apart on a pin.
BUILD_ARGS=(
"FEDORA_VERSION=$FEDORA_VERSION" "OPENTOFU_VERSION=$OPENTOFU_VERSION" "AGE_VERSION=$AGE_VERSION"
"SOPS_VERSION=$SOPS_VERSION" "FLUX_VERSION=$FLUX_VERSION" "KUBECTL_VERSION=$KUBECTL_VERSION"
"HELM_VERSION=$HELM_VERSION" "KYVERNO_VERSION=$KYVERNO_VERSION"
"KYVERNO_JSON_VERSION=$KYVERNO_JSON_VERSION" "CHAINSAW_VERSION=$CHAINSAW_VERSION"
"YAMLLINT_VERSION=$YAMLLINT_VERSION" "YQ_VERSION=$YQ_VERSION"
"CHECK_JSONSCHEMA_VERSION=$CHECK_JSONSCHEMA_VERSION" "PROMTOOL_VERSION=$PROMTOOL_VERSION"
)
if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
ARG_FLAGS=()
for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
LABEL_ARGS=()
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
# --- verify --------------------------------------------------------------------
# The assertions live in the Containerfile's `verify` stage (every tool on PATH, the
# version-sensitive ones report their versions.env pins — promtool above all, because
# ci/versions.env asserts a promtool/Prometheus lockstep — and no network), so the SAME proof
# runs here under podman and in CI under rootless BuildKit. A failing assertion fails the build.
if [ "${VERIFY}" -eq 1 ]; then
log "Verifying (Containerfile target: verify)"
"$ENGINE" build "${ARG_FLAGS[@]}" --target verify -f "${HERE}/Containerfile" "${HERE}"
log "Toolbox verified"
fi
log "Building ${IMAGE}"
"$ENGINE" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target toolbox \
-t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}"
log "Built ${IMAGE}"
printf ' image id %s\n' "$("$ENGINE" image inspect --format '{{.Id}}' "$IMAGE")"
+4 -16
View File
@@ -72,22 +72,10 @@ esac
CLUSTER_KUBECONFIG="${REPO_ROOT}/infra/ansible/.cache/${CLUSTER}.kubeconfig"
build() {
"$ENGINE" build \
--build-arg FEDORA_VERSION="$FEDORA_VERSION" \
--build-arg OPENTOFU_VERSION="$OPENTOFU_VERSION" \
--build-arg AGE_VERSION="$AGE_VERSION" \
--build-arg SOPS_VERSION="$SOPS_VERSION" \
--build-arg FLUX_VERSION="$FLUX_VERSION" \
--build-arg KUBECTL_VERSION="$KUBECTL_VERSION" \
--build-arg HELM_VERSION="$HELM_VERSION" \
--build-arg KYVERNO_VERSION="$KYVERNO_VERSION" \
--build-arg KYVERNO_JSON_VERSION="$KYVERNO_JSON_VERSION" \
--build-arg CHAINSAW_VERSION="$CHAINSAW_VERSION" \
--build-arg YAMLLINT_VERSION="$YAMLLINT_VERSION" \
--build-arg YQ_VERSION="$YQ_VERSION" \
--build-arg CHECK_JSONSCHEMA_VERSION="$CHECK_JSONSCHEMA_VERSION" \
--build-arg PROMTOOL_VERSION="$PROMTOOL_VERSION" \
-t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}"
# The build recipe lives in build-image.sh so that CI (.gitea/workflows/images.yaml), which
# builds and PUBLISHES this image, and a local `dev.sh build` cannot drift apart. --no-verify
# keeps the local loop as fast as it always was; CI runs the verification.
IMAGE="$IMAGE" ENGINE="$ENGINE" "${HERE}/build-image.sh" --no-verify
}
run() {
+104
View File
@@ -0,0 +1,104 @@
#!/usr/bin/env bash
# Copy the image builder's CLIENT certificate from the cluster into the three repository Actions
# secrets the image pipeline reads: BUILDKIT_CA, BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY.
#
# ./infra/gitea/buildkit-client-secret.sh # copy and verify
# ./infra/gitea/buildkit-client-secret.sh --check # compare only; changes nothing
#
# RUNNING IT IS AN OPERATOR ACTION. It writes repository secrets.
#
# WHERE THE CERTIFICATE COMES FROM. platform/buildkit/certificates.yaml has cert-manager issue
# `buildkit-client-ci` from a CA that exists only for the builder; buildkitd admits a connection
# only with a certificate signed by that CA. CI jobs hold NO cluster credential (R-SEC-02), so they
# cannot read the Secret themselves: this script is the one bridge, run by a human who has both
# a kube context and the Gitea credential.
#
# WHEN TO RUN IT
# 1. once, after Flux has applied platform/buildkit (Kustomization `buildkit` Ready, and
# `kubectl -n buildkit get certificate buildkit-client-ci` READY=True);
# 2. after every renewal of that certificate (duration 1y, renewed 30 days early). Until then
# the image pipeline fails LOUDLY at its builder preflight — it never skips.
# `--check` tells you whether the copy is stale.
#
# NOTHING SECRET IS PRINTED. The key goes from `kubectl get secret` into the Gitea secrets API
# through shell variables and nowhere else. The script prints only names, sizes and the
# certificate's subject and expiry (public data).
#
# NEEDS: kubectl with a context for the cluster (default k3s-prod), jq, curl, openssl, base64,
# and the Gitea credential `git` already has stored for the host — the same requirements as
# infra/gitea/conformance-trigger-kubeconfig.sh, run from the same place.
set -euo pipefail
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
NS=buildkit; SECRET=buildkit-client-ci; KCTX="${KCTX:-k3s-prod}"
CHECK=0
for a in "$@"; do
case "$a" in
--check) CHECK=1 ;;
-h|--help) sed -n '2,/^set -euo/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown argument: $a" >&2; exit 2 ;;
esac
done
die() { echo "ERROR: $*" >&2; exit 1; }
for t in kubectl jq curl openssl base64 git; do command -v "$t" >/dev/null || die "missing required tool: $t"; done
echo "==> reading Secret ${NS}/${SECRET} (context ${KCTX}; values not shown)"
js="$(kubectl --context "$KCTX" -n "$NS" get secret "$SECRET" -o json)" \
|| die "cannot read ${NS}/${SECRET} — is platform/buildkit deployed and the Certificate Ready?"
CA="$(jq -r '.data["ca.crt"] // empty' <<<"$js" | base64 -d)"
CRT="$(jq -r '.data["tls.crt"] // empty' <<<"$js" | base64 -d)"
KEY="$(jq -r '.data["tls.key"] // empty' <<<"$js" | base64 -d)"
unset js
[ -n "$CA" ] && [ -n "$CRT" ] && [ -n "$KEY" ] || die "${NS}/${SECRET} lacks ca.crt/tls.crt/tls.key"
# Sanity before anything is written: the leaf is a CLIENT cert issued by this CA, and the key
# belongs to it. A wrong pairing would otherwise surface only as a TLS failure in CI.
openssl verify -CAfile <(printf '%s\n' "$CA") <(printf '%s\n' "$CRT") >/dev/null 2>&1 \
|| die "tls.crt does not verify against ca.crt"
openssl x509 -noout -ext extendedKeyUsage <<<"$CRT" 2>/dev/null | grep -q "TLS Web Client Authentication" \
|| die "tls.crt is not a client-auth certificate"
[ "$(openssl x509 -noout -pubkey <<<"$CRT" | openssl sha256)" = "$(openssl pkey -pubout <<<"$KEY" 2>/dev/null | openssl sha256)" ] \
|| die "tls.key does not match tls.crt"
echo " $(openssl x509 -noout -subject <<<"$CRT") / $(openssl x509 -noout -enddate <<<"$CRT")"
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"; unset creds
[ -n "$U" ] && [ -n "$P" ] || die "no stored credential for $HOST"
echo "==> authenticating to Gitea as '${U}' (secret not shown)"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
API="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets"
if [ "$CHECK" -eq 1 ]; then
# Gitea never returns a secret's value, only its name and creation time. "Stale" therefore
# means: the secret was created before the certificate's current notBefore.
names="$(api "$API" | jq -c '[.[]? | {name, created_at}]')"
nb="$(date -u -d "$(openssl x509 -noout -startdate <<<"$CRT" | cut -d= -f2)" +%s)"
rc=0
for n in BUILDKIT_CA BUILDKIT_CLIENT_CERT BUILDKIT_CLIENT_KEY; do
c="$(jq -r --arg n "$n" '.[] | select(.name==$n) | .created_at' <<<"$names")"
if [ -z "$c" ]; then echo " MISSING $n"; rc=1
elif [ "$(date -u -d "$c" +%s)" -lt "$nb" ]; then echo " STALE $n (stored $c, certificate re-issued since)"; rc=1
else echo " current $n (stored $c)"; fi
done
exit "$rc"
fi
put() { # name value
local code
code="$(api -o /dev/null -w '%{http_code}' -X PUT -H 'Content-Type: application/json' \
-d "$(jq -n --arg d "$2" '{data:$d}')" "${API}/$1")"
[ "$code" = 201 ] || [ "$code" = 204 ] || die "PUT $1 failed: HTTP ${code}"
echo " stored $1 (${#2} bytes, value not shown)"
}
echo "==> storing repository Actions secrets"
put BUILDKIT_CA "$CA"
put BUILDKIT_CLIENT_CERT "$CRT"
put BUILDKIT_CLIENT_KEY "$KEY"
unset CA CRT KEY
# VERIFY BY READING BACK (names only — Gitea never returns values). curl -sS exits 0 on an HTTP
# error, which is how set-ci-token.sh once claimed a store that never happened.
names="$(api "$API" | jq -r '[.[]?.name] | join(" ")')"
for n in BUILDKIT_CA BUILDKIT_CLIENT_CERT BUILDKIT_CLIENT_KEY; do
grep -qw "$n" <<<"$names" || die "stored $n but it is NOT listed (${names}) — refusing to claim success"
done
echo "==> stored and VERIFIED present. Re-run the images workflow (workflow_dispatch) to use them."
@@ -252,11 +252,16 @@ step "preconditions — repo side (these block ONLY arming ${VARIABLE}; the secr
# R1: the runner image is built and digest-pinned. Without it the Job cannot run run.sh at
# all (the stock chainsaw image has no bash/python3/kubectl/git) — plan §3 item 1.
# EVALUATED, not grepped: the line is `CONFORMANCE_RUNNER_IMAGE="${..._REPO}@${..._DIGEST}"`, so
# its literal text never contains 64 hex digits and the grep this replaced could never pass. The
# all-zero digest is the "not published yet" placeholder (Renovate replaces it after CI's first
# publish, .gitea/workflows/images.yaml) and must not count as pinned.
VERSIONS="${REPO_ROOT}/conformance/versions.env"
if grep -Eq '^[[:space:]]*CONFORMANCE_RUNNER_IMAGE=.*@sha256:[0-9a-f]{64}' "$VERSIONS" 2>/dev/null; then
pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env"
runner_img="$( (set +u; . "$VERSIONS" >/dev/null 2>&1; printf '%s' "${CONFORMANCE_RUNNER_IMAGE:-}") )"
if [[ "$runner_img" =~ @sha256:[0-9a-f]{64}$ ]] && [[ ! "$runner_img" =~ @sha256:0{64}$ ]]; then
pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env (${runner_img##*@})"
else
fail_arm "conformance/versions.env has no digest-pinned CONFORMANCE_RUNNER_IMAGE (plan §3 item 1) — the Job cannot run ./run.sh"
fail_arm "conformance/versions.env has no PUBLISHED digest for CONFORMANCE_RUNNER_IMAGE (got '${runner_img}') — the image pipeline has not published it yet, or Renovate's bump is unmerged (plan §3 item 1)"
fi
# R2: the Job template is the real thing, not the skeleton: the admitted image AND an
+130
View File
@@ -0,0 +1,130 @@
#!/usr/bin/env bash
# Registry retention for this repo's OWN images (toolbox, agent-runner, conformance-runner):
# keep what is deployed and what is recent, delete the rest. DRY RUN unless --apply.
#
# ./infra/gitea/registry-retention.sh # show what would be deleted
# ./infra/gitea/registry-retention.sh --apply # delete it
# KEEP=15 ./infra/gitea/registry-retention.sh # keep the 15 newest sha- tags (default 10)
#
# RUNNING IT IS AN OPERATOR ACTION. It deletes registry content, so it is not wired into CI and
# no agent runs it with --apply.
#
# WHY A SCRIPT, AND NOT GITEA'S BUILT-IN "CLEANUP RULE" — measured against Gitea 1.24.3's source:
#
# 1. There is NO API for cleanup rules. routers/api/v1/api.go has no such route; the rules live
# only behind the web UI's session+CSRF forms (routers/web/web.go, org settings
# `/packages/rules/add`). A rule therefore cannot be created, read or drift-checked as code —
# which is also why ci/gitea-settings.sh cannot assert one exists.
# 2. A built-in rule would DELETE WHAT IS DEPLOYED. Every reference to these images is
# digest-only (`<repo>@sha256:...`, required by admission). Gitea stores a manifest as a
# file OF THE TAG VERSION it was pushed under (routers/api/packages/container/manifest.go
# createPackageAndVersion), and resolves a by-digest pull by searching those files
# (models/packages/container/search.go). So deleting the tag a digest was pushed under makes
# that digest unpullable — MANIFEST_UNKNOWN, i.e. ImagePullBackOff on the next reschedule.
# The built-in rule keeps "the newest N" and a name pattern, and knows nothing about which
# digest git currently pins. Renovate's bump PRs can lag, so "newest N" is not a guarantee.
#
# WHAT IS KEPT, per image:
# * the `main` and `latest` tags;
# * the KEEP newest `sha-<commit>` tags (by creation time);
# * EVERY tag whose manifest digest is referenced anywhere in the tracked tree at REF
# (default origin/main) — i.e. whatever Flux is deploying, and the toolbox digest
# agent-runner builds FROM;
# * every version whose NAME is a digest (`sha256:...`): those are index children, and Gitea's
# own cleanup handles them (services/packages/container/cleanup.go ShouldBeSkipped).
# Everything else is deleted. Gitea's `cleanup_packages` cron then reclaims the unreferenced
# blobs (default: daily, blobs older than 24h) — space comes back the day after, not instantly.
#
# THE STORAGE IT PROTECTS. Packages live on the SAME 10Gi Longhorn PVC as every repository and
# Gitea's database (catalog/gitea.yaml). The toolbox is ~1.4 GB uncompressed and every main push
# that touches its inputs adds a complete, non-reproducible copy. Unpruned, the registry is what
# fills the volume the git host lives on.
#
# CREDENTIAL: the one `git` has stored for the host (as the other scripts here). It needs
# `read:package` to list and `write:package` to --apply, and the account needs package write on
# the org. The secret is never printed.
set -euo pipefail
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; NS=home-network
IMAGES=(toolbox agent-runner conformance-runner)
KEEP="${KEEP:-10}"
REF="${REF:-origin/main}"
APPLY=0
for a in "$@"; do
case "$a" in
--apply) APPLY=1 ;;
-h|--help) sed -n '2,/^set -euo/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown argument: $a" >&2; exit 2 ;;
esac
done
[[ "$KEEP" =~ ^[0-9]+$ ]] && [ "$KEEP" -ge 1 ] || { echo "KEEP must be a positive integer" >&2; exit 2; }
command -v jq >/dev/null || { echo "jq is required" >&2; exit 1; }
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
git -C "$REPO_ROOT" fetch -q origin main 2>/dev/null || echo " (could not fetch origin/main; using the local ${REF})" >&2
git -C "$REPO_ROOT" rev-parse -q --verify "${REF}^{commit}" >/dev/null || { echo "no such ref: ${REF}" >&2; exit 1; }
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
echo "authenticating as '${U}' (secret not shown)"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
REG_TOKEN="$(api "https://${HOST}/v2/token?service=container_registry" | jq -r '.token // empty')"
[ -n "$REG_TOKEN" ] || { echo "registry token exchange failed (does the credential have read:package?)" >&2; exit 1; }
manifest_digest() { # <image> <tag> -> digest, or nothing
curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "$REG_TOKEN") \
-H 'Accept: application/vnd.oci.image.manifest.v1+json' \
-H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
-H 'Accept: application/vnd.oci.image.index.v1+json' \
-H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
"https://${HOST}/v2/${NS}/$1/manifests/$2" \
| tr -d '\r' | awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' | tail -1
}
TOTAL_DEL=0
for img in "${IMAGES[@]}"; do
echo
echo "==> ${NS}/${img}"
# Digests git pins at REF. The marker convention puts the digest on the line after
# `# renovate-digest: image=<repo>`; the plain `<repo>@sha256:` grep catches any other use.
repo="${HOST}/${NS}/${img}"
pinned="$( { git -C "$REPO_ROOT" grep -h -A1 -F "renovate-digest: image=${repo} " "$REF" -- . 2>/dev/null || true
git -C "$REPO_ROOT" grep -h -F "${repo}@sha256:" "$REF" -- . 2>/dev/null || true; } \
| grep -oE 'sha256:[0-9a-f]{64}' | grep -v 'sha256:0\{64\}' | sort -u || true)"
echo " pinned at ${REF}: $(wc -w <<<"$pinned") digest(s)"
versions="[]"; page=1
while :; do
chunk="$(api "https://${HOST}/api/v1/packages/${OWNER}?type=container&q=${img}&limit=50&page=${page}")"
jq -e 'type=="array"' >/dev/null <<<"$chunk" || { echo " package list failed: $(head -c 200 <<<"$chunk")" >&2; exit 1; }
[ "$(jq length <<<"$chunk")" -gt 0 ] || break
versions="$(jq -c --argjson a "$versions" --arg n "$img" '$a + [ .[] | select(.name==$n) | {version, created_at} ]' <<<"$chunk")"
page=$((page + 1))
done
echo " versions: $(jq length <<<"$versions")"
keep_sha="$(jq -r '[ .[] | select(.version|startswith("sha-")) ] | sort_by(.created_at) | reverse | .[:'"$KEEP"'][] .version' <<<"$versions")"
while IFS= read -r v; do
[ -n "$v" ] || continue
case "$v" in
main|latest) echo " keep ${v} (moving tag)"; continue ;;
sha256:*) echo " keep ${v} (digest-named version; Gitea's cleanup owns those)"; continue ;;
esac
if grep -qxF "$v" <<<"$keep_sha"; then echo " keep ${v} (among the ${KEEP} newest)"; continue; fi
d="$(manifest_digest "$img" "$v")"
if [ -z "$d" ]; then echo " keep ${v} (could not read its digest — never delete what cannot be checked)"; continue; fi
if grep -qxF "$d" <<<"$pinned"; then echo " keep ${v} (${d} is pinned at ${REF})"; continue; fi
if [ "$APPLY" -eq 1 ]; then
code="$(api -o /dev/null -w '%{http_code}' -X DELETE \
"https://${HOST}/api/v1/packages/${OWNER}/container/${img}/$(jq -rn --arg v "$v" '$v|@uri')")"
if [ "$code" = 204 ]; then echo " DELETED ${v} (${d})"; TOTAL_DEL=$((TOTAL_DEL + 1));
else echo " FAILED ${v} HTTP ${code}" >&2; fi
else
echo " would delete ${v} (${d})"; TOTAL_DEL=$((TOTAL_DEL + 1))
fi
done < <(jq -r '.[].version' <<<"$versions")
done
echo
if [ "$APPLY" -eq 1 ]; then echo "deleted ${TOTAL_DEL} version(s). Blobs are reclaimed by Gitea's cleanup_packages cron."
else echo "DRY RUN: ${TOTAL_DEL} version(s) would be deleted. Re-run with --apply to delete them."; fi
+74
View File
@@ -0,0 +1,74 @@
# platform/buildkit — rootless BuildKit, the CI image builder
The builder behind [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). It
builds `toolbox`, `agent-runner` and `conformance-runner`, and on `main` pushes them to the
Gitea registry. Flux reconciles it as the Kustomization `buildkit`
([`clusters/base/buildkit.yaml`](../../clusters/base/buildkit.yaml), `dependsOn: platform`).
## Why it exists (operator decision on PR #81, 2026-10-29)
CI jobs run as containers on the act runner's privileged DinD sidecar, and **cannot reach any
Docker daemon**. act_runner 0.2.11 forces `docker_host: "-"` for a `tcp://` DinD, and gitea/act
strips every bind mount when `valid_volumes` is empty. Measured on PR #81's first run. Giving jobs
that daemon would make every branch push root on the node. So the rule is: **no job may reach a
privileged daemon.** Builds go to this rootless daemon over mTLS instead.
## Shape
| Piece | What it is |
|---|---|
| `deployment.yaml` | `moby/buildkit:v0.34.1-rootless`, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The `securityContext` is commented line by line with how each item was measured |
| `certificates.yaml` | cert-manager: a self-signed CA used only here, plus the server cert `buildkitd-tls` and the CI client cert `buildkit-client-ci` |
| `service.yaml` | ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`, mTLS only. Never on the Gateway |
| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and the Gateway (`ingress` entity) for the registry push |
| `configmap.yaml` | `buildkitd.toml`: cache GC capped at 4 GB, because the cache is an `emptyDir` on node disk |
| `versions.env` | The CI side: pinned `buildx` (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource |
## Admission
**No Kyverno exception is needed.**
- Every ClusterPolicy that is actually applied (the list in
[`platform-config/kustomization.yaml`](../../platform-config/kustomization.yaml)) matches only
namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`. This namespace, like
`gitea-actions`, does not carry that label.
- The job-shape policies (`pm-agent-job-shape`, `conformance-trigger-job-shape`) match only their
own namespaces.
- There is no cluster-wide Pod Security default. This namespace is deliberately left without a
`pod-security` label, because PSS `baseline` forbids the `Unconfined` seccomp and AppArmor
profiles that rootless BuildKit needs.
## Prerequisites (operator, in order)
1. **Unprivileged user namespaces on the nodes.** Already true. Measured 2026-10-29 on all three
Debian 13.7 nodes (kernel 7.1.8): `kernel.unprivileged_userns_clone = 1`,
`user.max_user_namespaces = 127206`, and no Ubuntu-style AppArmor userns restriction.
2. **Merge.** Flux applies this directory. Check that the Kustomization `buildkit` is Ready and
that all three Certificates in namespace `buildkit` are READY.
3. **Copy the client certificate to CI.** Run
[`infra/gitea/buildkit-client-secret.sh`](../../infra/gitea/buildkit-client-secret.sh).
Re-run it yearly, after renewal; `--check` reports whether the copy is stale.
4. **Restart buildkitd after the server certificate renews.** It reads the certificate at start,
and cert-manager renews it 30 days before expiry, so a restart inside that window is needed.
Until step 3 is done, the `images` workflow fails at its builder preflight with a message naming
the cause. It never skips.
## What a build step can do
The daemon runs with `--oci-worker-no-process-sandbox`, so build steps share its PID namespace.
buildkitd warns about this itself at startup.
- **A build step CAN** signal or ptrace the daemon and read what it can read, including this
pod's TLS key and the build cache.
- **A build step CANNOT** reach the node. Everything runs as uid 1000 in a user namespace,
unprivileged, with no host mounts.
- **Who can run one:** anyone holding the client certificate, i.e. every CI job, including pull
requests.
## Local equivalent
The pinned image runs under rootless podman with the same restrictions:
`--cap-drop ALL --cap-add SETUID --cap-add SETGID --security-opt seccomp=unconfined --read-only`,
plus tmpfs at the same five paths. That is how the `securityContext` was measured. Rootless podman
maps only 65,537 uids, so the image's `/etc/subuid` (`user:100000:65536`) must be overridden with
two ranges that fit: `user:1:999` and `user:1001:64535`.
+115
View File
@@ -0,0 +1,115 @@
---
# mTLS for buildkitd, issued by cert-manager from a CA that exists ONLY for this purpose.
#
# buildkit-selfsigned (Issuer) --signs--> buildkit-ca (CA Certificate, 10y)
# buildkit-ca (Issuer) --signs--> buildkitd-tls (server, the daemon presents it)
# --signs--> buildkit-client-ci (client, CI presents it)
#
# buildkitd verifies EVERY client certificate against this CA (--tlscacert), so holding a client
# certificate signed by it is what "may build" means. Nothing else in the cluster trusts this CA,
# and it trusts nothing else.
#
# THE CLIENT CERTIFICATE REACHES CI AS ACTIONS SECRETS, not through the cluster: CI jobs hold no
# cluster credential (R-SEC-02). infra/gitea/buildkit-client-secret.sh copies ca.crt/tls.crt/
# tls.key from the Secret below into the repository secrets BUILDKIT_CA, BUILDKIT_CLIENT_CERT and
# BUILDKIT_CLIENT_KEY without printing them. Re-run it after every renewal (see renewBefore).
#
# WHY A PRIVATE CA AND NOT LET'S ENCRYPT. The platform's TLS rule (R-TLS-01/02, docs/08,
# ADR-0006: ACME DNS-01) is about EXTERNALLY REACHABLE services; ADR-0006 itself says "mTLS between
# workloads is a separate concern … not covered here", and a public CA cannot issue for a
# `*.svc.cluster.local` name or a client-auth certificate anyway. buildkitd is ClusterIP-only. Each
# Certificate below is therefore annotated `homelab.keithjackson.dev/tls-scope: workload-mtls`,
# which the L0 check tls-lifecycle-automated accepts ONLY if the certificate proves it: no
# Gateway serves its Secret, every name is cluster-internal, and the issuer is a namespaced
# selfSigned/CA Issuer whose CA is itself issued here. Issuance and renewal stay unattended.
#
# ROTATION. cert-manager renews both leaves `renewBefore` their expiry and rewrites the Secrets.
# * server: buildkitd reads its certificate at START, so the pod must restart inside the 30-day
# renewal window (a node reboot by kured, a Deployment rollout, or a manual restart).
# * client: the Actions secrets are a COPY; re-run the script. Until then CI fails loudly at
# its TLS preflight (.gitea/workflows/scripts/images.sh), never silently.
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: buildkit-selfsigned
namespace: buildkit
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: buildkit-ca
namespace: buildkit
annotations:
homelab.keithjackson.dev/tls-scope: workload-mtls # see the header: R-TLS-01 is about external TLS
spec:
isCA: true
commonName: buildkit-ca
secretName: buildkit-ca
duration: 87600h # 10 years: rotating the CA means re-issuing everything and re-copying
privateKey:
algorithm: ECDSA
size: 256
issuerRef:
name: buildkit-selfsigned
kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: buildkit-ca
namespace: buildkit
spec:
ca:
secretName: buildkit-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: buildkitd-tls
namespace: buildkit
annotations:
homelab.keithjackson.dev/tls-scope: workload-mtls
spec:
secretName: buildkitd-tls
commonName: buildkitd.buildkit.svc.cluster.local
dnsNames:
- buildkitd.buildkit.svc.cluster.local # what CI dials (platform/buildkit/versions.env)
- buildkitd.buildkit.svc
- buildkitd
duration: 8760h
renewBefore: 720h
usages:
- server auth
- digital signature
privateKey:
algorithm: ECDSA
size: 256
rotationPolicy: Always
issuerRef:
name: buildkit-ca
kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: buildkit-client-ci
namespace: buildkit
annotations:
homelab.keithjackson.dev/tls-scope: workload-mtls
spec:
secretName: buildkit-client-ci
commonName: gitea-actions-image-pipeline
duration: 8760h
renewBefore: 720h
usages:
- client auth
- digital signature
privateKey:
algorithm: ECDSA
size: 256
rotationPolicy: Always
issuerRef:
name: buildkit-ca
kind: Issuer
+24
View File
@@ -0,0 +1,24 @@
---
# buildkitd.toml. Validated against the pinned image (v0.34.1-rootless): `buildctl debug workers
# -v` reports "Maximum used space: 4.29GB / Reserved space: 1.07GB" for the default policy.
#
# WHY 4GB. The cache lives in an emptyDir on the NODE's disk (deployment.yaml), and the nodes had
# 7-9 GB free when this was written. The largest image (toolbox) is ~1.4 GB uncompressed; the
# three images share their Fedora base. The emptyDir's sizeLimit sits above this cap, so GC
# trims the cache well before the kubelet would evict the pod for exceeding it.
apiVersion: v1
kind: ConfigMap
metadata:
name: buildkitd-config
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
data:
buildkitd.toml: |
[worker.oci]
enabled = true
gc = true
maxUsedSpace = "4GB"
reservedSpace = "1GB"
[worker.containerd]
enabled = false
+142
View File
@@ -0,0 +1,142 @@
---
# buildkitd, ROOTLESS. The builder behind .gitea/workflows/images.yaml; see README.md.
#
# THE SECURITY CONTEXT IS THE BOUNDARY, AND EVERY LINE OF IT WAS MEASURED (2026-10-29, the pinned
# image under podman with the same restrictions; decisions/2026-10-29-image-pipeline.md):
# * privileged: false — never. This is the whole point of the operator's decision on PR #81:
# the alternative was handing CI the privileged DinD daemon, i.e. root on the node.
# * runAsUser/Group 1000 — the image's `user`; rootlesskit maps it to uid 0 INSIDE a user
# namespace only. The nodes allow that (measured: kernel.unprivileged_userns_clone = 1,
# user.max_user_namespaces = 127206 on all three Debian 13 nodes).
# * capabilities: drop ALL, add SETUID + SETGID — the image's newuidmap/newgidmap are NOT
# setuid; they carry FILE capabilities, which need SETUID/SETGID in the bounding set.
# Measured: with drop ALL and nothing added, rootlesskit fails "newuidmap ... operation not
# permitted" and the daemon never starts.
# * allowPrivilegeEscalation: true — required for the same reason: no_new_privs (what `false`
# sets) stops a file capability from ever taking effect. It does NOT grant privileges the
# bounding set above does not contain.
# * seccompProfile: Unconfined — measured: with the default seccomp profile the daemon starts
# but every RUN step fails (the runtime cannot create the build step's namespaces/mounts).
# * appArmorProfile: Unconfined — upstream's documented requirement (buildkit
# examples/kubernetes, *.rootless.yaml); the nodes run AppArmor (enabled=Y) and containerd's
# default profile denies mount. NOT measured here: the test host runs SELinux, not AppArmor.
# * readOnlyRootFilesystem: true — measured to work with the five emptyDirs below.
#
# --oci-worker-no-process-sandbox: build steps share the daemon's PID namespace instead of getting
# their own (creating one needs an unmasked /proc, i.e. more privilege). Consequence, stated by
# buildkitd itself at startup: a build step can signal or ptrace the daemon and read what the
# daemon can read — including this pod's TLS key and the build cache. So a build step is trusted
# at the level of "may impersonate this builder", which is the level anyone holding the client
# certificate already has. It can NOT reach the node: everything still runs as uid 1000 in a user
# namespace, unprivileged, with no host mounts.
apiVersion: apps/v1
kind: Deployment
metadata:
name: buildkitd
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
spec:
replicas: 1
strategy:
type: Recreate # one builder, one cache; never two daemons fighting over a node's disk
selector:
matchLabels:
app.kubernetes.io/name: buildkitd
template:
metadata:
labels:
app.kubernetes.io/name: buildkitd
spec:
automountServiceAccountToken: false # the builder needs no Kubernetes API at all
enableServiceLinks: false
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000 # makes the 0440 TLS Secret below readable by uid 1000, and nobody else
containers:
- name: buildkitd
# Pinned by digest (the multi-arch index). The kubernetes manager in renovate.json5
# tracks platform/**, so Renovate proposes bumps; the CI client (buildx, pinned in
# versions.env) talks to any recent daemon.
image: docker.io/moby/buildkit:v0.34.1-rootless@sha256:f68752a5a94acfcdad2d07867b7075601a421638e17fc5f28efc1c814d0c4f89
args:
- --addr
- unix:///run/user/1000/buildkit/buildkitd.sock # local only: the probes below
- --addr
- tcp://0.0.0.0:1234 # CI, mTLS only
- --tlscacert
- /certs/ca.crt # client certs are VERIFIED against this CA: no cert, no build
- --tlscert
- /certs/tls.crt
- --tlskey
- /certs/tls.key
- --config
- /etc/buildkit/buildkitd.toml
- --oci-worker-no-process-sandbox
ports:
- name: buildkit
containerPort: 1234
protocol: TCP
readinessProbe:
exec:
command: ["buildctl", "debug", "workers"]
initialDelaySeconds: 5
periodSeconds: 30
livenessProbe:
exec:
command: ["buildctl", "debug", "workers"]
initialDelaySeconds: 30
periodSeconds: 60
timeoutSeconds: 10
securityContext:
privileged: false
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
allowPrivilegeEscalation: true
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
add: ["SETUID", "SETGID"]
seccompProfile:
type: Unconfined
appArmorProfile:
type: Unconfined
resources:
# Build steps (dnf, npm, pip) run INSIDE this container's cgroup, so the limit covers
# the daemon and the build together.
requests:
cpu: 250m
memory: 512Mi
ephemeral-storage: 2Gi
limits:
memory: 4Gi
volumeMounts:
- { name: certs, mountPath: /certs, readOnly: true }
- { name: config, mountPath: /etc/buildkit, readOnly: true }
- { name: cache, mountPath: /home/user/.local/share/buildkit }
- { name: run, mountPath: /run/user }
- { name: tmpdir, mountPath: /home/user/.local/tmp } # the image's TMPDIR
- { name: tmp, mountPath: /tmp }
- { name: clicfg, mountPath: /home/user/.docker } # buildctl (the probes) writes here
volumes:
- name: certs
secret:
secretName: buildkitd-tls # cert-manager: ca.crt + tls.crt + tls.key
defaultMode: 0440
- name: config
configMap:
name: buildkitd-config
- name: cache
emptyDir:
sizeLimit: 6Gi # GC caps use at 4GB (configmap.yaml); this is the backstop
- name: run
emptyDir: {}
- name: tmpdir
emptyDir: {}
- name: tmp
emptyDir: {}
- name: clicfg
emptyDir: {}
+13
View File
@@ -0,0 +1,13 @@
---
# platform/buildkit — rootless BuildKit, the CI image builder. Reconciled by the Flux
# Kustomization `buildkit` (clusters/base/buildkit.yaml). See README.md.
# versions.env is NOT a resource: it pins the CI-side client (buildx) and the address CI dials.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- certificates.yaml
- configmap.yaml
- deployment.yaml
- service.yaml
- networkpolicy.yaml
+26
View File
@@ -0,0 +1,26 @@
---
# buildkit — the image builder for CI (.gitea/workflows/images.yaml). Operator decision on PR #81
# (2026-10-29): images are built by a ROOTLESS BuildKit daemon reached over mTLS, and NO CI job
# may reach a privileged daemon. See README.md for the whole design.
#
# A NAMESPACE OF ITS OWN, deliberately NOT `gitea-actions`:
# * The act-runner namespace exists to hold a PRIVILEGED DinD sidecar. Putting the builder
# beside it would put a TLS server key next to a privileged container for no benefit.
# * Its NetworkPolicies, Secrets and quota then belong to one component, and the ingress rule
# ("only the act-runner pod may connect") can be written from the outside.
#
# NO `app-namespace` LABEL, like gitea-actions: this is CI plumbing, not a user-facing app. That
# is also why NO Kyverno exception is needed — every applied ClusterPolicy
# (platform-config/kustomization.yaml) matches only namespaces labelled
# homelab.keithjackson.dev/app-namespace: "true", and none of the job-shape policies match here.
#
# NO pod-security.kubernetes.io LABEL, on purpose. Rootless BuildKit needs seccomp AND AppArmor
# `Unconfined` (measured — see deployment.yaml), which PSS `baseline` forbids; there is no
# cluster-wide PSA default here, so an unlabelled namespace admits it. It is NOT privileged and
# the deployment's securityContext is the boundary — read it before relaxing anything.
apiVersion: v1
kind: Namespace
metadata:
name: buildkit
labels:
homelab.keithjackson.dev/ci: buildkit
+88
View File
@@ -0,0 +1,88 @@
---
# Default-deny for the namespace, then two narrow allows.
#
# WHO MAY CONNECT — AND WHY THE ACT-RUNNER POD IS THE RIGHT SOURCE (measured 2026-10-29).
# CI jobs do not run as pods. act_runner starts each job as a container on its DinD sidecar, on a
# docker bridge network INSIDE the act-runner pod's network namespace, and dockerd masquerades it:
# kubectl -n gitea-actions exec deploy/act-runner -c dind -- iptables -t nat -S POSTROUTING
# -A POSTROUTING -s 172.18.0.0/16 ! -o br-c0f13d041ad6 -j MASQUERADE
# -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
# So a job's connection leaves with the act-runner POD's IP and therefore Cilium's identity for
# that pod (namespace gitea-actions, app.kubernetes.io/name=act-runner). That is exactly what the
# ingress rule names. It cannot distinguish one job from another, or a job from act_runner
# itself — per-job authentication is the mTLS client certificate, not this policy. What the
# policy adds is that NOTHING ELSE in the cluster can even open the port.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: buildkit
labels:
homelab.keithjackson.dev/baseline: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress: []
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: buildkitd-allow-ci-ingress
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: buildkitd
ingress:
- fromEndpoints:
- matchLabels:
io.kubernetes.pod.namespace: gitea-actions
app.kubernetes.io/name: act-runner
toPorts:
- ports:
- { port: "1234", protocol: TCP }
---
# WHAT A BUILD NEEDS TO REACH. Build steps download (dnf mirrors, GitHub releases, dl.k8s.io,
# npm, PyPI, Fedora/Docker registries) and the result is pushed to the Gitea registry.
# * world 443/80: the downloads. 80 because Fedora's mirror list includes plain-http mirrors
# (dnf verifies package GPG signatures, not transport).
# * ingress (the Gateway's Envoy, reserved:ingress): the push. gitea.home.keithjackson.dev
# resolves in-cluster to the Gateway VIP (split-horizon CoreDNS), and Cilium hands that
# traffic to the Gateway, whose identity is `ingress` — the same identity gitea's own
# ingress allow admits.
# Nothing else in the cluster: not the API server, not other pods. A build step is code from a
# pull request.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: buildkitd-allow-egress
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: buildkitd
egress:
- toEntities:
- world
toPorts:
- ports:
- { port: "443", protocol: TCP }
- { port: "80", protocol: TCP }
- toEntities:
- ingress
+19
View File
@@ -0,0 +1,19 @@
---
# The address CI dials: tcp://buildkitd.buildkit.svc.cluster.local:1234 (platform/buildkit/
# versions.env), mTLS only. ClusterIP — never exposed on the Gateway.
apiVersion: v1
kind: Service
metadata:
name: buildkitd
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: buildkitd
ports:
- name: buildkit
port: 1234
targetPort: buildkit
protocol: TCP
+15
View File
@@ -0,0 +1,15 @@
# platform/buildkit/versions.env — the CI side of the builder. Sourced by
# .gitea/workflows/scripts/images.sh. NOT a Kubernetes resource (kustomization.yaml omits it).
#
# The CLIENT. CI drives the daemon with `buildx` (remote driver) rather than bare `buildctl`
# because the image pipeline needs `bake`'s `target:` contexts: on a pull request agent-runner is
# built FROM the toolbox built in the same run, which never exists in any registry.
# Pinned with the sha256 from the release's own checksums.txt
# (https://github.com/docker/buildx/releases/download/v0.38.1/checksums.txt); the pipeline
# refuses a download that does not match. Bump BOTH lines together.
BUILDX_VERSION=0.38.1
BUILDX_SHA256=e25d5cd675092bb5d1d03632fdc46cdf3fede7b514dc5f023a105e00d3ce5ad4
# The DAEMON's address and the name its certificate carries (platform/buildkit/certificates.yaml).
BUILDKIT_HOST=tcp://buildkitd.buildkit.svc.cluster.local:1234
BUILDKIT_SERVERNAME=buildkitd.buildkit.svc.cluster.local
+14 -7
View File
@@ -45,11 +45,18 @@ waive the sensitive-path human review on its own pull request.
Both are deliberate, and both fail **visibly** rather than silently:
1. **The image does not exist.** `platform/pm-agent/deployment.yaml` and
`PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder. Build it with
`infra/agent/build-image.sh` (which verifies it and refuses to push), push it as the
operator, then update **both** places with the digest `podman push --digestfile` returned,
in one pull request. Until then: `ImagePullBackOff`.
1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
`PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder until the first
publish. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
(`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that
`infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to
`gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
Renovate then follows the `main` tag through the `# renovate-digest:` marker above **both**
lines and moves them together in one pull request. Nobody builds-and-pushes this image from
a workstation, and nobody pastes a digest. Until the first publish, and until that PR
merges: `ImagePullBackOff`. The one operator prerequisite on the publishing side is the
`REGISTRY_PUSH_TOKEN` Actions secret (see the header of `.gitea/workflows/images.yaml`);
without it the publish job fails loudly rather than skipping.
**Two things have to be true before any digest will pull, and neither is declared in this
repository.** Both were measured after PR #57 merged — this service is the first workload
@@ -78,8 +85,8 @@ Both are deliberate, and both fail **visibly** rather than silently:
account whose token carries `read:package`, referenced from the Deployment and the
committed worker Job template — not built yet, because minting that token is yours.
The same two apply to `conformance-runner`: same registry path, same push-then-pin model,
never pulled either.
The same two apply to `conformance-runner`: same registry path, same publish-then-Renovate
model, never pulled either.
2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the
repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off —
which Gitea account to create, which scopes, and why it must *not* be an administrator —
+7 -6
View File
@@ -52,12 +52,13 @@ data:
# contract. A tag here is refused at startup (and would be DENIED by admission anyway).
#
# PLACEHOLDER UNTIL THE IMAGE IS PUBLISHED. The all-zero digest below is deliberately
# impossible: `infra/agent/build-image.sh` builds and verifies the image, the OPERATOR
# pushes it, and the authoritative digest from `--digestfile` replaces this value in a
# reviewed pull request (the same two-step the conformance-runner image follows, and for
# the same reason: a registry credential is not an agent's to hold). Until then the
# dispatcher starts, fails to pull, and says so — which is the visible failure, not a
# silent one.
# impossible. CI builds, verifies and — on main — publishes agent-runner
# (.gitea/workflows/images.yaml), and Renovate replaces this value with the digest the
# REGISTRY returned for the image's `main` tag, in a pull request that also moves the
# Deployment's image (same marker, same branch). Nobody pushes this image from a workstation.
# Until the first publish the dispatcher starts, fails to pull, and says so — which is the
# visible failure, not a silent one.
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000"
# --- the model -------------------------------------------------------------
+5 -2
View File
@@ -66,9 +66,12 @@ spec:
- name: dispatch
# THE SAME IMAGE THE WORKERS RUN, at the same digest. They share the committed Job
# template and the env-name contract baked into it, so a mismatched pair would
# disagree about the shape the admission policy enforces. Replaced together, in one
# pull request, with the digest `infra/agent/build-image.sh` printed — the all-zero
# disagree about the shape the admission policy enforces. Replaced together, in ONE
# Renovate pull request: both lines carry the same marker, so Renovate treats them as
# one dependency and bumps them in the same branch. The digest comes from CI's publish
# of agent-runner's `main` tag (.gitea/workflows/images.yaml) — the all-zero
# placeholder below is impossible on purpose (see configmap.yaml PM_WORKER_IMAGE).
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000
imagePullPolicy: IfNotPresent
command:
-99
View File
@@ -1,99 +0,0 @@
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17 RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags. It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009 split that gave image tags to Flux image-automation is formally closed — see .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write anyway; and enabling it would need an in-cluster git WRITE credential for the repo that defines the cluster). image-automation has been commented out at platform/kustomization.yaml:52 since it was authored and the cluster holds zero ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down. CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag` line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER: .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.",
"extends": [
"config:recommended",
":dependencyDashboard",
":semanticCommits",
"helpers:pinGitHubActionDigests"
],
"timezone": "America/New_York",
"schedule": ["after 3am and before 6am"],
"prConcurrentLimit": 5,
"prHourlyLimit": 2,
"commitMessagePrefix": "chore(deps):",
"labels": ["dependencies", "automated-update"],
"ignoreUnstable": true,
"respectLatest": true,
"enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
"flux": {
"managerFilePatterns": ["/(^|/)platform/.+\\.ya?ml$/", "/(^|/)apps/.+\\.ya?ml$/"]
},
"kustomize": {
"managerFilePatterns": ["/(^|/)kustomization\\.ya?ml$/"]
},
"_kubernetes_comment": "ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes manifests, and it is NOT auto-detected — it only looks at files matched here, which is why this block is as broad as it is. Scope covers every place this repo writes a container image: platform components, the rendered app waves, the ServiceDefinition catalog that generates them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the rendered output is reverted by the next render and would also fail ci/render-golden.sh. Renovate will raise BOTH, and both must land together.",
"kubernetes": {
"managerFilePatterns": [
"/(^|/)platform/.+\\.ya?ml$/",
"/(^|/)platform-config/.+\\.ya?ml$/",
"/(^|/)apps/.+\\.ya?ml$/",
"/(^|/)catalog/.+\\.ya?ml$/",
"/(^|/)runbooks/serving/.+\\.ya?ml$/",
"/(^|/)infra/agent/.+\\.ya?ml$/"
]
},
"customManagers": [
{
"customType": "regex",
"description": "Inline `# renovate: datasource=... depName=...` markers on pinned URLs / version literals (e.g. platform/kubevirt/kustomization.yaml release URLs, ci/versions.env, infra/devcontainer/versions.env).",
"managerFilePatterns": [
"/(^|/)kustomization\\.ya?ml$/",
"/(^|/)versions\\.env$/"
],
"matchStrings": [
"#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n.*?(?<currentValue>v?[0-9]+\\.[0-9]+\\.[0-9]+)"
],
"versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
}
],
"packageRules": [
{
"description": "Auto-merge the allowed low-risk classes on green CI — no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.",
"matchUpdateTypes": ["minor", "patch", "digest"],
"automerge": true,
"automergeType": "pr",
"platformAutomerge": true
},
{
"description": "MAJOR / potentially-breaking bumps require human review (docs/11 safety rail). Never auto-merged.",
"matchUpdateTypes": ["major"],
"automerge": false,
"addLabels": ["needs-review", "breaking"]
},
{
"description": "Keep the KubeVirt + CDI operator pair on a compatible track — grouped so they bump together (platform/kubevirt/kustomization.yaml).",
"matchDepNames": ["kubevirt/kubevirt", "kubevirt/containerized-data-importer"],
"groupName": "kubevirt-cdi"
},
{
"description": "Pinned tool versions (ci/versions.env, infra/devcontainer/versions.env) are a deliberate change surface — label but do not auto-merge; a version bump there changes CI reproducibility.",
"matchFileNames": ["ci/versions.env", "infra/devcontainer/versions.env"],
"automerge": false,
"addLabels": ["ci-tooling"]
},
{
"description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check — it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
"matchPackageNames": [
"kured",
"longhorn",
"cilium",
"cloudnative-pg",
"kube-prometheus-stack"
],
"automerge": false,
"addLabels": ["needs-review", "blast-radius"]
}
],
"_unmanageable_comment": "WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest, lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete version before any automation can maintain them, and until then they drift on every pull with nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it has been told not to leave; raising those ceilings is deliberate per-component work and is NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.",
"ignorePaths": [
"**/*.example",
"**/*.sops.yaml",
"conformance/**",
"platform/image-automation/per-service/**",
"platform/kyverno/tests/**"
],
"_ignorePaths_comment": "platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest` and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed shape, not software to keep current. `conformance/**` was already ignored for the same reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise have opened.",
"_docker_comment": "The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation over a running tag. That controller was never enabled and never will be here (see _comment), so the only thing those two flags were achieving was leaving every plain `image:` tag in the repo unmanaged. `docker` is left out of enabledManagers because this repo contains no Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.",
"docker": { "enabled": true }
}
+272
View File
@@ -0,0 +1,272 @@
// =================================================================================================
// Renovate configuration — JSON5, so the rationale can live NEXT TO what it explains.
// =================================================================================================
// THIS FILE WAS `renovate.json` AND IT WAS INVALID. It carried five `_comment` / `_*_comment`
// pseudo-comment keys, because JSON has no comments. Renovate REJECTS unknown top-level options, so
// the whole config failed validation and the first manual run of the workflow ERRORED OUT. Proven
// with the official validator (`ghcr.io/renovatebot/renovate renovate-config-validator`):
//
// ERROR: Found errors in configuration
// Invalid configuration option: _comment
// Invalid configuration option: _docker_comment
// Invalid configuration option: _ignorePaths_comment
// Invalid configuration option: _kubernetes_comment
// Invalid configuration option: _unmanageable_comment
//
// Renovate discovers `renovate.json5` natively (no RENOVATE_CONFIG_FILE needed, and nothing in
// .gitea/ or ci/ pins the old filename — only prose in comments), and JSON5 permits real `//`
// comments. So the fix keeps every word of the original rationale, positioned, instead of
// flattening it into a `description` string.
//
// VALIDATE BEFORE COMMITTING — it is one command and it is the only thing that catches this class:
// podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \
// ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json5
//
// CORRECTION (2026-10-29): THE VALIDATOR ABOVE WAS NOT THE RENOVATE THE WORKFLOW RUNS. The bare
// `ghcr.io/renovatebot/renovate` tag is `latest` (44.167.0 when this was measured), while
// renovatebot/[email protected] defaults `renovate-version` to `39` (its action.yml), i.e.
// 39.264.0. Validated against THAT, this file failed with five errors — every
// `managerFilePatterns` key ("Custom Manager contains disallowed fields", "Invalid configuration
// option: flux.managerFilePatterns", ...), because 39.x only knows `fileMatch`; the rename landed
// in 40.2.0. So even after the workflow found this file, every run would have stopped at
// CONFIG_VALIDATION. renovate.yaml now pins `renovate-version` to the exact release this file is
// validated against. VALIDATE WITH THAT SAME TAG:
// podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \
// ghcr.io/renovatebot/renovate:<renovate-version from renovate.yaml> \
// renovate-config-validator /tmp/renovate.json5
// (Not `--strict`: that also fails on a pending migration of the pre-existing top-level
// `"docker": {"enabled": true}` block at the bottom, which is a warning, not an error.)
//
// `regex` vs `custom.regex` in enabledManagers, RESOLVED FROM SOURCE (Renovate 39.264.0 and
// later): both work. lib/config/migrations/custom/enabled-managers-migration.ts rewrites
// `regex` -> `custom.regex`, and that migration runs on every real repository run
// (lib/workers/repository/init/merge.ts -> migrateAndValidate), not only in the validator;
// lib/modules/manager/index.ts getEnabledManagersList accepts either spelling. `custom.regex` is
// the canonical name (introduced in 37.51.1, renovatebot/renovate#24079), so it is used below and
// nothing depends on the migration any more.
{ "$schema": "https://docs.renovatebot.com/renovate-schema.json",
// Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17
// RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags.
// It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009
// split that gave image tags to Flux image-automation is formally closed — see
// .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than
// opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with
// user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write
// anyway; and enabling it would need an in-cluster git WRITE credential for the repo that
// defines the cluster). image-automation has been commented out at
// platform/kustomization.yaml:52 since it was authored and the cluster holds zero
// ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down.
// CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid
// fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag`
// line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth
// sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER:
// .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.
"extends": ["config:recommended", ":dependencyDashboard", ":semanticCommits", "helpers:pinGitHubActionDigests"],
"timezone": "America/New_York",
"schedule": ["after 3am and before 6am"],
"prConcurrentLimit": 5,
"prHourlyLimit": 2,
"commitMessagePrefix": "chore(deps):",
"labels": ["dependencies", "automated-update"],
"ignoreUnstable": true,
"respectLatest": true,
"enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "custom.regex", "github-actions"],
"flux": {
"managerFilePatterns": [
"/(^|/)platform/.+\\.ya?ml$/",
"/(^|/)apps/.+\\.ya?ml$/"
]
},
"kustomize": {
"managerFilePatterns": [
"/(^|/)kustomization\\.ya?ml$/"
]
},
// ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes
// manifests, and it is NOT auto-detected — it only looks at files matched here, which is why
// this block is as broad as it is. Scope covers every place this repo writes a container image:
// platform components, the rendered app waves, the ServiceDefinition catalog that generates
// them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as
// `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the
// rendered output is reverted by the next render and would also fail ci/render-golden.sh.
// Renovate will raise BOTH, and both must land together.
"kubernetes": {
"managerFilePatterns": [
"/(^|/)platform/.+\\.ya?ml$/",
"/(^|/)platform-config/.+\\.ya?ml$/",
"/(^|/)apps/.+\\.ya?ml$/",
"/(^|/)catalog/.+\\.ya?ml$/",
"/(^|/)runbooks/serving/.+\\.ya?ml$/",
"/(^|/)infra/agent/.+\\.ya?ml$/"
]
},
"customManagers": [
{
"customType": "regex",
"description": "Inline `# renovate: datasource=... depName=...` markers on pinned URLs / version literals (e.g. platform/kubevirt/kustomization.yaml release URLs, ci/versions.env, infra/devcontainer/versions.env).",
"managerFilePatterns": [
"/(^|/)kustomization\\.ya?ml$/",
"/(^|/)versions\\.env$/"
],
"matchStrings": [
"#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n.*?(?<currentValue>v?[0-9]+\\.[0-9]+\\.[0-9]+)"
],
"versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
},
// ADDED 2026-10-29 — THIS REPO'S OWN IMAGES (toolbox, agent-runner, conformance-runner).
// CI publishes them (.gitea/workflows/images.yaml) under an immutable `sha-<commit>` tag and a
// moving `main` tag, and never writes to git. Every reference to them is DIGEST-ONLY
// (`<repo>@sha256:<hex>`, no tag), because the admission policies
// (platform/kyverno/admission/pm-agent-job-shape.yaml, conformance-trigger-job-shape.yaml)
// admit exactly that form. A digest alone gives Renovate nothing to follow, so the tag to
// follow rides in a marker comment on the line ABOVE the reference:
//
// # renovate-digest: image=<registry>/<namespace>/<image> tag=main
// <anything>sha256:<64 hex>
//
// `main` is not a version, so Renovate skips the release lookup and proposes a `digest`
// update whenever the tag's digest moves (lib/workers/repository/process/lookup/index.ts:
// an unversioned currentValue WITH a currentDigest yields updateType `digest`). The
// `sha256:` prefix is INSIDE currentDigest on purpose: Renovate replaces that capture
// verbatim with the new digest, which carries the prefix.
//
// DELIBERATELY NOT TRACKED: infra/agent/worker-job-template.yaml. It is baked INTO
// agent-runner, so bumping it would rebuild agent-runner and loop forever; pm-dispatch
// overwrites that field from PM_WORKER_IMAGE at runtime anyway.
// REGISTRY AUTH: the Gitea registry answers 401 to anonymous reads, and the platform token is
// scoped to hostType `gitea`, which a docker lookup never matches
// (lib/util/host-rules.ts find). renovate.yaml supplies a hostType `docker` rule.
{
"customType": "regex",
"description": "This repo's own CI-published images, pinned by digest, following the tag named in a `# renovate-digest:` marker on the previous line.",
"managerFilePatterns": [
"/(^|/)versions\\.env$/",
"/(^|/)platform/pm-agent/.+\\.ya?ml$/"
],
"matchStrings": [
"#\\s*renovate-digest:\\s*image=(?<depName>\\S+)\\s+tag=(?<currentValue>\\S+)\\s*\\n[^\\n]*?(?<currentDigest>sha256:[a-f0-9]{64})"
],
"datasourceTemplate": "docker",
"versioningTemplate": "docker"
}
],
"packageRules": [
{
"description": "Auto-merge the allowed low-risk classes on green CI \u2014 no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.",
"matchUpdateTypes": [
"minor",
"patch",
"digest"
],
"automerge": true,
"automergeType": "pr",
"platformAutomerge": true
},
{
"description": "MAJOR / potentially-breaking bumps require human review (docs/11 safety rail). Never auto-merged.",
"matchUpdateTypes": [
"major"
],
"automerge": false,
"addLabels": [
"needs-review",
"breaking"
]
},
{
"description": "Keep the KubeVirt + CDI operator pair on a compatible track \u2014 grouped so they bump together (platform/kubevirt/kustomization.yaml).",
"matchDepNames": [
"kubevirt/kubevirt",
"kubevirt/containerized-data-importer"
],
"groupName": "kubevirt-cdi"
},
{
"description": "Pinned tool versions (ci/versions.env, infra/devcontainer/versions.env) are a deliberate change surface \u2014 label but do not auto-merge; a version bump there changes CI reproducibility.",
"matchFileNames": [
"ci/versions.env",
"infra/devcontainer/versions.env"
],
"automerge": false,
"addLabels": [
"ci-tooling"
]
},
{
"description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check \u2014 it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
"matchPackageNames": [
"kured",
"longhorn",
"cilium",
"cloudnative-pg",
"kube-prometheus-stack"
],
"automerge": false,
"addLabels": [
"needs-review",
"blast-radius"
]
},
{
"description": "ADDED 2026-10-29. This repo's own images are followed ONLY through the `# renovate-digest:` markers above. The `kubernetes` manager also sees platform/pm-agent/deployment.yaml and infra/agent/worker-job-template.yaml, where the image has a digest and no tag; left on, it would look those up as `latest` (a tag CI never pushes) and, for the template, re-open the self-referential loop the custom manager deliberately avoids.",
"matchManagers": ["kubernetes"],
"matchPackageNames": [
"gitea.home.keithjackson.dev/home-network/toolbox",
"gitea.home.keithjackson.dev/home-network/agent-runner",
"gitea.home.keithjackson.dev/home-network/conformance-runner"
],
"enabled": false
},
{
"description": "ADDED 2026-10-29. Digest bumps for this repo's own images: the image was built and verified by CI before it was published, so these auto-merge on a green gate like any digest (rule 1). Labelled so they are easy to find. NOTE: the conformance-runner bump edits conformance/versions.env, a sensitive path (infra/gitea/approval-policy.json), so it will wait for an administrator's /approve-override regardless of this rule.",
"matchManagers": ["custom.regex"],
"matchPackageNames": [
"gitea.home.keithjackson.dev/home-network/toolbox",
"gitea.home.keithjackson.dev/home-network/agent-runner",
"gitea.home.keithjackson.dev/home-network/conformance-runner"
],
"addLabels": ["own-image"]
}
],
// WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A
// FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it
// and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest,
// lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak
// (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete
// version before any automation can maintain them, and until then they drift on every pull with
// nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12,
// e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it
// has been told not to leave; raising those ceilings is deliberate per-component work and is
// NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.
// platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest`
// and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed
// shape, not software to keep current. `conformance/**` was already ignored for the same
// reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise
// have opened.
// conformance/** NARROWED 2026-10-29 to its subdirectories: ignorePaths applies to custom
// managers too (lib/workers/repository/extract/file-match.ts getMatchingFiles ->
// filterIgnoredFiles), so the blanket glob also hid conformance/versions.env, which now carries
// the conformance-runner digest Renovate must maintain. Everything else under conformance/
// stays ignored, for the original reason (test fixtures with deliberately fixed shapes).
"ignorePaths": [
"**/*.example",
"**/*.sops.yaml",
"conformance/config/**",
"conformance/incluster/**",
"conformance/lib/**",
"conformance/requirements/**",
"conformance/runs/**",
"conformance/tests/**",
"platform/image-automation/per-service/**",
"platform/kyverno/tests/**"
],
// The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were
// BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation
// over a running tag. That controller was never enabled and never will be here (see _comment),
// so the only thing those two flags were achieving was leaving every plain `image:` tag in the
// repo unmanaged. `docker` is left out of enabledManagers because this repo contains no
// Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.
"docker": {
"enabled": true
}
}
+2 -2
View File
@@ -67,10 +67,10 @@ Work through these in order; the first three are by far the most likely.
| Symptom in the log | Cause | Fix |
|---|---|---|
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first |
| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been published; the committed digest is the deliberate placeholder | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | restore the digest form; the digest comes from CI's publish (`.gitea/workflows/images.yaml` job summary) and Renovate's bump PR, never from a hand push |
| `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
| `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |
| `at the worker cap` / `worker limit reached` | the per-cycle cap, draining a backlog one issue at a time | expected; it will continue next cycle |