Compare commits

...
Author SHA1 Message Date
jakman85 74ba90c0b3 Merge branch 'main' into fix/renovate-hold-auth-and-self
gate / gate (pull_request) Successful in 2m55s
gate / conformance (pull_request) Has been skipped
2026-10-30 08:14:15 +00:00
jakman85 fb45d7441b Merge pull request 'feat(renovate): track the Keycloak operator (the real Keycloak version), never auto-merge it' (#88) from feat/renovate-track-keycloak-operator into main
gate / gate (push) Successful in 2m54s
gate / conformance (push) Has been skipped
renovate / renovate (push) Successful in 50s
Reviewed-on: #88
2026-10-30 08:14:02 +00:00
jakman85andClaude Opus 5.5 8ed4e30201 docs(decisions): handoff for feat/renovate-track-keycloak-operator
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 03:06:56 -05:00
jakman85andClaude Opus 5.5 24db269ea8 feat(renovate): track the Keycloak operator (the real Keycloak version), never auto-merge it
The Keycloak CR has no spec.image, so the operator's RELATED_IMAGE_KEYCLOAK picks the
server image; the three keycloak-k8s-resources release URLs in
platform/keycloak/kustomization.yaml are therefore THE Keycloak version, and Renovate
could not see them (no `# renovate:` marker). They now carry markers under one depName
(one branch, groupName keycloak-operator), with a packageRule: automerge false, labels
needs-review + blast-radius; majors still wait on the dashboard.

The untagged quay.io/keycloak/keycloak in catalog/keycloak.yaml renders
apps/wave-1/keycloak.yaml, which no kustomization lists. It is pinned to the operator's
26.0.6 as documentation, re-rendered, and Renovate is disabled for that image so it
cannot raise PRs against a file nothing deploys. The render is kept (not deleted): it is
the fixture for apps/wave-1/tests/app-slis.test.yaml's keycloak SLI rule tests.

README: an "Upgrading Keycloak" section — an operator bump must be checked against
passkey-reconciler.yaml's 26.0.6 Admin API assumptions.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 03:02:25 -05:00
jakman85andClaude Opus 5.5 df96be98bf fix(renovate): never auto-merge the oauth2-proxy sidecar or Renovate's own version
gate / gate (pull_request) Successful in 2m51s
gate / conformance (pull_request) Has been skipped
Operator decision. Both are held at every update type and labelled needs-review:

- quay.io/oauth2-proxy/oauth2-proxy (render.py constant, rendered sidecars, crafty). Since v7.13.0
  oauth2-proxy validates refreshed OIDC sessions against the access token, every sidecar uses
  --cookie-refresh, and that path is untested against this Keycloak.
- ghcr.io/renovatebot/renovate (and renovatebot/github-action if ever reintroduced). Nothing
  re-validates renovate.json5 against the new version before it takes effect.

The validator passes on 44.167.0. Evaluated with Renovate's own packageRules engine: exactly 2 of 29
branch rows change (automerge true -> false, +needs-review); the other 27 are identical.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 02:54:51 -05:00
jakman85 b4ff2879be Merge pull request 'fix(renovate): edit catalog sources, track CNPG imageName, honour update.range, guard real-run auto-merges' (#86) from fix/renovate-run-hygiene into main
gate / gate (push) Successful in 2m53s
gate / conformance (push) Has been skipped
Reviewed-on: #86
2026-10-30 07:43:28 +00:00
jakman85andClaude Opus 5.5 d9571fc6d8 fix(renovate): edit catalog sources, track CNPG imageName, honour update.range, guard real-run auto-merges
gate / gate (pull_request) Successful in 2m55s
gate / conformance (pull_request) Has been skipped
Audit of the first completed dry runs (353/354) plus a debug-level local reproduction:

- App-image updates edited only the generated apps/wave-*. A ServiceDefinition has no apiVersion,
  so the kubernetes manager extracts nothing from catalog/, and every such PR would fail
  render-golden (reproduced: GOLDEN DRIFT). New regex managers for catalog source.image and
  render.py OAUTH2_PROXY_IMAGE put the source and generated edits in one branch (render-golden
  rc 0, reproduced).
- CNPG Cluster imageName was untracked (3 Clusters), while the grant Job was tracked. New manager.
- allowedVersions mirror catalog update.range: this drops gitea 28.x, qbittorrent 20.x, radarr 6,
  nextcloud 35 and postgresql 18.
- Never auto-merge: gitea, act_runner, postgresql (blast radius), flux range widening, python
  minors. Majors wait for a Dependency Dashboard tick.
- velero plugin RC: semver versioning (v1.14.3-rc.2 -> v1.14.4 now proposed).
- flux docker duplicates disabled (the "Cannot find replaceString" line). Two false comments
  corrected.

Local dry run: regex deps 7 -> 20, branches 33 -> 29. The validator passes on 44.167.0.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 02:35:33 -05:00
jakman85 593d0fad15 Merge pull request 'fix(renovate): document and probe the five token scopes Renovate needs; lint daemon-bound actions' (#85) from fix/renovate-auth into main
gate / gate (push) Successful in 2m54s
gate / conformance (push) Has been skipped
Reviewed-on: #85
Reviewed-by: Keith Jackson <[email protected]>
2026-10-30 07:10:02 +00:00
jakman85andClaude Opus 5.5 0c6ee4921d fix(renovate): document and probe the five token scopes Renovate needs; lint daemon-bound actions
gate / gate (pull_request) Successful in 2m54s
gate / conformance (pull_request) Has been skipped
Run 350 cleared platform init (/user 200, /version 200, /repos 200), then stopped at
GET /api/v1/orgs/Home-Network with Gitea's own 403:
  token does not have at least one of required scope(s), required=[read:organization],
  token scope=read:package,write:repository,read:user
The workflow's header listed only those three scopes, and the operator minted exactly that.
Renovate's initRepo -> isOrg needs read:organization, and Gitea files /issues and /labels under
the issue scope, which the dashboard and PR labels need: write:issue.

- renovate.yaml: corrects the scope list (read:user, read:organization, write:repository,
  write:issue, read:package), with the call each serves. A new preflight step probes them with
  GETs, the token passed to curl on stdin, and names a missing scope instead of Renovate's bare
  "Authentication failure".
- ci/lint.sh: rejects workflow `uses:` lines that need a Docker daemon (docker://,
  renovatebot/github-action, docker/*-action), the class fixed in #84.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 01:55:39 -05:00
jakman85 d550a501e2 Merge pull request 'fix(renovate): run Renovate as the job container; the action needs a Docker daemon jobs do not get' (#84) from fix/renovate-first-run into main
gate / gate (push) Successful in 2m52s
gate / conformance (push) Has been skipped
Reviewed-on: #84
Reviewed-by: Keith Jackson <[email protected]>
2026-10-30 06:36:11 +00:00
jakman85andClaude Opus 5.5 365e43cdea fix(renovate): run Renovate as the job container; the action needs a Docker daemon jobs do not get
gate / gate (pull_request) Successful in 2m52s
gate / conformance (pull_request) Has been skipped
renovatebot/github-action is a wrapper around `docker run`. On this runner a job has no Docker
daemon by decision (decisions/2026-10-29-image-pipeline.md), so every run, including dispatch
run 343, died before Renovate started:

  failed to connect to the docker API at unix:///var/run/docker.sock ... no such file or directory

The pinned image ghcr.io/renovatebot/renovate:44.167.0 is now the job container, and a run step
execs renovate-entrypoint.sh. actions/checkout is dropped, since Renovate clones the repo itself.
An optional RENOVATE_GITHUB_COM_TOKEN secret feeds GITHUB_COM_TOKEN.

Local dry run (--platform=local, uid 12021, the step body verbatim): 179 deps extracted, 42
updates, and all 4 own-image digest lookups against the Gitea registry succeed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 01:23:36 -05:00
jakman85 dfc432c6ea Merge pull request 'fix(keycloak-passkey): passkey reconciler with review fixes — bind only on git's say-so (default false), protect real sub-flows, body-id screening, failure alert' (#82) from feat/keycloak-passkey-reconcile into main
gate / gate (push) Successful in 2m50s
gate / conformance (push) Has been skipped
Reviewed-on: #82
Reviewed-by: Keith Jackson <[email protected]>
2026-10-30 05:59:11 +00:00
jakman85 262c6e4a97 Merge branch 'main' into feat/keycloak-passkey-reconcile
gate / gate (pull_request) Successful in 2m56s
gate / conformance (pull_request) Has been skipped
2026-10-30 05:56:00 +00:00
jakman85 523cd9118f Merge pull request 'fix(buildkit): allow buildkitd egress to Gitea so the Gateway stops 403ing the registry push' (#83) from fix/gitea-admit-buildkit into main
gate / gate (push) Successful in 1m55s
gate / conformance (push) Has been skipped
Reviewed-on: #83
2026-10-30 05:50:12 +00:00
jakman85andClaude Opus 5.5 d2c4bd5194 fix(buildkit): allow buildkitd egress to Gitea's backend so the Gateway stops 403ing the push
gate / gate (pull_request) Successful in 1m56s
gate / conformance (pull_request) Has been skipped
Images run 337's push failed with "403 Forbidden: Access denied". Measured
read-only: Gitea never logged the request; an unauthenticated GET /v2/ via the
Gateway gets 401 from Gitea when sent from act-runner (egress unrestricted) but
403 from buildkitd; Hubble logs it as an L7 http-request DROPPED with
traffic_direction EGRESS. Cilium >= 1.16.8 (CVE-2025-30162) re-checks the
source pod's egress against the proxied backend identity, and buildkitd's
egress only allowed world and the ingress entity. Same fix as 68690e5 for
Keycloak. Gitea's ingress already admits reserved:ingress and is unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 00:38:54 -05:00
jakman85 0dd57fbaf0 Merge pull request 'feat(images): build, verify and publish the three images in CI on rootless BuildKit; Renovate carries the digests' (#81) from feat/image-pipeline into main
gate / gate (push) Successful in 1m57s
images / images (push) Failing after 5s
gate / conformance (push) Has been skipped
Reviewed-on: #81
Reviewed-by: Keith Jackson <[email protected]>
2026-10-30 05:15:16 +00:00
jakman85andClaude Opus 5.5 04eb38584a fix(conformance): workload-mTLS certificates may use a private CA, if they prove they are workload-internal
gate / gate (pull_request) Successful in 1m58s
images / images (pull_request) Failing after 7s
gate / conformance (pull_request) Has been skipped
R-TLS-01/docs/08 govern externally reachable TLS and ADR-0006 puts workload mTLS out of scope, but
tls-lifecycle-automated required ACME for every Certificate, which platform/buildkit's internal CA
cannot satisfy. Clause (5b) accepts the tls-scope=workload-mtls annotation only when the
certificate proves it: no Gateway serves it, internal names only, namespaced selfSigned/ca issuer
backed by an exempt CA certificate. Also fixes yamllint comma spacing in the buildkitd mounts.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5 f65c12b395 feat(images): build on rootless BuildKit over mTLS; verification moves into Containerfile verify stages
Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched.

- platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by
  digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined,
  read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP
  :1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it).
  No Kyverno exception: every applied policy matches app-namespace=true only.
- Each Containerfile names its shipped stage and ends in a `verify` stage holding every former
  `docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for
  the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now.
- images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify
  targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes.
  Fails loudly when the builder secrets are absent or buildkitd is unreachable.
- infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5 8f35234fcc fix(images): name the measured runner blocker; the job container gets no Docker daemon
PR #81's first images run failed in 6s: act_runner 0.2.11 forces docker_host '-' for a tcp://
DOCKER_HOST, and gitea/act drops every bind when valid_volumes is empty. Giving jobs the
privileged DinD daemon is node-root for any workflow author, so it is left to the operator;
options are in the handoff entry. The driver's failure now names the cause.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5 e539109830 feat(images): build, verify and publish the three images in CI; Renovate carries the digests
Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a
workstation build/push. Nothing had ever been pushed to the Gitea registry.

- .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main
  builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as
  sha-<commit> + main, reading the digest back from the registry. Only changed images build.
- Each image's own build-image.sh does the build and in-image verification, now engine-neutral
  (DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh.
- Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex
  manager for them. The worker Job template is deliberately untracked (it would loop).
- renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the
  current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION.
- Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in
  Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85 27da5555a4 Merge pull request 'fix(renovate): the workflow still named renovate.json, so every run since #79 died with ENOENT' (#80) from fix/renovate-config-filename into main
gate / gate (push) Successful in 1m54s
gate / conformance (push) Has been skipped
Reviewed-on: #80
Reviewed-by: Keith Jackson <[email protected]>
2026-10-30 04:48:42 +00:00
jakman85andClaude Opus 5.5 da1b36eb64 fix(renovate): pin renovate-version — the action's default (39) rejects renovate.json5
gate / gate (pull_request) Successful in 1m55s
gate / conformance (pull_request) Has been skipped
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 21:49:38 -05:00
jakman85andClaude Opus 5.5 aacf361bb8 fix(renovate): the workflow still named renovate.json, so every run since #79 died with ENOENT
gate / gate (pull_request) Failing after 1m58s
gate / conformance (pull_request) Has been skipped
#79 renamed the config to renovate.json5 and left configurationFile pointing at the old
name. Runs 315-322 (2026-10-22..29) all failed in ~1s before Renovate started.

Also add a ci/lint.sh assertion that every workflow configurationFile: resolves to a real
path — valid YAML, so yamllint could not catch it, and the gate stayed green throughout.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 20:58:15 -05:00
52 changed files with 3521 additions and 378 deletions

No files matched your search

+45
View File
@@ -115,14 +115,59 @@ CI. That is a real coverage gap, not a phase.
| variable | `REQUIRE_CLUSTER_TESTS` | `'true'` (Phase 1+) makes unverified cluster-needing touched reqs a hard block |
| secret | `CONFORMANCE_TRIGGER_KUBECONFIG` | narrow creds to create/await the in-cluster Chainsaw Job (not cluster-admin) |
## How images are built
This repo's three images — `toolbox` (`infra/devcontainer/`), `agent-runner` (`infra/agent/`,
built `FROM` the toolbox) and `conformance-runner` (`conformance/incluster/`) — are built,
verified and published by [`images.yaml`](images.yaml), never from a workstation
(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh).
**The builder is rootless BuildKit** ([`platform/buildkit/`](../../platform/buildkit/README.md)),
reached over mTLS with a pinned `buildx` — no job here touches a Docker daemon, privileged or
otherwise (operator decision on PR #81). The client certificate reaches jobs as the Actions
secrets `BUILDKIT_CA`, `BUILDKIT_CLIENT_CERT`, `BUILDKIT_CLIENT_KEY`.
**Verification is a build target.** Each Containerfile ends in a `verify` stage that runs every
in-image assertion at build time as the non-root uid with `RUN --network=none` (tool presence and
pinned versions, `PM_SELFTEST=1 pm-dispatch`, the pm-worker probe, the `conformance-clone`
refusals, no writable path outside the workspace). The pipeline builds `<image>-verify` before it
builds, let alone pushes, the image. Build args come from each image's own
`build-image.sh --print-build-args`, which is also the local podman path — so the recipe and the
proof exist once.
| Trigger | What happens |
|---|---|
| pull request touching an image's inputs | build + verify only. Nothing is pushed and **no registry credential is referenced** |
| push to `main` touching an image's inputs | build + verify + publish to `gitea.home.keithjackson.dev/home-network/<image>` as `sha-<commit>` (immutable) and `main` (moving); the **registry-returned** digest is in the job summary |
| `workflow_dispatch` | the same as a push when run on `main` (all images by default); verify-only elsewhere |
Only images whose inputs changed are built. The publish job **never writes to git**: Renovate
follows each image's `main` tag through a `# renovate-digest:` marker next to every digest
reference (`platform/pm-agent/`, `infra/agent/versions.env`, `conformance/versions.env`) and
opens the bump PR, which goes through the gate like anything else. A PR opened with the Actions
token would not trigger the required gate, which is why it is Renovate's job and not this one's.
Credential: the repository secret **`REGISTRY_PUSH_TOKEN`** (scope `write:package`, owned by an
account with package write on the org). The automatic Actions token cannot push in Gitea 1.24.3
— the reasoning, with source citations, is in the header of `images.yaml`. Retention:
[`infra/gitea/registry-retention.sh`](../../infra/gitea/registry-retention.sh), an operator
action, keeps every digest git pins plus the newest tags.
Cost: the job shares the single act_runner slot with the gate, but the build itself runs in the
`buildkitd` pod, not in DinD, so the runner's disk and memory are untouched. Until
`platform/buildkit` is deployed and its client certificate copied
(`infra/gitea/buildkit-client-secret.sh`), the job fails loudly at its builder preflight.
## Files
```
.gitea/workflows/
merge-gate.yaml the pipeline (lint → l0 → l1 → conformance → decision)
images.yaml build/verify/publish this repo's container images (see above)
scripts/
install-tools.sh installs the pinned tools per stage (from ci/versions.env)
decide.sh the R-TEST-03 merge decision over job results
images.sh the image pipeline's driver (what to build, base, push, digest)
README.md this file
```
+185
View File
@@ -0,0 +1,185 @@
# Container images — built, verified and published by CI, never from a workstation.
#
# ═══════════════════════════════════════════════════════════════════════════════════════════════
# WHY THIS FILE EXISTS (operator rule, 2026-10-29)
# ═══════════════════════════════════════════════════════════════════════════════════════════════
# "Container image creation, hosting and lifecycle must be part of CI/CD and the codebase, never a
# one-off build/push from a workstation." Before this workflow, three images existed only as
# workstation builds and NOTHING had ever been pushed to the Gitea registry: the Home-Network org's
# package list was empty, platform/pm-agent/ pinned an all-zero digest and sat in ImagePullBackOff,
# and conformance/versions.env carried an empty digest. Both build scripts said "publishing is an
# operator action" — a step with no log, no recorded inputs and no verification anyone could read.
#
# WHAT IT DOES — the logic is in scripts/images.sh; this file only wires triggers and secrets.
# pull_request -> build + VERIFY only. No push, and no registry credential is referenced.
# push to main -> build + verify + PUBLISH to gitea.home.keithjackson.dev/home-network/<image>
# dispatch -> same as a push when run on main; verify-only on any other ref.
# Images: toolbox (infra/devcontainer), agent-runner (infra/agent, FROM toolbox) and
# conformance-runner (conformance/incluster). Only images whose inputs changed are built.
#
# THE DIGEST LIFECYCLE — this workflow NEVER writes to git.
# 1. publish pushes `sha-<commit>` (immutable) and `main` (moving), and prints the digest the
# REGISTRY returns in the job summary.
# 2. Renovate (renovate.yaml) follows the `main` tag of each image through the
# `# renovate-digest:` markers next to every digest reference, and opens the bump PR.
# 3. That PR goes through the gate like any other change, and Flux deploys it.
# The bump PR is Renovate's and not this job's because a PR opened with the Actions token does not
# trigger workflows, so it could never pass the REQUIRED gate check; Renovate's token is a real
# user's.
#
# THE PUSH CREDENTIAL — `REGISTRY_PUSH_TOKEN`, a repository Actions secret, referenced BY NAME.
# The automatic Actions token CANNOT be used. Gitea 1.24.3 source, verified:
# * services/auth/basic.go (Basic.Verify) turns a task token into the pseudo-user
# `gitea-actions` (ActionsUserID -2) and sets NO token scope;
# * routers/api/packages/container/container.go (Authenticate, the /v2/token exchange every
# docker client uses) then requires read:package / write:package / all scope and answers 401;
# * services/context/package.go (determineAccessMode) has `// TODO: ActionUser permission check`
# and no branch for that user, so even past login it would get at most READ on a public org.
# So: a personal access token with scope `write:package`, minted by an account that has package
# WRITE on the Home-Network org (org owner, or a team with the Packages unit at write). A dedicated
# bot account in a packages-only team is the least-privilege choice. The username sent with it is
# not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER`
# is optional and only labels the login.
#
# THE BUILDER — rootless BuildKit (operator decision on PR #81, 2026-10-29).
# The act runner's job containers get NO Docker daemon (act_runner 0.2.11 forces docker_host "-"
# for a tcp:// DinD, and gitea/act strips every bind when valid_volumes is empty — measured on
# this PR's first run), and handing them the privileged DinD daemon would be root on the node for
# anyone who can push a branch. So images are built by platform/buildkit/: a ROOTLESS buildkitd
# (not privileged; uid 1000 in a user namespace), reached over mTLS at
# buildkitd.buildkit.svc.cluster.local:1234 with a pinned `buildx` (remote driver). Its client
# certificate reaches jobs as three Actions secrets — BUILDKIT_CA, BUILDKIT_CLIENT_CERT,
# BUILDKIT_CLIENT_KEY — copied from cert-manager by infra/gitea/buildkit-client-secret.sh.
# Pull requests get the client certificate (they must build) but never a registry credential.
#
# UNTIL platform/buildkit IS DEPLOYED (Flux applies it only after merge) THIS JOB FAILS, LOUDLY,
# at its builder preflight. A skipped Gitea job reports `success`, so it is red on purpose.
#
# VERIFICATION is each Containerfile's `verify` stage, built before anything is pushed.
#
name: images
on:
pull_request:
branches: [main]
paths:
- infra/devcontainer/Containerfile
- infra/devcontainer/versions.env
- infra/devcontainer/build-image.sh
- infra/agent/**
- conformance/incluster/Containerfile
- conformance/incluster/conformance-clone
- conformance/incluster/build-image.sh
- .gitea/workflows/images.yaml
- .gitea/workflows/scripts/images.sh
- platform/buildkit/versions.env
push:
branches: [main]
paths:
- infra/devcontainer/Containerfile
- infra/devcontainer/versions.env
- infra/devcontainer/build-image.sh
- infra/agent/**
- conformance/incluster/Containerfile
- conformance/incluster/conformance-clone
- conformance/incluster/build-image.sh
- .gitea/workflows/images.yaml
- .gitea/workflows/scripts/images.sh
- platform/buildkit/versions.env
workflow_dispatch:
inputs:
images:
description: "Which images: 'all', or a space-separated subset of: toolbox agent-runner conformance-runner"
required: false
default: "all"
concurrency:
# Never cancel a publish half-way: a pushed `sha-` tag with no matching `main` move is
# recoverable (a rerun skips what is already published) but pointless to provoke.
group: images-${{ github.ref }}
cancel-in-progress: false
jobs:
images:
runs-on: ubuntu-latest
# Measured on the PR that introduced this file — see its decisions entry. Generous because a
# cold DinD store downloads the Fedora base and every pinned tool on each run.
timeout-minutes: 75
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0 # the per-image change detection diffs against the base commit
# ── pull requests: build + verify, NO credential ───────────────────────────────────────
- name: Build and verify (pull request — nothing is pushed)
if: github.event_name == 'pull_request'
env:
MODE: verify
IMAGES: auto
PR_BASE_REF: ${{ github.base_ref }}
# The BUILDER's client certificate (platform/buildkit). Not a registry credential:
# a pull request can build, never push.
BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
run: |
set -euo pipefail
BASE_SHA="$(git merge-base "origin/${PR_BASE_REF}" HEAD)"
export BASE_SHA
./.gitea/workflows/scripts/images.sh
# ── non-main dispatch: verify only ──────────────────────────────────────────────────────
- name: Build and verify (dispatch off main — nothing is pushed)
if: github.event_name == 'workflow_dispatch' && github.ref != 'refs/heads/main'
env:
MODE: verify
IMAGES: ${{ inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images }}
BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
run: ./.gitea/workflows/scripts/images.sh
# ── main: publish ───────────────────────────────────────────────────────────────────────
- name: Preflight - the push token must exist, and its absence must be LOUD
if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
env:
REGISTRY_PUSH_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
run: |
set -euo pipefail
if [ -z "${REGISTRY_PUSH_TOKEN:-}" ]; then
echo "::error::REGISTRY_PUSH_TOKEN is not set on this repository." >&2
cat >&2 <<'MSG'
Images cannot be published without a registry credential, and this step fails rather
than skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would
make "nothing was published" look exactly like "published". That is how this repo went
a month with images that existed only on one workstation.
WHY NOT THE BUILT-IN ACTIONS TOKEN: Gitea 1.24.3 gives it no package scope at the
registry's /v2/token exchange (routers/api/packages/container/container.go,
Authenticate) — see the header of .gitea/workflows/images.yaml.
TO FIX: as an account with package WRITE on the Home-Network org (ideally a dedicated
bot in a packages-only team), create an access token with scope `write:package` and
nothing else, then add it to this repository as the Actions secret
REGISTRY_PUSH_TOKEN. Optionally set the Actions variable REGISTRY_PUSH_USER to that
account's name. Do not commit the token anywhere. Then re-run this workflow
(workflow_dispatch on main, images: all).
MSG
exit 1
fi
echo "REGISTRY_PUSH_TOKEN is present (value not printed)."
- name: Build, verify and publish (main)
if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
env:
MODE: publish
IMAGES: ${{ github.event_name == 'workflow_dispatch' && (inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images) || 'auto' }}
BASE_SHA: ${{ github.event.before }}
REGISTRY_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
REGISTRY_USER: ${{ vars.REGISTRY_PUSH_USER }}
BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
run: ./.gitea/workflows/scripts/images.sh
+146 -15
View File
@@ -60,8 +60,21 @@
# entry's "What I did NOT verify" — that gap is stated, not papered over.
#
# OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE:
# `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository` and
# `read:user`. Set it as an Actions secret on this repository. Nothing in this repo mints,
# `RENOVATE_TOKEN` — a Gitea access token for the bot account (`renovate-bot`) with FIVE scopes.
# Each is required by a call Renovate 44.x makes against Gitea 1.24.3, and Gitea refuses a
# token missing one with `403 token does not have at least one of required scope(s)`:
# read:user GET /user (platform init: initPlatform)
# read:organization GET /orgs/{owner}, /orgs/{o}/labels (initRepo -> isOrg, then org labels)
# write:repository clone/push branches, /pulls, /statuses
# write:issue /issues, /labels, /issues/{n}/labels (dependency dashboard, PR labels;
# Gitea files all of these under the ISSUE scope, not repository)
# read:package the Gitea container registry (RENOVATE_HOST_RULES below)
# CORRECTED 2026-10-30. This list used to name only write:repository + read:user (+ read:package),
# and the operator minted exactly that. Run 350 then failed at
# `GET /api/v1/orgs/Home-Network 403 ... required=[read:organization], token scope=read:package,
# write:repository,read:user`; write:issue would have been the next wall. The preflight below now
# PROBES these scopes, so a short token fails in seconds with the missing scope named.
# Set it as an Actions secret on this repository. Nothing in this repo mints,
# stores or references its value; it is consumed by name only.
# Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a
# visible red is the correct state for "automation configured but not credentialed".
@@ -115,15 +128,37 @@ concurrency:
jobs:
renovate:
runs-on: ubuntu-latest
# RENOVATE RUNS AS THE JOB CONTAINER, NOT THROUGH renovatebot/github-action. FIXED 2026-10-30.
# That action does exactly one thing: `docker run ghcr.io/renovatebot/renovate:<version>`. On
# this runner a job has NO Docker daemon, by decision (decisions/2026-10-29-image-pipeline.md:
# act_runner sets container.docker_host to `-` when DOCKER_HOST is tcp://, and the DinD daemon
# is privileged, so a job that could reach it would be root on the node). So every run of the
# action died before Renovate started. Dispatch run 343 (2026-10-30), verbatim:
# [command]/usr/bin/docker run --env LOG_LEVEL ... --env RENOVATE_TOKEN=*** ...
# failed to connect to the docker API at unix:///var/run/docker.sock; ... no such file or directory
# ::error::Error: The process '/usr/bin/docker' failed with exit code 1
# No auth, scope, config or network question was ever reached. Making the Renovate image the
# job container needs no daemon: act starts it through the runner's own DinD exactly as it
# starts the default ubuntu image, then `exec`s each step inside it.
#
# PINNED here (was the action's `renovate-version: 44.167.0`). renovate.json5 is validated
# against this exact tag — keep them in step. Renovate's github-actions manager reads
# `.gitea/workflows` and extracts `container.image`, so this pin is still self-maintained.
#
# The image runs as uid 12021 (`ubuntu`), and that is fine under act: the step command files
# ($GITHUB_STEP_SUMMARY etc.) are created mode 0666 (gitea/act v0.261.2 pkg/runner/step.go),
# HOME comes from the image env, and no step writes the workspace. There is deliberately NO
# actions/checkout: Renovate clones the repo itself from RENOVATE_ENDPOINT, and as uid 12021
# checkout could not `git init` the root-owned workspace volume anyway.
container:
image: ghcr.io/renovatebot/renovate:44.167.0
# Renovate scans every manager across the whole repo; 40m is generous for this repo's size and
# well under a runner's patience. Without a timeout a wedged run holds the `renovate`
# concurrency group and silently blocks every subsequent scheduled run.
timeout-minutes: 40
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Preflight - the token must exist, and its absence must be LOUD
shell: bash
env:
RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
run: |
@@ -138,23 +173,94 @@ jobs:
That is the failure mode that let renovate.json sit inert in this repo for months while
seven files claimed "Renovate maintains this pin".
TO FIX: create a Gitea access token for the bot account with scopes
`write:repository` and `read:user`, then add it to this repository as the Actions
secret RENOVATE_TOKEN. Do not commit it anywhere.
TO FIX: create a Gitea access token for the bot account (renovate-bot) with scopes
read:user, read:organization, write:repository, write:issue and read:package (the
header of .gitea/workflows/renovate.yaml says which call needs which), then add it to
this repository as the Actions secret RENOVATE_TOKEN. Do not commit it anywhere.
MSG
exit 1
fi
echo "RENOVATE_TOKEN is present (value not printed)."
# Proves the step is running INSIDE the pinned Renovate image, not the default job image.
echo "renovate $(renovate --version)"
- name: Preflight - the token has every scope Renovate needs
shell: bash
env:
RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
# ADDED 2026-10-30. Renovate reports EVERY platform-init failure as the bare
# "Authentication failure" (lib/modules/platform/gitea/index.ts initPlatform swallows the
# HTTP error), and a later scope gap surfaces as "Repository has unknown error". Two
# dispatches were spent learning one missing scope at a time. These probes are GETs only,
# and they name the missing scope. The token never reaches argv: curl reads it from stdin
# (`--config -`), so it is not in `ps` output or in the log.
run: |
set -uo pipefail
GITEA=https://gitea.home.keithjackson.dev
REPO=Home-Network/home-network-k3s-impl
fail=0
body=$(mktemp)
probe() { # <scope it proves> <path> <auth: bearer|basic>
local code
if [ "$3" = basic ]; then
code=$(printf 'user = "renovate:%s"\n' "$RENOVATE_TOKEN" \
| curl -sS --config - -o "$body" -w '%{http_code}' "$GITEA$2") || code=000
else
code=$(printf 'header = "Authorization: Bearer %s"\n' "$RENOVATE_TOKEN" \
| curl -sS --config - -o "$body" -w '%{http_code}' "$GITEA$2") || code=000
fi
if [ "$code" = 200 ]; then
echo "ok $1 (GET ${2%%\?*} -> 200)"
return
fi
fail=1
echo "::error::scope $1: GET ${2%%\?*} -> $code"
case "$code" in
401) echo " 401 = Gitea rejected the token VALUE (wrong, expired or revoked)." ;;
403) if grep -q 'Access denied' "$body"; then
echo " 403 'Access denied' = a Cilium/Envoy policy denial, not Gitea."
else
echo " Gitea: $(jq -r '.message // empty' "$body" 2>/dev/null | head -c 300)"
fi ;;
*) echo " body: $(head -c 200 "$body")" ;;
esac
}
probe read:user "/api/v1/user" bearer
probe read:organization "/api/v1/orgs/${REPO%%/*}" bearer
probe read:repository "/api/v1/repos/$REPO" bearer
probe read:issue "/api/v1/repos/$REPO/labels?limit=1" bearer
probe read:package "/v2/token?service=container_registry&scope=repository:home-network/toolbox:pull" basic
# WRITE scopes cannot be proven by a GET. Gitea's scope-refusal message, however, lists
# the token's scopes (names only, never the value): ask for an admin route this bot
# cannot have, and read `token scope=` from the 403. Advisory if the format changes.
printf 'header = "Authorization: Bearer %s"\n' "$RENOVATE_TOKEN" \
| curl -sS --config - -o "$body" "$GITEA/api/v1/admin/orgs?limit=1" || true
scopes=$(jq -r '.message // empty' "$body" 2>/dev/null | sed -n 's/.*token scope=//p')
if [ -n "$scopes" ]; then
echo "token scopes (from Gitea): $scopes"
for want in write:repository write:issue; do
case ",$scopes," in
*",$want,"*|*",all,"*) echo "ok $want" ;;
*) echo "::error::scope $want is missing (token scope=$scopes)"; fail=1 ;;
esac
done
else
echo "::warning::could not read the token's scope list; write:repository and write:issue are UNVERIFIED"
fi
rm -f "$body"
if [ "$fail" -ne 0 ]; then
echo "::error::RENOVATE_TOKEN failed the checks above (a 401 is the value, a 403 is a scope). Re-mint it for renovate-bot with read:user, read:organization, write:repository, write:issue, read:package and overwrite the secret."
exit 1
fi
- name: Run Renovate
uses: renovatebot/[email protected]
with:
# Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers
# includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`.
# This is the one dependency in the repo whose updater is its own updater.
configurationFile: renovate.json
token: ${{ secrets.RENOVATE_TOKEN }}
shell: bash
# The repo config is discovered by Renovate from the cloned repo (renovate.json5,
# RENOVATE_REQUIRE_CONFIG=required). The action used to ALSO load it as the global config
# via RENOVATE_CONFIG_FILE; that double-loading is gone. ci/lint.sh's `configurationFile:`
# check therefore has nothing to resolve in this file, and passes.
env:
RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
# Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
# header said were "set at runtime by the operator's renovate runner" — this is that
# runner, so they are set here rather than in the repo config, which keeps renovate.json
@@ -170,6 +276,31 @@ jobs:
RENOVATE_REQUIRE_CONFIG: "required"
RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }}
LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }}
# ADDED 2026-10-29: registry read access for this repo's OWN images (toolbox,
# agent-runner, conformance-runner — published by .gitea/workflows/images.yaml and
# followed through `# renovate-digest:` markers). The Gitea registry answers 401 to
# anonymous reads, and Renovate scopes the platform token to hostType `gitea`, which a
# docker lookup never matches — so without this rule every digest lookup fails. Same
# token; it therefore needs scope `read:package` in addition to the ones listed in the
# preflight above. Gitea ignores the username for token auth (services/auth/basic.go).
RENOVATE_HOST_RULES: '[{"hostType":"docker","matchHost":"gitea.home.keithjackson.dev","username":"renovate","password":"${{ secrets.RENOVATE_TOKEN }}"}]'
# OPTIONAL, ADDED 2026-10-30. A github.com token (a fine-grained PAT with NO permissions,
# public-repo read only) for lookups of GitHub-hosted dependencies. Without it Renovate
# SKIPS them with `github-token-required` — measured in a local dry run: 17 deps, i.e.
# every `uses: actions/*` here plus kubevirt/kubevirt, kubevirt/containerized-data-importer
# and intel/intel-device-plugins-for-kubernetes. Unset is safe: the step below drops the
# empty variable, and those deps simply stay unmaintained, with a WARN in the log.
GITHUB_COM_TOKEN: ${{ secrets.RENOVATE_GITHUB_COM_TOKEN }}
run: |
set -euo pipefail
# An expression that evaluates to '' still EXPORTS the variable, as an empty string.
# Drop those so an unset value means "Renovate's default", never "set to empty".
for v in RENOVATE_DRY_RUN GITHUB_COM_TOKEN; do
[ -n "${!v:-}" ] || unset "$v"
done
# Through the image's entrypoint, as `docker run` would have gone: it sources the
# containerbase env and initialises the bundled tools before exec'ing Renovate.
exec renovate-entrypoint.sh renovate
- name: Summary
if: always()
+373
View File
@@ -0,0 +1,373 @@
#!/usr/bin/env bash
###############################################################################
# .gitea/workflows/scripts/images.sh — the image pipeline's driver (images.yaml calls this).
#
# Builds, verifies and (on main) PUBLISHES the three images this repo ships:
#
# toolbox infra/devcontainer/ the pinned dev/gate toolchain
# agent-runner infra/agent/ FROM toolbox; the PM dispatcher + worker
# conformance-runner conformance/incluster/ the in-cluster L2/L3 Chainsaw runner
#
# THE BUILDER (operator decision on PR #81): a ROOTLESS BuildKit daemon, platform/buildkit/,
# reached over mTLS. No CI job may reach a privileged daemon, so there is no `docker build` and no
# `docker run` here: this script drives buildkitd with a pinned `buildx` (remote driver) and
# nothing else. The client certificate arrives as Actions secrets (BUILDKIT_CA,
# BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY; infra/gitea/buildkit-client-secret.sh puts them
# there). If the daemon is unreachable this FAILS LOUDLY — it never skips.
#
# VERIFICATION IS A BUILD TARGET. Each Containerfile ends in a `verify` stage that runs every
# in-image assertion at build time (as the non-root uid, with RUN --network=none); this script
# builds `<image>-verify` and only then the image. Build args come from each image's own
# `build-image.sh --print-build-args`, the same list the local podman path uses.
#
# MODES (env MODE):
# verify build the verify targets, never push, hold no registry credential. Pull requests.
# publish verify, then build + push to ${REGISTRY}/${NAMESPACE}/<image> with tags
# `sha-<12 hex>` (immutable: never re-pushed) and `main` (moving; Renovate follows
# it). The digest recorded is the one the REGISTRY returns, read back after the push.
#
# WHAT GETS BUILT (env IMAGES): `auto` (default) diffs BASE_SHA..HEAD_SHA against each image's
# inputs; otherwise a space-separated list. A change to the pipeline itself (images.yaml, this
# file, platform/buildkit/versions.env) builds everything in verify mode, and nothing extra in
# publish mode: re-publishing unchanged inputs would only mint new digests and churn Renovate PRs.
#
# THE agent-runner BASE:
# * verify mode -> FROM the toolbox built in this run, linked with a bake `target:` context —
# a pull request holds no registry credential and that toolbox exists in no registry. If the
# toolbox's inputs did not change it is built anyway, as the base.
# * publish mode, toolbox published in this run -> FROM the digest the registry just returned.
# * publish mode otherwise -> AGENT_BASE_IMAGE from infra/agent/versions.env (the published
# toolbox digest Renovate keeps current). If that is still the all-zero placeholder the
# toolbox has never been published, so it is built and published first (the bootstrap).
#
# LOOP-FREEDOM. Renovate turns each publish into a digest-bump PR. Those bumps land in
# platform/pm-agent/ (no image input), conformance/versions.env (no image input) and
# infra/agent/versions.env (agent-runner input -> ONE more agent-runner build, whose own bump
# lands in platform/pm-agent/ and stops). The worker Job template is deliberately NOT tracked:
# it is baked INTO agent-runner, so bumping it would rebuild agent-runner, mint a new digest,
# bump it again, forever. pm-dispatch overwrites that field at runtime anyway.
###############################################################################
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
cd "${REPO_ROOT}"
# shellcheck disable=SC1091
BK_HOST_OVERRIDE="${BUILDKIT_HOST:-}"; BK_NAME_OVERRIDE="${BUILDKIT_SERVERNAME:-}"
. ./platform/buildkit/versions.env
BUILDKIT_HOST="${BK_HOST_OVERRIDE:-${BUILDKIT_HOST}}"
BUILDKIT_SERVERNAME="${BK_NAME_OVERRIDE:-${BUILDKIT_SERVERNAME}}"
MODE="${MODE:-verify}"
IMAGES="${IMAGES:-auto}"
REGISTRY="${REGISTRY:-gitea.home.keithjackson.dev}"
NAMESPACE="${NAMESPACE:-home-network}"
HEAD_SHA="${HEAD_SHA:-$(git rev-parse HEAD)}"
BASE_SHA="${BASE_SHA:-}"
SUMMARY="${GITHUB_STEP_SUMMARY:-/dev/null}"
ZERO_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000"
ALL_IMAGES=(toolbox agent-runner conformance-runner)
SHA_TAG="sha-${HEAD_SHA:0:12}"
MOVING_TAG="main"
log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
info() { printf ' %s\n' "$*"; }
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; printf -- '- **ERROR**: %s\n' "$*" >>"${SUMMARY}"; exit 1; }
yn() { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; }
case "${MODE}" in verify|publish) ;; *) die "MODE must be verify or publish, got '${MODE}'" ;; esac
for t in git jq curl sha256sum; do command -v "$t" >/dev/null || die "the job image lacks '$t'"; done
# --- per-image facts ------------------------------------------------------------------------
# Inputs: a change under any of these rebuilds that image. conformance-runner takes its pins from
# infra/devcontainer/versions.env. conformance/versions.env is NOT an input (it only RECORDS the
# published digest), and must not become one: Renovate writes it after every publish -> a loop.
inputs() {
case "$1" in
toolbox) echo "infra/devcontainer/Containerfile infra/devcontainer/versions.env infra/devcontainer/build-image.sh" ;;
agent-runner) echo "infra/agent/" ;;
conformance-runner) echo "conformance/incluster/Containerfile conformance/incluster/conformance-clone conformance/incluster/build-image.sh infra/devcontainer/versions.env" ;;
esac
}
context_of() { case "$1" in toolbox) echo infra/devcontainer ;; agent-runner) echo infra/agent ;; conformance-runner) echo conformance/incluster ;; esac; }
PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh platform/buildkit/versions.env"
declare -A WANT=() BUILT=() DIGEST=() SECS=() NOTE=()
changed_any() { # <paths...> -> 0 if BASE_SHA..HEAD_SHA touches any
[ -n "$(git diff --name-only "${BASE_SHA}" "${HEAD_SHA}" -- "$@")" ]
}
select_images() {
local i
if [ "${IMAGES}" != "auto" ]; then
for i in ${IMAGES}; do
case " ${ALL_IMAGES[*]} " in *" ${i} "*) WANT[$i]=1; NOTE[$i]="requested" ;; *) die "unknown image '${i}'" ;; esac
done
return
fi
if [ -z "${BASE_SHA}" ] || [[ "${BASE_SHA}" =~ ^0+$ ]] || ! git cat-file -e "${BASE_SHA}^{commit}" 2>/dev/null; then
info "no usable diff base ('${BASE_SHA}'): building everything"
for i in "${ALL_IMAGES[@]}"; do WANT[$i]=1; NOTE[$i]="no diff base"; done
return
fi
info "diff base ${BASE_SHA} .. ${HEAD_SHA}"
for i in "${ALL_IMAGES[@]}"; do
# shellcheck disable=SC2046
if changed_any $(inputs "$i"); then WANT[$i]=1; NOTE[$i]="inputs changed"; fi
done
# shellcheck disable=SC2086
if [ "${MODE}" = verify ] && changed_any ${PIPELINE_FILES}; then
for i in "${ALL_IMAGES[@]}"; do [ -n "${WANT[$i]:-}" ] || { WANT[$i]=1; NOTE[$i]="pipeline changed"; }; done
fi
}
# --- the builder: pinned buildx + mTLS to rootless buildkitd ---------------------------------
WORK="$(mktemp -d)"
trap 'rm -rf "${WORK}"' EXIT # client key + registry auth live only here, only this long
export BUILDX_CONFIG="${WORK}/buildx" DOCKER_CONFIG="${WORK}/docker"
# Single-manifest images: the admission policies, Renovate and registry-retention.sh all reason
# about one manifest per tag. (Provenance attestations would turn every push into an index.)
export BUILDX_NO_DEFAULT_ATTESTATIONS=1
mkdir -p "${BUILDX_CONFIG}" "${DOCKER_CONFIG}" "${WORK}/certs"
echo '{}' > "${DOCKER_CONFIG}/config.json"
setup_builder() {
log "Builder: rootless buildkitd at ${BUILDKIT_HOST} (mTLS)"
BUILDX="${BUILDX_BIN:-}"
if [ -z "${BUILDX}" ]; then
BUILDX="${WORK}/buildx-bin"
curl -fsSL --retry 3 -o "${BUILDX}" \
"https://github.com/docker/buildx/releases/download/v${BUILDX_VERSION}/buildx-v${BUILDX_VERSION}.linux-amd64" \
|| die "could not download buildx v${BUILDX_VERSION}"
echo "${BUILDX_SHA256} ${BUILDX}" | sha256sum -c --quiet - \
|| die "buildx v${BUILDX_VERSION} does not match the sha256 pinned in platform/buildkit/versions.env — refusing to run it"
chmod +x "${BUILDX}"
fi
info "client: $("${BUILDX}" version)"
local missing=()
[ -n "${BUILDKIT_CA:-}" ] || missing+=(BUILDKIT_CA)
[ -n "${BUILDKIT_CLIENT_CERT:-}" ] || missing+=(BUILDKIT_CLIENT_CERT)
[ -n "${BUILDKIT_CLIENT_KEY:-}" ] || missing+=(BUILDKIT_CLIENT_KEY)
[ "${#missing[@]}" -eq 0 ] || die "the builder's client certificate is missing: ${missing[*]} not set.
These Actions secrets are copied from the cert-manager Secret buildkit/buildkit-client-ci by
infra/gitea/buildkit-client-secret.sh (an operator action, after platform/buildkit is
deployed). Without them nothing can be built — this fails rather than skipping ON PURPOSE."
umask 077
printf '%s\n' "${BUILDKIT_CA}" > "${WORK}/certs/ca.pem"
printf '%s\n' "${BUILDKIT_CLIENT_CERT}" > "${WORK}/certs/cert.pem"
printf '%s\n' "${BUILDKIT_CLIENT_KEY}" > "${WORK}/certs/key.pem"
umask 022
"${BUILDX}" create --name ci-remote --driver remote \
--driver-opt "cacert=${WORK}/certs/ca.pem,cert=${WORK}/certs/cert.pem,key=${WORK}/certs/key.pem,servername=${BUILDKIT_SERVERNAME}" \
"${BUILDKIT_HOST}" >/dev/null
if ! out="$(timeout 90 "${BUILDX}" inspect --bootstrap --builder ci-remote 2>&1)"; then
printf '%s\n' "${out}" >&2
die "buildkitd is UNREACHABLE at ${BUILDKIT_HOST} — nothing can be built, and this job fails
rather than skipping. In order of likelihood:
1. platform/buildkit is not deployed yet. Flux applies it only after it merges to main
(Kustomization 'buildkit'); a pull request that ADDS it cannot reach it.
2. the BUILDKIT_* secrets are stale (the client cert was renewed) or from another CA:
re-run infra/gitea/buildkit-client-secret.sh.
3. the NetworkPolicy (platform/buildkit/networkpolicy.yaml) does not admit this runner."
fi
printf '%s\n' "${out}" | grep -E '^(Name|Driver|Status|BuildKit version|Platforms):' | sed 's/^/ /' || true
}
# KEY=VALUE lines from an image's build-image.sh --print-build-args -> a JSON object.
build_args_json() {
local ctx; ctx="$(context_of "$1")"
"./${ctx}/build-image.sh" --print-build-args | jq -Rn '[inputs | select(length>0) | capture("^(?<k>[^=]+)=(?<v>.*)$")] | map({(.k): .v}) | add'
}
# bake_run <bakefile> <targets...>
bake_run() {
local f="$1"; shift
"${BUILDX}" bake --builder ci-remote --progress plain -f "$f" \
--metadata-file "${WORK}/meta.json" "$@"
}
# Write a bake file with a verify target and a ship target for one image.
# $1 image, $2 agent base mode: "" | "link" | <image ref>
write_bake() {
local img="$1" base="${2:-}" ctx args f
ctx="$(context_of "$img")"
args="$(build_args_json "$img")"
f="${WORK}/bake-${img}.json"
local contexts='{}'
if [ "${img}" = agent-runner ]; then
if [ "${base}" = link ]; then
# FROM ${AGENT_BASE_IMAGE} resolves to the named context below, i.e. the toolbox target
# of this same bake file — built here, never pushed anywhere.
args="$(jq -c '. + {AGENT_BASE_IMAGE: "toolbox-in-this-run"}' <<<"${args}")"
contexts='{"toolbox-in-this-run": "target:toolbox"}'
else
args="$(jq -c --arg b "${base}" '. + {AGENT_BASE_IMAGE: $b}' <<<"${args}")"
fi
fi
jq -n --arg img "$img" --arg ctx "$ctx" --argjson args "$args" --argjson contexts "$contexts" \
--arg tbargs "$( [ "${base}" = link ] && build_args_json toolbox || echo '{}')" \
--arg rev "${HEAD_SHA}" --arg repo "${REGISTRY}/${NAMESPACE}/${img}" \
--arg sha "${SHA_TAG}" --arg moving "${MOVING_TAG}" --arg mode "${MODE}" '
{target: (
{
($img + "-verify"): {context: $ctx, dockerfile: "Containerfile", target: "verify",
args: $args, contexts: $contexts, output: ["type=cacheonly"]},
($img): ({context: $ctx, dockerfile: "Containerfile", target: $img, args: $args,
contexts: $contexts,
labels: {"org.opencontainers.image.revision": $rev}}
+ (if $mode == "publish"
then {tags: [($repo + ":" + $sha), ($repo + ":" + $moving)], output: ["type=image,push=true"]}
else {output: ["type=cacheonly"]} end))
}
+ (if ($contexts | has("toolbox-in-this-run"))
then {toolbox: {context: "infra/devcontainer", dockerfile: "Containerfile", target: "toolbox",
args: ($tbargs | fromjson), output: ["type=cacheonly"]}}
else {} end)
)}' > "$f"
echo "$f"
}
# --- registry helpers (publish mode only) ----------------------------------------------------
REG_TOKEN=""
registry_token() {
[ -n "${REG_TOKEN}" ] && return 0
REG_TOKEN="$(curl -fsS --config <(printf 'user = "%s:%s"\n' "${REGISTRY_USER}" "${REGISTRY_TOKEN}") \
"https://${REGISTRY}/v2/token?service=container_registry&scope=*" | jq -r '.token // empty')" \
|| die "could not obtain a registry bearer token from https://${REGISTRY}/v2/token"
[ -n "${REG_TOKEN}" ] || die "registry returned no bearer token"
}
# Prints the manifest digest the REGISTRY serves for <image>:<tag>; nothing if it is absent (404).
# Any other answer is fatal: "cannot tell" must never be read as "not published", or the
# immutable-tag guard would be bypassed by an auth hiccup.
registry_digest() {
registry_token
local hdrs code digest
hdrs="$(curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "${REG_TOKEN}") \
-H 'Accept: application/vnd.oci.image.manifest.v1+json' \
-H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
-H 'Accept: application/vnd.oci.image.index.v1+json' \
-H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
"https://${REGISTRY}/v2/${NAMESPACE}/$1/manifests/$2" | tr -d '\r')" \
|| die "registry unreachable while reading $1:$2"
code="$(awk 'NR==1{print $2}' <<<"${hdrs}")"
digest="$(awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' <<<"${hdrs}" | tail -1)"
case "${code}" in
200) [[ "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]] || die "registry answered 200 for $1:$2 with no usable digest"
echo "${digest}" ;;
404) ;;
*) die "registry answered HTTP ${code} for $1:$2 — cannot tell whether it exists" ;;
esac
}
# build_one <image> [agent base] — verify, then (publish) build + push + read back.
build_one() {
local img="$1" base="${2:-}" t0 f pushed got existing
t0=$(date +%s)
f="$(write_bake "$img" "$base")"
log "VERIFY ${img} (Containerfile target: verify, on rootless BuildKit)"
bake_run "$f" "${img}-verify"
BUILT[$img]=1
if [ "${MODE}" = publish ]; then
existing="$(registry_digest "${img}" "${SHA_TAG}")"
# Immutable tag: never overwrite. A rebuild is not byte-reproducible, so re-pushing would
# silently change what `sha-<commit>` means after something may already have pinned it.
[ -z "${existing}" ] || die "${REGISTRY}/${NAMESPACE}/${img}:${SHA_TAG} already exists (${existing}); tags are immutable"
log "BUILD + PUSH ${img} -> ${REGISTRY}/${NAMESPACE}/${img}:{${SHA_TAG},${MOVING_TAG}}"
bake_run "$f" "${img}"
pushed="$(jq -r --arg t "$img" '.[$t]["containerimage.digest"] // empty' "${WORK}/meta.json")"
# The AUTHORITATIVE digest is what the registry serves, read back — not what the client
# reported. They should agree; if they do not, nothing downstream may trust either.
got="$(registry_digest "${img}" "${SHA_TAG}")"
[ -n "${got}" ] || die "pushed ${img}:${SHA_TAG} but the registry does not serve it"
[ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] || die "${img}: client reported ${pushed} but the registry serves ${got}"
[ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \
|| die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})"
DIGEST[$img]="${got}"
info "${img}: registry digest ${got}"
fi
SECS[$img]=$(( $(date +%s) - t0 ))
info "${img}: done in ${SECS[$img]}s"
}
# --- main ---------------------------------------------------------------------------------
log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}"
select_images
if [ "${MODE}" = publish ]; then
: "${REGISTRY_TOKEN:?REGISTRY_TOKEN is required in publish mode}"
REGISTRY_USER="${REGISTRY_USER:-ci-image-publisher}"
# buildx (and through it buildkitd, which does the push) reads registry auth from this file.
jq -n --arg r "${REGISTRY}" --arg a "$(printf '%s:%s' "${REGISTRY_USER}" "${REGISTRY_TOKEN}" | base64 -w0)" \
'{auths: {($r): {auth: $a}}}' > "${DOCKER_CONFIG}/config.json"
chmod 600 "${DOCKER_CONFIG}/config.json"
registry_token
# Publish idempotence: an image already published at THIS commit is not rebuilt. That is what
# makes a re-run after a half-finished publish safe — the immutable tag is never pushed twice.
for i in "${ALL_IMAGES[@]}"; do
[ -n "${WANT[$i]:-}" ] || continue
d="$(registry_digest "${i}" "${SHA_TAG}")"
if [ -n "${d}" ]; then
info "${i}: already published at ${SHA_TAG} (${d}); not rebuilding"
unset "WANT[$i]"; DIGEST[$i]="${d}"; NOTE[$i]="already published at this commit"
fi
done
fi
# Decide the agent-runner base (see header).
AGENT_BASE=""
if [ -n "${WANT[agent-runner]:-}" ]; then
PINNED_BASE="$(. ./infra/agent/versions.env; echo "${AGENT_BASE_IMAGE}")"
if [ "${MODE}" = verify ]; then
AGENT_BASE=link
[ -n "${WANT[toolbox]:-}" ] || { WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)"; }
elif [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then
case "${PINNED_BASE}" in
*@"${ZERO_DIGEST}") WANT[toolbox]=1; NOTE[toolbox]="never published; required as agent-runner's base" ;;
*@sha256:*) AGENT_BASE="${PINNED_BASE}" ;;
*) die "infra/agent/versions.env AGENT_BASE_IMAGE='${PINNED_BASE}' is not digest-pinned; CI will not build on a floating base" ;;
esac
fi
fi
for i in "${ALL_IMAGES[@]}"; do info "plan: ${i}: $(yn "${WANT[$i]:-}" BUILD skip) (${NOTE[$i]:-inputs unchanged})"; done
if [ -z "${WANT[toolbox]:-}${WANT[agent-runner]:-}${WANT[conformance-runner]:-}" ]; then
info "nothing to build"
else
setup_builder
fi
if [ -n "${WANT[toolbox]:-}" ]; then build_one toolbox; fi
if [ -n "${WANT[agent-runner]:-}" ]; then
if [ -z "${AGENT_BASE}" ]; then
AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}"
fi
info "agent-runner base: $( [ "${AGENT_BASE}" = link ] && echo "the toolbox built in this run (bake target: context)" || echo "${AGENT_BASE}")"
build_one agent-runner "${AGENT_BASE}"
fi
if [ -n "${WANT[conformance-runner]:-}" ]; then build_one conformance-runner; fi
# --- summary ------------------------------------------------------------------------------
{
echo "## Images (${MODE})"
echo
echo "commit \`${HEAD_SHA}\` — builder: rootless BuildKit \`${BUILDKIT_HOST}\` (mTLS)"
echo
echo "| image | verify target built | seconds | published digest (registry-returned) | note |"
echo "|---|---|---|---|---|"
for i in "${ALL_IMAGES[@]}"; do
d="${DIGEST[$i]:-}"
echo "| \`${i}\` | $(yn "${BUILT[$i]:-}" yes no) | ${SECS[$i]:--} | $(yn "${d}" "\`${REGISTRY}/${NAMESPACE}/${i}@${d}\`" -) | ${NOTE[$i]:-inputs unchanged} |"
done
echo
if [ "${MODE}" = publish ]; then
echo "Tags pushed: \`${SHA_TAG}\` (immutable) and \`${MOVING_TAG}\` (moving)."
echo "Nothing here writes to git. Renovate opens the digest-bump PR on its next run."
else
echo "Verify mode: nothing was pushed and no registry credential was present."
fi
} >>"${SUMMARY}"
log "done"
+1 -1
View File
@@ -75,7 +75,7 @@ spec:
spec:
containers:
- name: keycloak
image: quay.io/keycloak/keycloak
image: quay.io/keycloak/keycloak:26.0.6
ports:
- containerPort: 8080
volumeMounts:
+8 -1
View File
@@ -5,8 +5,15 @@
# Validates against schemas/servicedefinition.schema.json.
name: keycloak
kind: container
# DOCUMENTATION-ONLY IMAGE. The running server is NOT chosen here: the Keycloak operator
# (platform/keycloak/kustomization.yaml, three pinned release URLs) picks it via its
# RELATED_IMAGE_KEYCLOAK, and the Keycloak CR has no spec.image. This SD's render,
# apps/wave-1/keycloak.yaml, is not listed in apps/wave-1/kustomization.yaml and deploys nothing
# (ci/render-golden.sh RENDER_GOLDEN_NOT_ONBOARDED). The tag records the operator's version for
# readers; renovate.json5 disables Renovate for this image so it cannot raise PRs against a file
# nothing deploys. The Keycloak version moves with the operator — update this tag by hand then.
source:
image: quay.io/keycloak/keycloak
image: quay.io/keycloak/keycloak:26.0.6
exposure: public # login/redirect flows for internet-exposed services; hardened (docs/15; R-DNS/R-NET)
egress: internet # normal WAN egress (docs/03; R-NET-05)
storage:
+31
View File
@@ -27,3 +27,34 @@ fi
yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${files[@]}"
ok "yaml/lint passed (${#files[@]} files)"
# A workflow's `configurationFile:` must name a file that exists. Renovate's did not after #79
# renamed renovate.json -> renovate.json5, and every scheduled run failed with ENOENT for 11 days
# while the gate stayed green. Valid YAML, so yamllint cannot see it.
missing=0
while IFS=: read -r wf _ value; do
value="${value#*configurationFile:}"
value="$(echo "$value" | sed -E 's/#.*//; s/^[[:space:]"'\'']+//; s/[[:space:]"'\'']+$//')"
if [ -n "$value" ] && [ ! -e "$value" ]; then
err "${wf}: configurationFile '${value}' does not exist in the repo"
missing=1
fi
done < <(grep -nE '^[[:space:]]*configurationFile:' .gitea/workflows/*.y*ml 2>/dev/null || true)
[ "$missing" -eq 0 ] || exit 1
ok "workflow configurationFile references resolve"
# A workflow must not use an action that needs a Docker daemon: jobs on this runner get NONE, by
# decision (decisions/2026-10-29-image-pipeline.md; act_runner maps docker_host to `-`). Such an
# action is valid YAML and passes every other stage, then dies at runtime with
# "failed to connect to the docker API at unix:///var/run/docker.sock". renovatebot/github-action
# did exactly that on every Renovate run until 2026-10-30 (run 343). Run the tool's image as the
# job `container:` instead (see .gitea/workflows/renovate.yaml). Container (`docker://`) actions
# and Docker's own daemon-bound actions are rejected for the same reason. Comments are ignored.
daemon_uses='uses:[[:space:]]*["'\'']?(docker://|renovatebot/github-action|docker/(build-push|setup-buildx|setup-qemu|login)-action)'
bad=0
while IFS= read -r hit; do
err "${hit%%:*}: needs a Docker daemon, which jobs here do not get: $(echo "${hit#*:*:}" | sed -E 's/^[[:space:]]+//')"
bad=1
done < <(grep -HnE "^[^#]*${daemon_uses}" .gitea/workflows/*.y*ml 2>/dev/null || true)
[ "$bad" -eq 0 ] || exit 1
ok "no workflow uses a Docker-daemon-bound action"
+21
View File
@@ -0,0 +1,21 @@
---
# Flux Kustomization -> ./platform/buildkit — rootless BuildKit, the image builder CI uses
# (.gitea/workflows/images.yaml). Its own Kustomization so a stuck Certificate or a failing
# builder can never stall any other tier (Flux health-checks Certificates under wait:true).
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: buildkit
namespace: flux-system
spec:
interval: 10m
retryInterval: 2m
timeout: 5m
prune: true
wait: true
dependsOn:
- name: platform # cert-manager CRDs (Issuer/Certificate) come from its HelmRelease
sourceRef:
kind: GitRepository
name: flux-system
path: ./platform/buildkit
+1
View File
@@ -98,4 +98,5 @@ resources:
# deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon.
# See clusters/base/kured.yaml and platform/kured/helmrelease.yaml.
- kured.yaml # Flux Kustomization -> ./platform/kured (dependsOn platform)
- buildkit.yaml # Flux Kustomization -> ./platform/buildkit (dependsOn platform)
- cluster-vars.yaml # overlay knob surface (base defaults)
+87 -1
View File
@@ -98,7 +98,10 @@ RUN set -eux; \
# --- runtime -----------------------------------------------------------------
ARG FEDORA_VERSION=44
FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION}
# Stage `conformance-runner` is the image that ships; stage `verify` (end of file) is a TEST stage
# built FROM it that only runs assertions and is never pushed. build-image.sh and the CI
# pipeline build `--target verify` to prove it and `--target conformance-runner` for the image.
FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION} AS conformance-runner
ARG CHAINSAW_VERSION=
ARG KUBECTL_VERSION=
@@ -198,3 +201,86 @@ WORKDIR /workspace
# prepend something the policy cannot see. `bash` is the interactive fallback for a human
# debugging the image locally.
CMD ["/bin/bash"]
# =============================================================================
# verify — a TEST stage, never shipped. Every assertion conformance/incluster/build-image.sh used
# to make with `run --user 65532 --read-only --tmpfs /workspace` now runs here, at build time,
# because CI builds on rootless BuildKit where no container runtime exists.
#
# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE:
# * uid 65532 ............ exact (USER inherited from the conformance-runner stage).
# * no network ........... exact: RUN --network=none, and the first check proves it.
# * --read-only rootfs ... NOT reproducible (a RUN step always has a writable layer).
# Approximated by permission: everything uid 65532 can write must lie
# under /workspace, /tmp or /var/tmp. In the Job only /workspace (and
# /results) are writable — the image points HOME and TMPDIR there — and
# readOnlyRootFilesystem is enforced by
# platform/kyverno/admission/conformance-trigger-job-shape.yaml.
# =============================================================================
FROM conformance-runner AS verify
RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
echo "--- no network (the build-time stand-in for the Job's isolation) ---"
if getent hosts gitea.home.keithjackson.dev >/dev/null 2>&1 || timeout 3 bash -c '</dev/tcp/1.1.1.1/443' 2>/dev/null; then
echo "UNEXPECTED: network reachable in the verify stage"; exit 1
fi
echo "confirmed: no network"
echo "--- id ---"; id
echo "--- HOME ---"; echo "HOME=${HOME}"
echo "--- bash ---"; bash --version | head -1
echo "--- python3 ---"; python3 --version
echo "--- git ---"; git --version
echo "--- kubectl ---"; kubectl version --client
echo "--- chainsaw ---"; chainsaw version
echo "--- baked versions ---"; cat /etc/conformance-runner-versions
echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
if [ -w /usr/local/bin/conformance-clone ]; then
echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
fi
echo "confirmed: not writable by uid $(id -u)"
# Each of these MUST fail. Assert the non-zero exit as well as the message: a check that only
# greps stderr would pass a build where the guard had stopped rejecting.
expect_fail() {
local label="$1"; shift
local out rc=0
out="$(env "$@" 2>&1)" || rc=$?
echo "--- ${label} ---"
printf "%s\n" "${out}" | head -2
if [ "${rc}" -eq 0 ]; then
echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
fi
echo "(rejected, exit ${rc})"
}
expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
conformance-clone --layer l2
expect_fail "conformance-clone rejects a missing SHA" \
conformance-clone
expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
CONFORMANCE_GIT_SHA=main conformance-clone
expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
expect_fail "conformance-clone rejects a HOME off the workspace volume" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
HOME=/etc conformance-clone
expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
expect_fail "conformance-clone refuses a clone target off the workspace volume" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
CONFORMANCE_SRC=/etc/src conformance-clone
echo "--- the root filesystem is not writable by the runner (expected: a failure) ---"
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi
echo "confirmed: / is not writable"
echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---"
stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \
| grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)"
if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf ' %s\n' ${stray} | head -20; exit 1; fi
echo "confirmed: no writable path outside /workspace, /tmp, /var/tmp"
echo "--- HOME is writable (the workspace emptyDir) ---"
touch "${HOME}/.probe" || { echo "UNEXPECTED: ${HOME} not writable"; exit 1; }
echo "confirmed: ${HOME} is writable"
echo "--- uid is non-zero ---"
[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; }
echo "confirmed: running as uid $(id -u), not root"
echo "CONFORMANCE-RUNNER VERIFIED"
VERIFY
+9 -4
View File
@@ -61,10 +61,15 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
> digest-pinned to the conformance-runner image, `command` exactly
> `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
> `env`/`envFrom` were the same shape of hole and are closed in the same policy.
- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
prints the exact `podman push --digestfile …` command for the operator instead. See
`conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
- `build-image.sh` — builds the above and **verifies the toolchain by running it**: the
Containerfile's `verify` stage executes every tool and every `conformance-clone` refusal as
uid 65532 with no network, at build time. It **never pushes**, and neither does anyone from a
workstation: CI (`.gitea/workflows/images.yaml`) builds the same Containerfile and `verify`
stage on rootless BuildKit (`platform/buildkit/`), and on a push to `main` publishes the
verified image under `sha-<commit>` and `main`. Renovate then moves
`CONFORMANCE_RUNNER_IMAGE_DIGEST` in `conformance/versions.env` onto the registry-returned
digest in a pull request (that file is a sensitive path, so the bump needs an
administrator's `/approve-override`).
`--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
default verify stays offline so an air-gapped build is fully checked, not half-checked).
+51 -129
View File
@@ -1,43 +1,49 @@
#!/usr/bin/env bash
###############################################################################
# conformance/incluster/build-image.sh — build the pinned `conformance-runner`
# image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
# image and prove it works. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
#
# ./build-image.sh # build + verify + print the digest and push command
# ./build-image.sh # build the verify target, then the image
# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes)
# ./build-image.sh --no-verify # build only (skip the in-image tool proof)
# ./build-image.sh --verify-clone # ALSO really clone this repo with conformance-clone.
# # Opt-in because it needs the Gitea host reachable; the
# # default verify is offline so an air-gapped build is
# # still fully checked rather than half-checked.
# ENGINE=docker ./build-image.sh # build with docker instead of podman
# IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this)
#
# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is
# not an agent's to hold, and the pushed digest is what a fail-closed admission policy
# then trusts. So this builds locally, proves the toolchain works INSIDE the image as the
# non-root uid the policy demands, and prints the exact `podman push` line for a human.
# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on
# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and
# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest
# in the job summary; Renovate then opens the pull request that moves conformance/versions.env
# onto that digest. The proof that the image works is the Containerfile's `verify` stage, which
# this script (podman locally) and CI both build.
#
# VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
# single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
# pins the toolbox gate runs with. Nothing in this file invents a version.
#
# THE DIGEST CAVEAT — read it before pasting a digest into anything.
# The digest printed here is the LOCAL manifest digest. It is normally identical to the
# registry digest (podman pushes the manifest it already has rather than recompressing),
# but that is a property of the push, not a guarantee. So the push command below uses
# `--digestfile`: the digest the registry actually returns is the authoritative one, and
# that is the one that goes into conformance/versions.env.
# NO DIGEST IS PRINTED. A manifest digest does not exist until a registry holds the manifest,
# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at
# push time (.gitea/workflows/scripts/images.sh) and Renovate carries it into
# conformance/versions.env.
###############################################################################
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
ENGINE="${ENGINE:-podman}"
TAG_OVERRIDE="${IMAGE_TAG:-}"
VERIFY=1
VERIFY_CLONE=0
PRINT_ARGS=0
for arg in "$@"; do
case "$arg" in
--no-verify) VERIFY=0 ;;
--print-build-args) PRINT_ARGS=1 ;;
--verify-clone) VERIFY_CLONE=1 ;;
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown argument: $arg" >&2; exit 2 ;;
@@ -48,8 +54,6 @@ log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
info() { printf ' %s\n' "$*"; }
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
# --- pins: sourced, never invented -------------------------------------------
TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
[ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
@@ -59,14 +63,21 @@ source "${TOOLCHAIN_ENV}"
: "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
: "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
# THE build args, in ONE place; `--print-build-args` hands them to the CI pipeline, which builds
# this Containerfile on rootless BuildKit.
BUILD_ARGS=("CHAINSAW_VERSION=${CHAINSAW_VERSION}" "KUBECTL_VERSION=${KUBECTL_VERSION}" "FEDORA_VERSION=${FEDORA_VERSION}")
if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
# --- image name: fixed by the admission policy -------------------------------
# platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
# is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here.
IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
# The tag is build convenience only (a human-readable handle for the push). Admission
# ignores it: only the digest is ever referenced from a Job.
IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}"
# The tag is a local handle only. CI re-tags what it publishes (git SHA + `main`), and
# admission ignores tags entirely: only the digest is ever referenced from a Job.
IMAGE_TAG="${TAG_OVERRIDE:-chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}}"
IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}"
log "Building ${IMAGE_REF}"
@@ -76,86 +87,31 @@ info " CHAINSAW ${CHAINSAW_VERSION}"
info " KUBECTL ${KUBECTL_VERSION}"
info " FEDORA ${FEDORA_VERSION}"
"${ENGINE}" build \
--file "${HERE}/Containerfile" \
--tag "${IMAGE_REF}" \
--build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
--build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \
--build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \
"${HERE}"
# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
LABEL_ARGS=()
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
ARG_FLAGS=()
for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
# --- verify: the point of the whole exercise ---------------------------------
# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the
# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root
# filesystem — the shape the admission policy forces. A pass here is the only evidence
# this image is usable; a build that merely succeeded is not.
# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So the proof is that
# the tools EXECUTE, as the NON-ROOT uid, with no network, that conformance-clone is not writable
# by that uid and refuses every hostile input, and that nothing outside /workspace is writable.
# That proof is the Containerfile's `verify` stage, built here and in CI alike (CI has no
# container runtime — rootless BuildKit only). See the stage's header for what a build step can
# and cannot reproduce of the Job's shape. A build that merely succeeded is not evidence.
if [ "${VERIFY}" -eq 1 ]; then
log "Verifying the toolchain inside the image (non-root, read-only rootfs)"
"${ENGINE}" run --rm \
--user 65532:65532 \
--read-only \
--tmpfs /workspace:rw,mode=1777 \
--entrypoint "" \
"${IMAGE_REF}" \
/bin/bash -euo pipefail -c '
echo "--- id ---"; id
echo "--- HOME ---"; echo "HOME=${HOME}"
echo "--- bash ---"; bash --version | head -1
echo "--- python3 ---"; python3 --version
echo "--- git ---"; git --version
echo "--- kubectl ---"; kubectl version --client
echo "--- chainsaw ---"; chainsaw version
echo "--- baked versions ---"; cat /etc/conformance-runner-versions
echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
if [ -w /usr/local/bin/conformance-clone ]; then
echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
fi
echo "confirmed: not writable by uid $(id -u)"
# Each of these MUST fail. Assert the non-zero exit as well as the message: a check
# that only greps stderr would pass a build where the guard had stopped rejecting.
# (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the
# expected failure — without it the verify dies at the first negative case.)
expect_fail() {
local label="$1"; shift
local out rc=0
out="$(env "$@" 2>&1)" || rc=$?
echo "--- ${label} ---"
printf "%s\n" "${out}" | head -2
if [ "${rc}" -eq 0 ]; then
echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
fi
echo "(rejected, exit ${rc})"
}
expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
conformance-clone --layer l2
expect_fail "conformance-clone rejects a missing SHA" \
conformance-clone
expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
CONFORMANCE_GIT_SHA=main conformance-clone
expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
expect_fail "conformance-clone rejects a HOME off the workspace volume" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
HOME=/etc conformance-clone
expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
expect_fail "conformance-clone refuses a clone target off the workspace volume" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
CONFORMANCE_SRC=/etc/src conformance-clone
echo "--- rootfs is read-only (expected: a failure) ---"
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
echo "confirmed: / is read-only"
echo "--- HOME is writable (the workspace emptyDir) ---"
touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable"
echo "--- uid is non-zero ---"
[ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root"
'
log "Verifying the toolchain (Containerfile target: verify)"
"${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}"
log "Toolchain verified"
fi
"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target conformance-runner \
--file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}"
# --- verify the clone helper against the REAL repository (opt-in: needs network) ----
# LOCAL ONLY: this one `run`s the image, which needs a container engine, so CI does not do it.
# The negative cases above prove conformance-clone refuses bad input. This proves the
# positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
# rev-parse assertion is only an unexercised line of shell.
@@ -184,46 +140,12 @@ if [ "${VERIFY_CLONE}" -eq 1 ]; then
log "conformance-clone verified against the real repository"
fi
# --- digest ------------------------------------------------------------------
DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
# --- result ------------------------------------------------------------------
IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
log "Built ${IMAGE_REF}"
info "local manifest digest ${DIGEST}"
info "image id ${IMAGE_ID}"
info "size $(( SIZE / 1024 / 1024 )) MiB"
cat <<EOF
================================================================================
NOT PUSHED. Publishing is an operator action — this script holds no registry
credential and will never acquire one.
OPERATOR: run these two commands, then paste the digest from the digestfile into
conformance/versions.env (CONFORMANCE_RUNNER_IMAGE) via a PR:
${ENGINE} login gitea.home.keithjackson.dev
${ENGINE} push --digestfile /tmp/conformance-runner.digest \\
${IMAGE_REF}
Then:
cat /tmp/conformance-runner.digest
# -> sha256:.... THIS is the authoritative digest, not the local one above.
# conformance/versions.env
CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)"
WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml
is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not
${IMAGE_REPO}@sha256:<digest>. A tag reference is denied,
and so is any other repository path.
TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard
\`${IMAGE_REPO}@sha256:*\` to be replaced with the
LITERAL digest once the image exists, closing the "anyone who can push to this path" gap.
That is a separate, human-reviewed PR against the security boundary.
================================================================================
EOF
info "image id ${IMAGE_ID}"
info "size $(( SIZE / 1024 / 1024 )) MiB"
info "Not pushed. CI publishes this image from main (.gitea/workflows/images.yaml) and"
info "Renovate moves conformance/versions.env onto the registry-returned digest."
@@ -29,7 +29,8 @@
# THEN (1) promtool is a PINNED member of the toolchain: PROMTOOL_VERSION is declared in BOTH
# infra/devcontainer/versions.env and ci/versions.env and the two AGREE (the repo's
# lockstep rule), the Containerfile takes it as a build ARG, installs it from the
# pinned prometheus release and records it for provenance, dev.sh passes it, and the
# pinned prometheus release and records it for provenance, build-image.sh (the one
# recipe `dev.sh build` and CI both use) passes it, and the
# CI runner installs it for the l0 stage;
# (2) the promtool ACTUALLY IN USE reports exactly that pinned version — a unit test is
# evidence only if the engine matches the one prod evaluates rules with;
@@ -246,10 +247,12 @@ check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text,
"infra/devcontainer/Containerfile does not record promtool in /etc/dev-toolchain-versions — "
"that provenance line is how an operator tells a stale image from a current one")
DEVSH = ROOT / "infra/devcontainer/dev.sh"
check(DEVSH.is_file() and "--build-arg PROMTOOL_VERSION=" in DEVSH.read_text(),
"infra/devcontainer/dev.sh does not pass --build-arg PROMTOOL_VERSION — the ARG would be "
"empty and the build would fetch a bogus URL")
# The build recipe moved from dev.sh into build-image.sh on 2026-10-29, so that `dev.sh build`
# and the CI image pipeline (.gitea/workflows/images.yaml) share ONE recipe. Check the recipe.
BUILDSH = ROOT / "infra/devcontainer/build-image.sh"
check(BUILDSH.is_file() and '"PROMTOOL_VERSION=$PROMTOOL_VERSION"' in BUILDSH.read_text(),
"infra/devcontainer/build-image.sh does not pass PROMTOOL_VERSION in its BUILD_ARGS — the ARG "
"would be empty and the build would fetch a bogus URL")
INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh"
if INSTALL.is_file():
@@ -42,6 +42,19 @@
# whose every solver is an automated challenge type (dns01 / http01) — the
# same code path serves issuance AND renewal, so a solver a human must
# complete would defeat both;
# (5b) the ONE exception to (5) — WORKLOAD mTLS, added 2026-10-29 — is a Certificate
# annotated `homelab.keithjackson.dev/tls-scope: workload-mtls`. The spec's TLS
# requirements are about EXTERNALLY REACHABLE services (R-TLS-01; docs/08), and
# ADR-0006 says outright that "mTLS between workloads is a separate concern … not
# covered here". A public ACME CA cannot issue for a `*.svc.cluster.local` name or a
# client-auth certificate at all. The annotation is NOT trusted: an exempt
# Certificate must PROVE it is workload-internal — its Secret is served by NO Gateway
# listener, every DNS name / commonName is cluster-internal (a bare label or
# `*.svc[.cluster.local]`), it carries no IP/URI/email SANs, and its issuer is a
# NAMESPACED Issuer in its own namespace whose spec is `selfSigned` or a `ca` backed
# by a Secret that a tracked, likewise-exempt Certificate produces. Issuance and
# renewal stay unattended (cert-manager renews CA-issued certs itself), and (9)
# still applies. Every exemption is printed. First user: platform/buildkit.
# (6) each dns01 solver's provider credential is referenced BY NAME and that
# Secret is delivered SOPS-encrypted from git via a kustomization — the
# operator's one-time token hand-off (docs/08 OQ-4), never a per-cert step;
@@ -265,6 +278,7 @@ CERT_SECRETS: dict[tuple[str, str], str] = { # (namespace, secretName)
for rel, d in CERTS
}
listeners_checked = 0
GATEWAY_SECRETS: set[tuple[str, str]] = set() # every (namespace, Secret) a listener serves
for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
gw_ns = ns_of(gw)
for lis in ((gw.get("spec") or {}).get("listeners") or []):
@@ -280,6 +294,7 @@ for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
for r in refs:
kind = str(r.get("kind", "Secret"))
key = (str(r.get("namespace") or gw_ns), str(r.get("name")))
GATEWAY_SECRETS.add(key)
check(kind == "Secret",
f"{rel}: listener {lis.get('name')!r} certificateRef kind is {kind!r}; only "
"Secret refs can be traced to a cert-manager Certificate")
@@ -324,8 +339,61 @@ CM_NS = ns_of(cm_releases[0][1], "cert-manager") if cm_releases else "cert-manag
ISSUER_INDEX = {(d.get("kind"), name_of(d)): (rel, d) for rel, d in ISSUERS}
solver_creds: list[tuple[str, str, str, str]] = [] # (issuer, ns, secretName, ref field)
issuers_verified: set[str] = set()
# (5b) workload-mTLS exemption — see the header. Proven, never taken on trust.
SCOPE_ANN = "homelab.keithjackson.dev/tls-scope"
INTERNAL_NAME = re.compile(r"^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*$")
NS_ISSUERS = {(ns_of(d), name_of(d)): (rel, d) for rel, d in of_kind("Issuer", "cert-manager.io/")}
EXEMPT_SECRETS = {(ns_of(d), str((d.get("spec") or {}).get("secretName", "")))
for _, d in CERTS
if ((d.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN) == "workload-mtls"}
exempted: list[str] = []
def workload_mtls_ok(rel: str, cert: dict) -> bool:
"""True if `cert` claims the workload-mTLS scope AND proves it; records failures."""
ann = ((cert.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN)
if ann is None:
return False
cns, cname = ns_of(cert), name_of(cert)
spec = cert.get("spec") or {}
ok = True
def need(cond, msg):
nonlocal ok
check(cond, f"{rel}: Certificate {cns}/{cname} claims {SCOPE_ANN}={ann!r} but {msg}")
ok = ok and bool(cond)
need(ann == "workload-mtls", "the only accepted value is 'workload-mtls'")
need((cns, str(spec.get("secretName", ""))) not in GATEWAY_SECRETS,
"its Secret is served by a Gateway listener — that is external TLS, which must be ACME (R-TLS-01)")
names = [str(n) for n in (spec.get("dnsNames") or [])]
if spec.get("commonName"):
names.append(str(spec["commonName"]))
for n in names:
internal = INTERNAL_NAME.match(n) and ("." not in n or n.endswith(".svc") or n.endswith(".svc.cluster.local"))
need(bool(internal), f"name {n!r} is not cluster-internal (bare label or *.svc[.cluster.local])")
for f in ("ipAddresses", "uris", "emailAddresses"):
need(not spec.get(f), f"it carries {f}, which a workload-internal certificate has no use for")
ref = spec.get("issuerRef") or {}
need(str(ref.get("kind", "Issuer")) == "Issuer",
"its issuer is not a NAMESPACED Issuer — a ClusterIssuer could sign for any namespace")
hit = NS_ISSUERS.get((cns, str(ref.get("name", ""))))
need(hit is not None, f"its Issuer {ref.get('name')!r} is not tracked in namespace {cns!r}")
if hit is not None:
ispec = hit[1].get("spec") or {}
kinds = sorted(ispec)
need(kinds in (["selfSigned"], ["ca"]),
f"its Issuer is {kinds} — only selfSigned or ca are accepted for workload mTLS")
if kinds == ["ca"]:
need((cns, str((ispec.get("ca") or {}).get("secretName", ""))) in EXEMPT_SECRETS,
"its CA Issuer's Secret is not produced by a tracked workload-mtls Certificate "
"in the same namespace — the CA itself would have to be minted by hand")
if ok:
exempted.append(f"{cns}/{cname}")
return ok
for rel, cert in CERTS:
spec = cert.get("spec") or {}
if workload_mtls_ok(rel, cert):
continue # (5b): proven workload-internal; (9) below still applies
ref = spec.get("issuerRef") or {}
kind = str(ref.get("kind", "Issuer"))
per_env = resolve_per_env(str(ref.get("name", "")))
@@ -379,6 +447,8 @@ for rel, cert in CERTS:
solver_creds.append((iname, want_ns, n, f"{provider}.{field}"))
check(len(issuers_verified) > 0, "no issuer was reachable from any Certificate — (5) proved nothing")
if exempted:
print(f" (5b) workload-mTLS certificates exempted from ACME after proving it: {', '.join(exempted)}")
# (6) the solver credential is delivered, SOPS-encrypted, from git — never per-cert by hand.
check(len(solver_creds) > 0,
+18 -21
View File
@@ -40,32 +40,29 @@ PYTHON_MIN="3.9"
# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*
# Any other path, and any TAG reference, is DENIED at admission. Do not "tidy" these values.
CONFORMANCE_RUNNER_IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
# Build handle only. Admission never sees the tag; a Job must reference the digest.
# Local build handle only. Admission never sees the tag; a Job must reference the digest.
# CI publishes under `sha-<commit>` and `main` instead (.gitea/workflows/images.yaml).
CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
#
# ############################################################################
# # DIGEST DELIBERATELY EMPTY — THE IMAGE HAS NOT BEEN PUBLISHED YET. #
# # THE DIGEST IS WRITTEN BY RENOVATE, NOT BY HAND. #
# ############################################################################
# The image has been BUILT and its toolchain verified locally (non-root uid 65532,
# read-only rootfs: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4, chainsaw 0.2.12,
# and `./run.sh --layer l2 --no-cluster` completing with rc 0). It has NOT been pushed:
# pushing needs a registry credential, and publishing the artifact a fail-closed security
# boundary then trusts is an OPERATOR action, not an agent's.
# CI builds and verifies the image (non-root uid 65532, read-only rootfs, the
# conformance-clone refusals) and, on a push to main, publishes it and prints the digest the
# REGISTRY returned (.gitea/workflows/images.yaml). Renovate follows the image's `main` tag
# through the marker below and opens the pull request that moves this value. Nobody pushes
# this image from a workstation, and nobody pastes a digest here.
#
# OPERATOR, to complete the pin:
# 1. ./conformance/incluster/build-image.sh # rebuild, verify, print the commands
# 2. podman login gitea.home.keithjackson.dev
# 3. podman push --digestfile /tmp/conformance-runner.digest \
# "${CONFORMANCE_RUNNER_IMAGE_REPO}:${CONFORMANCE_RUNNER_IMAGE_TAG}"
# 4. Set CONFORMANCE_RUNNER_IMAGE_DIGEST below to `cat /tmp/conformance-runner.digest`
# — the digest the REGISTRY returned, not the local manifest digest the build printed.
# 5. Land steps 4 in a PR (this file is GitOps input, not a live edit).
#
# UNTIL THEN: this is empty on purpose, NOT a placeholder to be guessed at. An empty value
# makes any consumer that renders the Job fail visibly; a fake or stale digest would instead
# produce an ImagePullBackOff that looks like a cluster fault. Do not invent one.
CONFORMANCE_RUNNER_IMAGE_DIGEST=""
CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_DIGEST:+${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}}"
# ALL-ZERO UNTIL THE FIRST PUBLISH. Before 2026-10-29 this was EMPTY, so that a consumer
# rendering the Job would fail visibly. It is now the same impossible all-zero digest
# platform/pm-agent/ uses, because Renovate can only replace a digest that is there to match.
# A consumer MUST treat the all-zero value as "not published" (no consumer exists yet: the
# L2/L3 trigger is unwired, docs/plans/l2-l3-conformance-ci-wiring.md). Note also that
# chainsaw-job.yaml's @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ takes the 64 hex digits WITHOUT the
# `sha256:` prefix this value carries — the trigger must strip it.
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/conformance-runner tag=main
CONFORMANCE_RUNNER_IMAGE_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000"
CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}"
#
# AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for
# its `@sha256:*` wildcard to be replaced with the LITERAL digest once the image exists —
+374
View File
@@ -0,0 +1,374 @@
---
date: 2026-10-29
branch: feat/image-pipeline
title: "Images are built, verified and published by CI on rootless BuildKit; Renovate carries the digests into git"
---
## What changed
Operator rule (2026-10-29): container image creation, hosting and lifecycle belong to CI/CD and
the codebase, never to a one-off workstation build/push. Before this branch the Home-Network org's
package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and
both `build-image.sh` scripts said "publishing is an operator action".
### The builder: rootless BuildKit — DECISION MADE (operator, on PR #81)
**The problem it answers, measured on this PR's first CI run (images run 325, failed in 6 s):** a
CI job on this runner gets **no Docker daemon at all**, for two reasons, both in the deployed
versions' source:
- act_runner v0.2.11 `internal/app/cmd/daemon.go` sets `container.docker_host` to `-` ("never
mount into jobs") whenever `DOCKER_HOST` is `tcp://`, which it is here (the DinD sidecar);
- gitea/act v0.261.2 `pkg/container/docker_run.go` `sanitizeConfig` drops **every** bind,
socket included, when `valid_volumes` is empty, which it is (the runner has no config file).
Giving jobs the DinD daemon would have made every branch push root on the node, because that
daemon is `privileged: true`.
**The decision:** no job may reach a privileged daemon, and images are built by a **rootless
BuildKit** daemon over mTLS. The runner (`apps/wave-1/gitea-actions.yaml`, DOCKER_HOST,
valid_volumes) is **untouched**.
- **`platform/buildkit/`** — a new hand-managed component in its own namespace `buildkit`. The
runner manifest is renderer-generated (`catalog/render.py` EXTRA_UNITS), and the builder has no
business sitting next to a privileged container. Flux reconciles it as the new Kustomization
`buildkit` (`clusters/base/buildkit.yaml`, `dependsOn: platform`).
- **Deployment:** `moby/buildkit:v0.34.1-rootless@sha256:f68752a5…0c4f89`, pinned by index
digest. The securityContext, every line measured (see *What I verified*):
- `privileged: false`;
- uid/gid 1000, `runAsNonRoot`;
- `capabilities: drop ALL, add SETUID, SETGID`;
- `allowPrivilegeEscalation: true`;
- `seccompProfile: Unconfined` and `appArmorProfile: Unconfined`;
- `readOnlyRootFilesystem: true`, with five emptyDirs.
Other settings:
- `--oci-worker-no-process-sandbox`;
- no ServiceAccount token;
- memory limit 4Gi, which covers build steps too;
- cache emptyDir with `sizeLimit: 6Gi`, and GC capped at 4 GB in `buildkitd.toml`
(ConfigMap).
- **mTLS via cert-manager** (`certificates.yaml`): a self-signed CA used only here, the server
cert `buildkitd-tls`, and the client cert `buildkit-client-ci`. buildkitd verifies every
client against that CA.
- **Service:** ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`.
- **NetworkPolicies:**
- default-deny;
- ingress on 1234 **only from the act-runner pod** (`gitea-actions`,
`app.kubernetes.io/name=act-runner`);
- egress: DNS, world :443/:80 for downloads, and the `ingress` entity (the Gateway) for the
registry push.
- `versions.env`: the CI-side client, `buildx` v0.38.1 with the sha256 from its
`checksums.txt`, plus the address and certificate name CI dials.
- **How DinD job containers appear to Cilium (measured, read-only).**
`iptables -t nat -S POSTROUTING` in the dind container shows
`-A POSTROUTING -s 172.18.0.0/16 ! -o br-… -j MASQUERADE` (and the same for docker0). So a job's
connection leaves with the **act-runner pod's IP, and therefore that pod's Cilium identity**.
That makes "only the act-runner pod" expressible and correct, but it cannot tell one job from
another, or a job from act_runner itself. Per-job authentication is the client certificate.
- **Admission: no Kyverno exception needed.**
- Every ClusterPolicy actually applied (`platform-config/kustomization.yaml`; live:
`conformance-trigger-job-shape`, `label-cdi-worker-pods`, `pm-agent-job-shape`,
`require-auth-mode-label`, `require-default-deny-baseline`, `require-egress-class-label`,
`restrict-host-namespaces`, `secret-provenance`, `validate-default-deny-shape`) matches only
namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`, or only its own
namespace in the job-shape policies' case. `buildkit` carries no such label.
- There are no namespaced Kyverno Policies and no cluster-wide PSA default (both measured).
- The namespace is deliberately left without a PSA label, because PSS `baseline` forbids
`Unconfined` seccomp and AppArmor.
- `validate-default-deny-shape` does apply to the namespace's default-deny NetworkPolicy, and
its shape (both directions) satisfies it.
- **Unprivileged user namespaces on the nodes (measured, read-only, via Ansible `sysctl`).** On
all three Debian 13.7 nodes, kernel `7.1.8+deb13-amd64`:
- `kernel.unprivileged_userns_clone = 1`;
- `user.max_user_namespaces = 127206`;
- no `kernel.apparmor_restrict_unprivileged_userns` (an Ubuntu-only key);
- AppArmor enabled (`Y`).
So this is a satisfied prerequisite, not an open one.
- **The TLS conformance check needed a narrow, self-proving exception** (`conformance/**`, so this
changes what green means; flagged for the reviewer).
- The first gate run failed L0 `tls-lifecycle-automated` clause (5): "every Certificate's
issuer must be ACME". That is broader than the spec it serves:
- R-TLS-01 and docs/08 concern *externally reachable* services;
- ADR-0006 says "mTLS between workloads is a separate concern … not covered here";
- a public CA cannot issue for a `*.svc.cluster.local` name or a client-auth certificate.
- The new clause (5b) exempts a Certificate annotated
`homelab.keithjackson.dev/tls-scope: workload-mtls` **only if it proves it**:
- no Gateway listener serves its Secret;
- every DNS name and the commonName are a bare label or `*.svc[.cluster.local]`;
- it has no IP, URI or email SANs;
- its issuer is a namespaced `selfSigned`/`ca` Issuer in its own namespace;
- a `ca` issuer's Secret is produced by a likewise-exempt tracked Certificate.
- Exemptions are printed, and clause (9), the renewal arithmetic, still applies.
- **Negative-tested:** a public DNS name, and a Gateway serving `buildkitd-tls`, each fail the
check with a specific message.
- **`infra/gitea/buildkit-client-secret.sh`** (operator action): copies `ca.crt`, `tls.crt` and
`tls.key` from the Secret `buildkit/buildkit-client-ci` into the Actions secrets `BUILDKIT_CA`,
`BUILDKIT_CLIENT_CERT` and `BUILDKIT_CLIENT_KEY`. Before writing anything it checks that:
- the cert verifies against the CA;
- the cert is a client-auth cert;
- the key matches the cert.
It never prints a value. `--check` reports whether the stored copy is older than the current
certificate.
### Verification moved into the Containerfiles
`docker run` cannot exist on BuildKit, so each Containerfile now names its shipped stage after the
image (`toolbox`, `agent-runner`, `conformance-runner`) and ends in a **`verify` test stage**,
`FROM <image> AS verify`. That stage runs every previous assertion at build time, as the image's
non-root uid (65532 for agent-runner and conformance-runner), in `RUN --network=none`. It is never
pushed.
- **Every existing assertion is preserved**, and the toolbox gained one it never had: every tool
on PATH, and five tools report their pinned versions.
- `probe-validation.sh` is `COPY`ed into the verify stage only, so it never ships.
- **Bug fixed on the way:** in the old `docker run` block, `cmd | grep -q … && echo ok` and
`[ … ] && echo` could never fail the script under `set -e` + `pipefail`. The "entrypoints refuse
arguments", "not root", "no API key baked in" and "HOME writable" checks asserted nothing. They
are real assertions now.
- **What a build step cannot reproduce, and how it is approximated** (also written in each stage's
header):
- **uid:** exact.
- **`--network none`:** exact, and each stage *proves* it. Its first check fails if DNS or a
TCP connection works. A negative control without `--network=none` fails that check.
- **`--read-only` rootfs:** not reproducible, because a RUN step always has a writable layer.
It is approximated by permission:
- `touch /should-fail` must fail;
- a `find / -xdev -writable ! -type l` inventory, run as the uid, must lie entirely under
`/workspace`, `/tmp` or `/var/tmp`. `! -type l` was needed: Fedora's `/usr/tmp` symlink
to `/var/tmp` tripped it.
The real control remains the pods' `readOnlyRootFilesystem`, which the admission policies
enforce.
- **`--tmpfs /workspace`:** approximated by the image's own `/workspace`, owned by 65532, which
is exactly what the pod's emptyDir replaces.
The opt-in `--verify-clone` still `run`s the image, so it remains local-only.
- **`build-image.sh` (all three):**
- verify is `build --target verify` and the image is `build --target <image>`;
- a new `--print-build-args` prints the pins as `KEY=VALUE`, which is what CI consumes. The
arg list therefore exists once;
- the local podman path still works.
- The L0 check `alert-rules-unit-tested` now looks for `"PROMTOOL_VERSION=$PROMTOOL_VERSION"` in
the toolbox's build-arg list.
### The pipeline (`.gitea/workflows/images.yaml` + `scripts/images.sh`)
- **The builder.** The driver downloads the pinned `buildx` and refuses a checksum mismatch. It
writes the client cert from the `BUILDKIT_*` secrets to a 0600 temp dir that is removed on exit,
creates a `remote` builder, and runs `buildx inspect --bootstrap`.
- **It fails loudly, never skips.** It stops with a clear message if:
- the secrets are absent;
- buildkitd is unreachable. The message lists the causes in order: not deployed yet (Flux
applies it only after merge), stale certificate, NetworkPolicy.
- **Pull request:** `buildx bake <image>-verify` for changed images. Nothing is pushed and no
registry credential is referenced. The `BUILDKIT_*` builder secrets *are* given to PR jobs,
because they must build.
- agent-runner is built FROM the toolbox built **in the same run** through a bake `target:`
context. That toolbox exists in no registry, so the toolbox is built as its base even when
its own inputs did not change.
- **Push to main / dispatch on main:** for each image, the verify target first, and only then the
image itself with `push=true`. Tags are `sha-<12 hex>` (immutable) and `main` (moving).
- The digest is read back from the registry and compared with the client's metadata.
- agent-runner builds FROM the toolbox digest just pushed, or else from the pinned
`AGENT_BASE_IMAGE` (the zero placeholder bootstraps).
- `BUILDX_NO_DEFAULT_ATTESTATIONS=1` keeps one manifest per tag, which is what admission,
Renovate and the retention script assume.
- **Unchanged from the first iteration:** change detection, publish idempotence,
`REGISTRY_PUSH_TOKEN` and its preflight, the Renovate markers, and `registry-retention.sh`. The
DinD prune step is gone, because nothing is built in DinD now.
### Unchanged from the first iteration (still true)
- **Digest references** carry `# renovate-digest: image=<repo> tag=main` markers:
- `platform/pm-agent/deployment.yaml` and `configmap.yaml`;
- `infra/agent/versions.env` `AGENT_BASE_IMAGE`, now the published toolbox by digest, zero
until the first publish;
- `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to zero.
The admission policies are untouched.
- **`renovate.json5`**:
- the `custom.regex` marker manager;
- `enabledManagers` `regex` → `custom.regex`;
- the `kubernetes` manager disabled for these three repos;
- `ignorePaths` narrowed so `conformance/versions.env` is visible.
`renovate.yaml` pins `renovate-version: 44.167.0` and adds a docker `RENOVATE_HOST_RULES` entry
using `RENOVATE_TOKEN`, which now needs `read:package`.
- **`infra/gitea/registry-retention.sh`** (dry run by default; operator action); the
`conformance-trigger-kubeconfig.sh` precondition fix; and the docs (pm-agent README, runbook,
plan, conformance README, devcontainer README, workflows README). The docs are updated again
for BuildKit, and there is a new `platform/buildkit/README.md`.
### Decisions and pushback
- **The push credential is a PAT secret, `REGISTRY_PUSH_TOKEN`; the Actions token cannot push.**
Checked against Gitea **v1.24.3** source:
- `services/auth/basic.go` `Basic.Verify` turns a task token into the pseudo-user
`gitea-actions` (ID −2) and sets no token scope;
- `routers/api/packages/container/container.go` `Authenticate` (the `/v2/token` exchange every
docker client performs) requires `read:package`/`write:package`/`all` scope, so it answers
401. The Actions token cannot even *pull*;
- `services/context/package.go` `determineAccessMode` has `// TODO: ActionUser permission check`
and no branch for that user. Even past login it would get at most read on a public org;
- `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3.
The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight
pattern. With BuildKit the credential reaches the daemon through the buildx session (a
0600 `config.json` in the job's temp dir), never through the cluster.
- **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate:
- `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` →
`custom.regex`;
- that migration runs on every repository run (`lib/workers/repository/init/merge.ts` →
`migrateAndValidate`), not only in the validator;
- `lib/modules/manager/index.ts` `getEnabledManagersList` accepts either spelling;
- `custom.regex` exists since 37.51.1 (renovatebot/renovate#24079).
- **The open question hid a bigger bug: the Renovate config on main is invalid for the Renovate
the workflow actually runs.**
- `renovatebot/[email protected]` defaults `renovate-version` to `39`
(`action.yml`, `src/docker.ts`), which is 39.264.0.
- The validator used in #79 was `ghcr.io/renovatebot/renovate` with no tag, i.e. `latest`,
44.167.0.
- **Measured:** `renovate:39.264.0 renovate-config-validator` on main's `renovate.json5` gives
`ERROR: Found errors in configuration` with five errors, all `managerFilePatterns`.
`fileMatch` was renamed in 40.2.0.
- So once #80 fixes the filename, every run would still die at `CONFIG_VALIDATION`.
- Fixed by pinning `renovate-version: 44.167.0`. This branch's config validates clean on it.
The only `--strict` complaint is a pending migration of the pre-existing top-level
`"docker": {"enabled": true}` block, which I left alone.
- **Unworkable as briefed: tracking the worker Job template's digest.**
- `infra/agent/worker-job-template.yaml` is `COPY`ed into agent-runner. A Renovate bump there
changes agent-runner's inputs, CI publishes a new digest, Renovate bumps it again, and so on
forever.
- `pm-dispatch` overwrites that field from `PM_WORKER_IMAGE` at runtime
(`pm-dispatch.py:649`), so nothing is lost by leaving it all-zero. It is commented as
deliberately untracked.
- For the same loop reason, `conformance/versions.env` is **not** an input of the
conformance-runner build, and a pipeline-only change does not republish on main.
- **Changed from the brief: retention is a script, not a Gitea cleanup rule.**
- Gitea 1.24.3 has **no API** for cleanup rules (none in `routers/api/v1/api.go`; web
session+CSRF forms only, in `routers/web/web.go`). So a rule cannot be created, or
drift-asserted, as code. **`ci/gitea-settings.sh` was therefore not extended.** A cheaper
proxy (count versions per image) would need `read:package` on `SETTINGS_API_TOKEN`, which
it lacks.
- A built-in rule is also **unsafe here**. Gitea stores a manifest as a file of the TAG
version it was pushed under (`routers/api/packages/container/manifest.go`
`createPackageAndVersion`), and resolves a by-digest pull from those files
(`models/packages/container/search.go`). Deleting the tag a deployed digest was pushed under
breaks the `@sha256:` pull, and the rule's "newest N" knows nothing about what git pins. The
script keeps every digest git pins.
- **Auto-merge.** Digest bumps of these images auto-merge on a green gate (the existing
`digest` rule). The exception is the conformance-runner bump: `conformance/versions.env` is a
sensitive path, so it waits for an administrator's `/approve-override` every time.
## What I verified
**Proved locally, against the exact pinned images:**
- **Rootless buildkitd, configured as the manifest configures it.**
- Setup: the pinned `moby/buildkit:v0.34.1-rootless` digest, under rootless podman, with
`--cap-drop ALL --cap-add SETUID --cap-add SETGID`, seccomp unconfined and `--read-only`;
tmpfs at the same five paths; the ConfigMap's `buildkitd.toml`; mTLS on :1234.
- The image's `/etc/subuid` was overridden with `user:1:999` and `user:1001:64535`, because
rootless podman maps only 65,537 uids. rootlesskit's real `newuidmap` user-namespace path
still ran.
- It started, `buildctl debug workers` listed the OCI worker, and `debug workers -v` showed the
GC config applied (max 4.29 GB, reserved 1.07 GB).
- **The securityContext, item by item.** Each requirement was measured by taking it away:
- **default seccomp:** daemon starts, every RUN step fails → `Unconfined` is required;
- **no `SETUID`/`SETGID`:** rootlesskit dies with
`newuidmap … operation not permitted` → both capabilities are required;
- **`--read-only` rootfs:** works once the five emptyDirs exist. Its only failure was the
probe client's `~/.docker`, which is now an emptyDir;
- **minimal caps + Unconfined seccomp:** works, and a RUN step executes.
- **The full CI path, in the actual CI job image.** Setup: `ghcr.io/catthehacker/ubuntu:act-22.04`
running `.gitea/workflows/scripts/images.sh` in verify mode. It downloaded and sha256-checked
buildx v0.38.1, connected over mTLS, and bake-built all three `verify` targets from an empty
cache. rc 0, 1m32s total:
- `TOOLBOX VERIFIED` (42 s);
- `AGENT-RUNNER VERIFIED` (26 s): selftest OK, probe "20 refusal(s), 0 failure(s)", and built
FROM the in-run toolbox through the `target:` context, with the log showing
`[agent-runner-verify toolbox 1/6]` and `base=toolbox-in-this-run`;
- `CONFORMANCE-RUNNER VERIFIED` (21 s): all seven refusals;
- every stage printed `confirmed: no network`.
- **mTLS rejects what it should.** Tested with a real BuildKit API call (`buildctl debug workers`):
- no client cert → `remote error: tls: certificate required`;
- a client cert from another CA → rejected the same way;
- the valid cert → accepted.
Through the driver:
- another CA's client cert → rc 1;
- the wrong CA for the server → rc 1;
- an unreachable host → rc 1 in about 20 s, with the explanatory message;
- missing secrets → rc 1, naming the three secrets.
- **The local podman path**, `build-image.sh` for all three (rc 0).
- Negative control: a verify stage expecting a different promtool version printed
`DRIFT promtool want 9.9.9` and failed the build.
- Negative control: the `--network=none` check fails when the flag is removed.
- **`buildkit-client-secret.sh`'s sanity checks**, run offline on local certs. They tell apart:
- a good client cert;
- a server cert (not client-auth);
- a mismatched key;
- a cert from another CA.
**Measured live, read-only:**
- node user namespaces, Debian 13.7 and AppArmor (via an Ansible `sysctl`/`cat`);
- DinD's MASQUERADE rules;
- the live ClusterPolicies, namespaced Policies, PSA labels, and cert-manager v1.21.3 with its
ClusterIssuers;
- the act-runner pod labels.
**Renovate (unchanged since the first iteration):**
- the validator with 44.167.0: exit 0;
- `extract` finds all four markers;
- the `kubernetes` duplicates are `skipReason: disabled`;
- 39.264.0 rejects `main`'s config.
**Local gate, CI results:** see the PR body. They are recorded there because this entry is part of
the tree being gated.
## What I did NOT verify
- **platform/buildkit has never run in the cluster.** It can't until it merges: agents may not
`kubectl apply`, and Flux reconciles only `main`. So this PR's own `images` job **cannot build
and is expected to fail at the builder preflight**, which is the designed loud failure. The
following are therefore all unproven in-cluster:
- kubelet admitting the pod;
- containerd honouring `appArmorProfile: Unconfined`;
- the five emptyDirs;
- cert-manager issuing the three certs;
- the CiliumNetworkPolicies.
- **AppArmor `Unconfined` is reasoned, not measured.** My test host runs SELinux, so the AppArmor
requirement comes from upstream's example plus the nodes having AppArmor enabled. If the pod
were to work under `RuntimeDefault` AppArmor, that line could be tightened; I did not test it.
- **The local userns mapping is narrower than the cluster's.** Locally it was 65,535 ids across
two ranges; in the cluster it is the image's own `100000:65536` against real host subuids. That
is the normal case, but it is the cluster's first test.
- **The NetworkPolicies are untested.** In particular:
- that `toEntities: [ingress]` admits the push to the Gateway VIP. The comment explains the
reasoning, but it is unmeasured;
- that world :443/:80 covers every download the builds make. A dnf mirror on another port
would fail a build visibly.
- **No push, anywhere.** The whole publish path is unexercised:
- `REGISTRY_PUSH_TOKEN` reaching buildkitd through the buildx session;
- `push=true`;
- the metadata digest compared with the registry read-back;
- agent-runner FROM `toolbox@sha256:`;
- the immutable-tag guard;
- idempotence.
So are the 15 s Envoy route-timeout risk on blob finalisation, and the Gitea PVC (10Gi, shared
with repositories) absorbing ~2 GB per toolbox publish until retention runs.
- **Node disk under real builds.** The cache emptyDir (6Gi limit, 4 GB GC cap) sits on nodes that
had 7–9 GB free. It is untested under concurrent pressure.
- **Certificate rotation is designed, not exercised.** The server cert renews 30 days early but
buildkitd only reads it at start; the client copy needs the script re-run.
- **`--oci-worker-no-process-sandbox` is an accepted risk, not a mitigated one.** A build step can
ptrace the daemon and read its TLS key and cache. Anyone with the client cert, which means every
PR job, can run build steps.
- **Renovate has still never completed a run.** It is blocked on #80, on `RENOVATE_TOKEN` gaining
`read:package`, and on this branch. The `[email protected]` + 44.x image pairing is
untested.
- **Pods cannot pull these images until PR #61** (the kubelet pull secret) merges.
- **`registry-retention.sh --apply` has never run**; its keep logic has only run against an empty
registry.
@@ -0,0 +1,57 @@
---
date: 2026-10-29
branch: fix/renovate-config-filename
title: Renovate has failed every run since #79 — the workflow still named renovate.json
---
## What changed
**`.gitea/workflows/renovate.yaml`** — `configurationFile: renovate.json` → `renovate.json5`.
PR #79 (`fix/renovate-invalid-config`) renamed the config to `renovate.json5` and did not update
the one line in the workflow that names it. Measured 2026-10-29 from the Gitea Actions log of run
322 (task 513, schedule, 08:15 UTC):
Error: ENOENT: no such file or directory, stat '/workspace/Home-Network/home-network-k3s-impl/renovate.json'
Runs 315–322 (2026-10-22 → 10-29, every one listed) are all `failure` on the `schedule` event,
each failing in about one second. Renovate never started. `RENOVATE_TOKEN` **does** exist as a
repository Actions secret (listed by name via the API; value not read), so the token preflight
passes and this filename is the only blocker observed.
**`ci/lint.sh`** — a new assertion after yamllint: every non-comment `configurationFile:` key in
`.gitea/workflows/*.y*ml` must name a path that exists. This is valid YAML, so yamllint cannot see
it, and the gate stayed green through 11 days of failures.
**Second blocker, same PR:** the workflow did not pin a Renovate version, so
`renovatebot/[email protected]` ran its default, Renovate **39** (`action.yml`:
`renovate-version` `default: '39'`, read at that tag). `renovate.json5` uses `managerFilePatterns`,
which needs >= 40.2.0. Under 39.264.0 the validator rejects the config with five errors. This was
found and reproduced by the image-pipeline worker (PR #81). #79 had validated against `latest`. Pinned
`renovate-version: 44.167.0`, the version the config validates clean against per that worker.
## What I verified
- The root cause, from the run log quoted above, and that `origin/main` has `renovate.json5` and
no `renovate.json` (`git ls-tree`).
- The new guard, both ways, run standalone: on this branch it prints
`workflow configurationFile references resolve`, rc 0. With the line reverted to `renovate.json`
it prints `configurationFile 'renovate.json' does not exist in the repo`, rc 1. My first version
also matched its own explanatory comment in the workflow and failed. That is why the grep is
anchored to non-comment lines.
- Toolbox gate: result and rc are reported in the PR, not predicted here.
## What I did NOT verify
- **The 39.x rejection and the 44.167.0 pass were not reproduced by me.** I confirmed only the
action's default (`'39'`). The validation results are the PR #81 worker's.
- **Renovate has still never completed a run.** This removes the observed blocker. Others may
follow it. The test is a `workflow_dispatch` with `dry_run: true` after merge, and its log is
the first evidence that Renovate authenticates, extracts dependencies, and honours `.json5`.
- **The `enabledManagers: ["regex"]` vs `custom.regex` question** (#73, #77) is still open. The
same dry-run log answers it: look for the regex/custom manager's dependency-extraction count.
- **No alert for "Renovate keeps failing".** The guard prevents this specific cause from recurring.
Nothing notices a scheduled workflow that is red every day. That gap stays open.
- **Sensitive path.** `.gitea/**` and `ci/**` are both in `approval-policy.json`. CI will block
on `approval-policy` until the operator posts `/approve-override` as a review. The local gate
reports that stage as N/A.
@@ -0,0 +1,81 @@
---
date: 2026-10-30
branch: fix/gitea-admit-buildkit
title: The registry push 403 is Cilium refusing buildkitd's EGRESS at the Gateway — fixed in platform/buildkit, not in Gitea's ingress
---
## What changed
**`platform/buildkit/networkpolicy.yaml`** — `buildkitd-allow-egress` gains one rule:
`toEndpoints {io.kubernetes.pod.namespace: gitea, app.kubernetes.io/name: gitea}` on TCP 3000,
with a comment carrying the measurement. `platform/buildkit/README.md`'s policy row says the same.
**What did NOT change, deliberately:** `catalog/gitea.yaml` `inClusterClients` and
`apps/wave-0/gitea.yaml`. The brief proposed admitting `buildkit` there. The evidence below says
that is the wrong policy: Gitea's ingress is not what refuses the push, and adding it would only
open the direct ClusterIP path from build code to Gitea, which nothing uses.
### Why: the brief's hypothesis was half right
Images run 337 failed its push with `HEAD .../blobs/sha256:5855... 403 Forbidden: Access denied`.
The PM's hypothesis was that this was Cilium's Envoy denying the request, not Gitea. **That part
holds.** The hypothesis also said the denial came from Gitea's *ingress* policy, which does not
admit the `buildkit` namespace. **That part is wrong.** The denial is enforced on buildkitd's own
*egress*. Since Cilium 1.16.8/1.17.2 (CVE-2025-30162; prod runs `cilium:v1.20.1`), the Gateway's
L7 load balancer re-evaluates the source pod's egress policy against the identity of the backend
it resolves to. buildkitd's egress allowed `world` :443/:80 and the `ingress` entity. That gets
the TCP connection to Envoy, but no rule matches Gitea's backend identity, so Envoy answers
`403 Access denied` without forwarding the request. This is the failure commit `68690e5` fixed for
every SSO app's hop to Keycloak, and the fix has the same shape. Keycloak's ingress names no app
namespaces either.
## What I verified
All measurements are read-only, against `k3s-prod`, on 2026-10-30 around 05:27–05:37 UTC.
- **Gitea never saw the failed push.** Its router log for the previous 12h had 316 `/v2/` lines
and **0** that were `toolbox/blobs`, PUT, POST or PATCH. It also had **0** from buildkitd's pod
IP (`10.0.0.7`). In the same window it logged the CI job's own `HEAD .../toolbox/manifests/
sha-0dd57fbaf002 → 404` from the act-runner pod (`10.0.0.38`). Requests that came through the
Gateway reach Gitea and get logged. This one never arrived.
- **A reproduction with no credentials.** An unauthenticated `GET https://gitea.home.keithjackson.dev/v2/`
(Gateway VIP `192.168.2.20`) gave:
- from the buildkitd pod: **`403 Forbidden`**, with no matching line in Gitea's log;
- from the act-runner pod's `dind` container: **`401 Unauthorized`**, which is Gitea's normal
registry challenge, logged by Gitea as `GET /v2/ for 10.0.0.38:0, 401`.
No token was involved, so this cannot be a package-permission problem. The bot account and the
`REGISTRY_PUSH_TOKEN` were not the cause, and the operator does not need to change them.
- **Hubble names the enforcement point.** On `cilium-55jv8`, buildkitd's node:
`buildkit/buildkitd-859d67c8f8-x44nh (ingress) -> gateway/cilium-gateway-shared:443 http-request
DROPPED (HTTP/1.1 GET https://gitea.home.keithjackson.dev/v2/)`, followed by `http-response
FORWARDED (HTTP/1.1 403 0ms)`. In the JSON form, **`traffic_direction: EGRESS`**. Envoy wrote
the 403 itself in 0 ms.
- **Gitea's ingress does not need the source namespace.** The act-runner pod's namespace
(`gitea-actions`) is **not** in Gitea's ingress allows. Its NetworkPolicy
`default-deny-ingress-allow-egress` has an `{}` egress rule, so its egress is unrestricted. Its
requests through the Gateway still reach Gitea. `cilium-dbg bpf policy get` on Gitea's endpoint
(1876 on `cilium-9rk5x`) shows all Gateway traffic admitted as `reserved:ingress 3000/TCP`
(391 MB, 1.19 M packets) and no row for `gitea-actions`. The source-identity check therefore
happens on the source's egress, not on the destination's ingress.
- **The policy gap is visible in the BPF map.** `bpf policy get 83` (buildkitd) has these egress
rows and nothing else: `reserved:world` 80/443, `reserved:ingress ANY`, and kube-dns 53. There
is no Gitea identity.
- The edited YAML parses (three documents, names unchanged).
## What I did NOT verify
- **That this rule turns the 403 into a 401/200. UNVERIFIED.** That needs the policy applied, which
is a Flux/operator action. My confidence is high but not proven, and it rests on two things. The
Hubble verdict is EGRESS. Commit `68690e5` made the identical change for Keycloak and recorded
"Verified live: headlamp-ns -> keycloak discovery 403 -> 200".
- **The port restriction.** The rule names `3000`, Gitea's container port, which is the upstream
port Envoy connects to. The Keycloak precedent has no port. If the 403 survives, the first thing
to test is whether L7LB enforcement uses the frontend :443 instead. I expect it does not.
- **Run 337's own flows.** The Hubble ring buffer had already rotated past them, and buildkitd had
been restarted since (the pod was 18 minutes old). The reproduction above uses the same path, the
same pod spec and the same policy, but it is not the original request.
- Envoy's access logs in the `cilium-envoy` pods. Hubble's L7 flow already shows the drop, so I
did not read them.
- The push itself, the Renovate digest flow, and whether the bot team's `Packages=Write` is enough
for a first push that creates a new package. These only become testable after this lands.
+108
View File
@@ -0,0 +1,108 @@
---
date: 2026-10-30
branch: fix/renovate-auth
title: Renovate's token needs five scopes, not three; the workflow documented the wrong list, and now probes it
---
## What changed
**The fix is operator-side.** `RENOVATE_TOKEN` lacks `read:organization`, and it will next lack
`write:issue`. The operator must re-mint it with five scopes:
- `read:user`
- `read:organization`
- `write:repository`
- `write:issue`
- `read:package`
The git half fixes the two things that sent the operator round this loop twice.
**`.gitea/workflows/renovate.yaml`**:
- **The header's scope list is corrected**, with the call each scope serves. The old list named
only `write:repository`, `read:user` and `read:package`, and the operator minted exactly that.
The preflight's "TO FIX" text is corrected the same way.
- **A new step, "Preflight - the token has every scope Renovate needs".**
- It probes read-only GETs: `/user` for read:user, `/orgs/Home-Network` for read:organization,
`/repos/{repo}` for read:repository, `/repos/{repo}/labels` for read:issue, and registry
`/v2/token` for read:package.
- It reads the token's own scope list out of Gitea's 403 from `/admin/orgs` (names only), and
requires `write:repository` and `write:issue` in it.
- It says what a failure means: 401 is the value; a Gitea 403 is a scope, and the probe prints
Gitea's message; an "Access denied" 403 is a Cilium/Envoy denial.
- The token is fed to curl on stdin (`--config -`), never in argv.
**`ci/lint.sh`**: a new guard rejects any non-comment `uses:` in `.gitea/workflows` that needs a
Docker daemon: `docker://`, `renovatebot/github-action` and Docker's daemon-bound
`docker/*-action`s. That is the class of bug fixed in #84.
### Root cause, with evidence
Run 350 (task 567, 06:49 UTC) **got further than 348 and 349.** Gitea's router log, filtered to
the act-runner pod `10.0.0.38`:
06:49:31 GET /api/v1/user 200 OK
06:49:31 GET /api/v1/version 200 OK
06:49:31 GET /api/v1/repos/Home-Network/home-network-k3s-impl 200 OK
06:49:31 GET /api/v1/orgs/Home-Network 403 Forbidden
There were no `/v2/` requests: Renovate stopped before extraction. From the job log, verbatim:
ERROR: Repository has unknown error (repository=Home-Network/home-network-k3s-impl)
"url": "https://gitea.home.keithjackson.dev/api/v1/orgs/Home-Network",
"message": "token does not have at least one of required scope(s), required=[read:organization], token scope=read:package,write:repository,read:user"
"x-envoy-upstream-service-time": "27", "server": "envoy"
INFO: Repository finished ... "result": "unknown-error", "exitCode": 8
**This is not Cilium.** The body is Gitea's JSON scope message, Gitea logged the request, and
`x-envoy-upstream-service-time` shows that Envoy forwarded it upstream.
**The secret WAS updated.** Runs 348 and 349 got `GET /user 403` from `tokenRequiresScopes`. Run 350
got `/user 200`, and its token scope list now includes `read:user`.
**Why `read:organization` is needed.** Renovate 44.167.0, `lib/modules/platform/gitea/index.ts`
`initRepo`, calls `helper.isOrg` → `GET /orgs/{owner}` and rethrows on error. Gitea 1.24.3,
`routers/api/v1/api.go`, puts `/orgs/{org}` behind
`tokenRequiresScopes(AccessTokenScopeCategoryOrganization)`.
**Why `write:issue` is needed next.** In the same `api.go`, `/repos/{o}/{r}/issues…`, `/labels`
and `/milestones` sit in a group with `tokenRequiresScopes(AccessTokenScopeCategoryIssue)`
(lines 1466–1582). Renovate's gitea-helper uses `/issues`, `/issues/{n}/labels`, `/labels` and
`/issues/{n}/comments`, for the dependency dashboard (`:dependencyDashboard` is extended) and for
PR labels. A dry run only reads them; a scheduled run writes them.
The other causes the coordinator listed are ruled out:
- **Schedule window and dry-run.** The run never reached branch processing.
- **Repository config.** The run never reached the clone (`"cloned": undefined`).
- **The Summary step.** It is not involved; the failure is Renovate's own exit 8.
## What I verified
- I read the whole run 350 job log (204 lines), and the router lines above. Runs 348 and 349 are
as recorded in `decisions/2026-10-30-renovate-first-run.md`'s follow-up: `/user` got 403 from
`tokenRequiresScopes`.
- **The probe step, in the pinned image.** I extracted the step body from the YAML and ran it in
`ghcr.io/renovatebot/renovate:44.167.0` (`curl` 8.5.0 and `jq` are present) with a deliberately
**bogus** token. Every probe reported 401 with the "token VALUE" diagnosis, the scope-list read
fell back to a warning, and the step exited 1.
- **The scope-list parser.** Against run 350's exact message format it reports `write:issue`
MISSING. Against a five-scope list it reports both OK. Against `all` it reports both OK.
- **The lint guard.** It flags `renovatebot/github-action` in the pre-#84 `renovate.yaml`
(dfc432c, line 152). It reports nothing on the current workflows, and it ignores commented
lines. `bash -n ci/lint.sh` passes.
- **Account facts, read-only.** `renovate-bot` is a collaborator with `write`, and the org is
public.
## What I did NOT verify
- **The probe with a real token. UNVERIFIED.** I never had the token. In particular, I have not
observed that `GET /api/v1/admin/orgs` gives renovate-bot's token a 403 with a `token scope=`
list. The source says the scope check runs before `reqSiteAdmin`. If the message format
differs, the step warns and does not fail.
- **That five scopes are enough for a full non-dry run.** They cover every helper endpoint Renovate
44.167.0 calls, mapped to Gitea 1.24.3's route categories by reading source. Branch pushes go
over git-over-HTTPS with the token; I believe that needs write:repository, but I have not
observed it.
- **Registry reads as renovate-bot.** `read:package` on a public org is expected to work. The new
probe is the first real test of it.
- **Sensitive paths.** `.gitea/**` and `ci/**` are both sensitive. CI's `approval-policy` will block
until the operator posts `/approve-override` as a PR review.
+113
View File
@@ -0,0 +1,113 @@
---
date: 2026-10-30
branch: fix/renovate-first-run
title: Renovate never started because its action needs a Docker daemon jobs here do not get; it now runs as the job container
---
## What changed
**`.gitea/workflows/renovate.yaml`**: the job no longer uses `renovatebot/github-action`. The
pinned Renovate image (`ghcr.io/renovatebot/renovate:44.167.0`, the same version the action was
pinned to) is now the job's `container:`. A `run:` step calls `renovate-entrypoint.sh renovate`
inside it. `actions/checkout` is gone, because Renovate clones the repo itself.
`GITHUB_COM_TOKEN` is wired from a new **optional** secret, `RENOVATE_GITHUB_COM_TOKEN`. The step
unsets it, and `RENOVATE_DRY_RUN`, when they evaluate to the empty string. The preflight also
prints `renovate --version`, which proves the step ran inside the pinned image.
**`renovate.json5`**: one comment now names `container.image` as the version pin, in place of
`renovate-version`. The config itself is unchanged.
### Why: the root cause
Every run of the action died before Renovate started. That includes the operator's dispatch,
run 343 (task 555, 2026-10-30 06:14 UTC), and the 11 scheduled runs before it (312–322). From
run 343's log:
RENOVATE_TOKEN is present (value not printed).
[command]/usr/bin/docker run --env LOG_LEVEL ... --env RENOVATE_TOKEN=*** --env RENOVATE_CONFIG_FILE=/github-action/renovate.json5 ...
failed to connect to the docker API at unix:///var/run/docker.sock; check if the path is correct and if the daemon is running: dial unix /var/run/docker.sock: connect: no such file or directory
Error: The process '/usr/bin/docker' failed with exit code 1
The action is only a wrapper around `docker run`. On this runner a job deliberately gets no
Docker daemon (`decisions/2026-10-29-image-pipeline.md`): act_runner maps `docker_host` to `-` when
`DOCKER_HOST` is `tcp://`, and the DinD daemon is privileged. So this was a
platform-compatibility failure. The run never reached authentication, token scope, config or the
network. **The re-minted token is not implicated, and nothing in this log can say whether its
scopes are right.**
A job `container:` needs no daemon. act creates it through the runner's DinD the same way it
creates the default `ubuntu-latest` image, then `exec`s each step inside it. I checked this
against gitea/act v0.261.2:
- The job container's entrypoint is `/bin/sleep` (`run_context.go`). The Renovate image has
`/usr/bin/sleep`.
- The step command files are written `0o666` (`step.go`), so the image's uid 12021 can write
`$GITHUB_STEP_SUMMARY`.
- `HOME` comes from the image env.
## What I verified
- **The root cause.** I read all of run 343's job log (162 lines, job 0; job 1 is 404). It has one
error, quoted above, and no Renovate output at all. Runs 312–322 are all `failure` from the
`schedule` event.
- **The config validates against the pinned tag.** `podman run ... ghcr.io/renovatebot/renovate:44.167.0
renovate-config-validator /tmp/renovate.json5` printed `Config validated successfully`. It also
printed the known migration diff for the top-level `docker` block.
- **A dry run with the fix extracts and looks up dependencies.** This was a local reproduction. I
made a clean snapshot of this branch's tracked files. I ran it in a podman container of the pinned
image with entrypoint `/bin/sleep`, as the image's own uid 12021 with `HOME=/home/ubuntu`.
Inside it I `exec`ed the workflow step's `run:` body, extracted verbatim from the YAML, with:
- `RENOVATE_PLATFORM=local`, `RENOVATE_DRY_RUN=full` and `GITHUB_COM_TOKEN=` (empty, to
exercise the unset);
- a docker host rule for `gitea.home.keithjackson.dev` using the `gitea-admin` credential from
`~/.git-credentials`, passed through env and never printed.
The run exited 0 and logged `Dependency extraction complete`:
- flux 15 deps / 13 files
- github-actions 24 / 4
- kubernetes 131 / 67
- kustomize 2 / 2
- regex (both custom managers) 7 / 6
- **total 179 deps / 92 files**
It found **42 updates**: 29 kubernetes, 6 flux, 2 kustomize, 1 github-actions and 4 regex
`digest`. 7 of them are `major`.
- **The `# renovate-digest:` lookups against the Gitea registry succeed.** All three
`/v2/token?scope=repository:home-network/<img>:pull` requests succeeded, and so did
`HEAD /v2/home-network/<img>/manifests/main`. They proposed `digest` updates for
`conformance/versions.env`, `infra/agent/versions.env`, `platform/pm-agent/configmap.yaml` and
`platform/pm-agent/deployment.yaml`, all of which carry the zero placeholder digest today. The
`kubernetes` manager's copies of those images report `disabled`, which is the intended effect of
the packageRule.
- **The new pin is still maintained by Renovate.** The github-actions manager extracts
`ghcr.io/renovatebot/renovate:44.167.0` from `renovate.yaml` as depType `container`, and proposes
44.198.0.
- **The `regex` vs `custom.regex` question from #73/#77 is closed.** Both custom managers produced
dependencies.
- **Missing GitHub token.** The local run logged `GitHub token is required for some dependencies`.
That accounts for 17 deps skipped with `github-token-required`, which is why
`RENOVATE_GITHUB_COM_TOKEN` is offered as an optional secret.
## What I did NOT verify
- **The workflow on the real runner. UNVERIFIED.** My reproduction ran with podman, not
act_runner + DinD, and used `--platform=local`. The act behaviour (sleep entrypoint, `exec` as the
image user, 0666 command files) comes from reading the gitea/act v0.261.2 source, not from a run.
The first real evidence will be a `workflow_dispatch` after merge.
- **The Gitea platform path and `RENOVATE_TOKEN`'s scopes. UNVERIFIED.** I have no access token, only
the `gitea-admin` password, which Renovate's Gitea platform cannot use. So these were not
exercised: platform init (`GET /user`, which needs `read:user`), repo clone, and the registry
lookups using the **token** with `read:package` rather than a password.
- **Behaviour inside the schedule window.** A dispatch outside 03:00–06:00 America/New_York
(07:00–10:00 UTC during EDT) will extract and look up, but will log "not within schedule"
instead of `DRY-RUN: Would create ...` lines. I did not change this.
- **Renovate as uid 12021 doing artifact updates** (for example helm lockfiles) on a real,
non-dry run. `--dry-run=full` does not run the tools.
- **The self-bump.** Renovate will propose `ghcr.io/renovatebot/renovate` 44.167.0 → 44.198.0 as a
minor, which the default rule auto-merges. `.gitea/**` is a sensitive path, so it will in fact
wait for an administrator override. Nothing re-runs `renovate-config-validator` on that bump,
and no gate check runs the validator.
- **No alert for "Renovate red every day".** That gap is still open. This failure ran for 12 days
without anyone noticing.
- **Sensitive path.** `.gitea/**` is in `infra/gitea/approval-policy.json`, so CI's
`approval-policy` stage will block until the operator posts `/approve-override` as a PR review.
@@ -0,0 +1,52 @@
---
date: 2026-10-30
branch: fix/renovate-hold-auth-and-self
title: Operator decision — the oauth2-proxy sidecar and Renovate's own version never auto-merge
---
## What changed
`renovate.json5` gains two packageRules. Both set `automerge: false` and `addLabels: ["needs-review"]`.
They sit after the auto-merge rule, so they override it at every update type. Each rule's
`description` carries the operator's reason.
1. **`quay.io/oauth2-proxy/oauth2-proxy`.** This is the single depName emitted for:
- `catalog/render.py` `OAUTH2_PROXY_IMAGE` (custom regex manager);
- the five rendered `apps/wave-*` sidecars;
- `platform/crafty/deployment.yaml` (kubernetes manager).
**Reason:** since v7.13.0, oauth2-proxy validates refreshed OIDC sessions against the access
token, not the id_token. Every sidecar sets `--cookie-refresh`, and that path is untested against
this Keycloak: the platform oauth2-proxy runs v7.15.5 without `--cookie-refresh`.
`render-golden` and `--selftest` pass on the bump, but both check rendered text, not the binary.
2. **`ghcr.io/renovatebot/renovate`.** This is the job `container.image` in
`.gitea/workflows/renovate.yaml`. The rule also lists `renovatebot/github-action`, so a
reintroduction would be held too. It is not used anywhere today; `ci/lint.sh` rejects it.
**Reason:** nothing re-validates `renovate.json5` against the new version before it takes
effect. The 39.x `CONFIG_VALIDATION` failure is the precedent.
## What I verified
- **The config validates on the pinned 44.167.0.** `renovate-config-validator` printed
`Config validated successfully`. The migration diff is only the pre-existing top-level `docker`
block.
- **The flag change is exactly the intended one.** I used Renovate 44.167.0's own
`resolveConfigPresets` and `applyPackageRules` to evaluate b4ff287's config and this branch's
config against the same updates (the local dry run of the #86 config). Exactly **2 of 29**
branch rows differ:
- `ghcr.io-renovatebot-renovate-44.x`: `automerge=true labels=` → `automerge=false labels=needs-review`;
- `quay.io-oauth2-proxy-oauth2-proxy-7.x`: the same change, and every upgrade in the branch
resolves `false`, the `catalog/render.py` one included.
The other 27 rows are byte-identical.
## What I did NOT verify
- **The three GitHub-hosted branches** in run 357 (`pin-dependencies`, `kubevirt-cdi`,
`intel-intel-device-plugins-for-kubernetes-0.x`) are not in the harness table: my local run has
no github.com token. Neither rule's package names can match them.
- **A live Renovate run with this config.** The next debug-level dispatch should show
`needs-review` on both branches.
- **The oauth2-proxy refresh path itself** remains untested. Exercising it on one app is the
review the label asks for.
@@ -0,0 +1,128 @@
---
date: 2026-10-30
branch: fix/renovate-run-hygiene
title: Renovate's first completed dry runs, audited; it would have opened red PRs for every app and missed the databases. Config fixed
---
## What changed
**`renovate.json5` only.** Each change carries a comment quoting the measurement behind it.
1. **Three new regex managers, so updates edit the SOURCE.**
- `catalog/*.yaml` `source.image`.
- `catalog/render.py` `OAUTH2_PROXY_IMAGE`.
- CNPG `imageName:` under `platform/` and `apps/`.
The first two match the rendered line's depName, so the catalog edit and the generated
`apps/wave-*` edit land in **one** branch.
2. **`allowedVersions` mirroring each ServiceDefinition's `update.range`:**
- gitea/gitea `<2.0.0`
- nextcloud `<32.0.0`
- qbittorrent `<6.0.0`
- radarr `<6.0.0`
- sonarr `<5.0.0`
- semaphore `<3.0.0`
Plus CNPG postgresql `<17.0.0`, because a Postgres major is a migration, not a bump.
3. **The blast-radius never-automerge list gains** `gitea/gitea`, `gitea/act_runner` and
`ghcr.io/cloudnative-pg/postgresql`.
4. **HelmRelease range widening** (`matchManagers: flux`, ranged currentValue) is never
auto-merged and is labelled `range-widen`.
5. **Majors need a Dependency Dashboard tick** (`dependencyDashboardApproval`) before a PR opens.
6. **`flux` manager docker deps are disabled.** These are duplicates of `kubernetes` manager
extractions.
7. **`velero/velero-plugin-for-aws` uses `semver` versioning**, so its RC pin can reach a stable
release.
8. **`python` minors are not auto-merged.**
9. **Two false comments are corrected.**
- Ranges "cap Renovate at patches": false, Renovate widens them.
- `nextcloud:31-apache` and `docker:27-dind` "cannot be maintained": false, they were.
PR #85 is merged, so nothing here overlaps an open branch.
### The audit: dispatch runs 353 (07:12 UTC) and 354 (07:23 UTC), on 593d0fa
Both runs ended `success`, Renovate `"result": "done"`, `exitCode 0`. Both were at `info` level, so
I reproduced the per-dependency detail with a debug-level local dry run of the same commit (see
below).
The runs were inside the 03:00–06:00 America/New_York window, so branches were processed:
- run 353: 37 "Would commit" branches;
- run 354: 40, because the operator had added `RENOVATE_GITHUB_COM_TOKEN` 9 s before it.
Gitea's router log for the act-runner pod during both runs shows 317 2xx responses. The only
non-2xx were all expected:
- `/v2/` 401: the registry challenge;
- `/admin/orgs` 403: the #85 scope probe, by design;
- 6× `HEAD …/manifests/sha256:000…` 404: the placeholder digests.
| # | Class | Count | Cause | Severity | Fix |
|---|---|---|---|---|---|
| 1 | App-image PRs edit only generated `apps/wave-*` | 9 branches | A ServiceDefinition has no `apiVersion`, so the `kubernetes` manager extracts 0 deps from `catalog/`. The oauth2-proxy sidecar is a Python constant (`catalog/render.py:132`) | **Real problem.** Every such PR fails `ci/render-golden.sh` (reproduced: rc 1 "GOLDEN DRIFT"), stays red and holds a PR slot | git (1) |
| 2 | CNPG `imageName` untracked | 3 Clusters | The `kubernetes` manager reads `image:` only. The one tracked postgres line was the grant Job (`nextcloud-db.yaml:303`, "keep in step with spec.imageName") | **Real problem.** A real run bumps the Job and leaves the databases behind | git (1) |
| 3 | Bogus or out-of-policy majors | 5 branches: gitea 28.2.0, qbittorrent 20.04.1, radarr 6.x, nextcloud 35, postgresql 18.6 | Odd legacy tags, plus `update.range` in `catalog/` not being read by Renovate | Risk if run for real: unmergeable PRs, which hold slots | git (2) |
| 4 | Auto-merge of the merge gate's own host | gitea 1.24.3→1.27.3, act_runner 0.2.11→0.6.1, both "minor" | Not in the blast-radius list. A 0.x minor is breaking by semver | **Risk if run for real** | git (3) |
| 5 | Range widening auto-merged | external-dns, headlamp (minor); longhorn (blast radius); kube-prometheus-stack (major) | Renovate widens `>=a <b`. The config's comment claimed the opposite | **Risk if run for real:** silently lets Flux adopt new minors | git (4, 9) |
| 6 | Majors hold PR slots | 10 majors in run 354 against `prConcurrentLimit: 5` | Majors never auto-merge, so they stay open | Risk: starves auto-mergeable PRs | git (5) |
| 7 | `Cannot find replaceString … alpine:3.22` (INFO) | 1 per run | The same HelmRelease `image:` line is extracted by `flux` and `kubernetes` (also the velero plugin) | Cosmetic. The branch is still right | git (6) |
| 8 | velero-plugin-for-aws stuck on `v1.14.3-rc.2` | 1 dep, 0 updates | Docker versioning treats `-rc.2` as a suffix, so stable `v1.14.3` and `v1.14.4` never qualify | **Real problem** (silent) | git (7) |
| 9 | python 3.13→3.15 as auto-mergeable "minor" | 2 files | Language runtime minor | Risk if run for real | git (8) |
| 10 | `WARN GitHub token is required` | run 353: 5 packages, 17 deps | No github.com token | Real (they were unmaintained) | **operator, done:** run 354 has no WARN and adds `pin-dependencies`, `kubevirt-cdi`, `actions-checkout-7.x`, `major-github-artifact-actions` |
| 11 | Floating or untagged images (`invalid-value`) | 4: jackett, plex `latest`; pihole, keycloak no tag | No version to compare | Real, already documented | none in git. Pinning by hand is an operator/PM decision |
| 12 | `runs-on: ubuntu-latest` `invalid-version` | 11 | The github-actions manager extracts runner labels | Cosmetic, debug-only | none |
| 13 | `Docker Manifest is unknown` (debug) | 3 | HEAD on the all-zero placeholder digests | Expected. Clears when the own-image digest PRs merge | none |
| 14 | `Docker: error fetching data from DockerHub` 403 (debug) | 5 | Hub refuses `page=11`. Renovate already has the tags | Cosmetic. Updates were still found | none |
| 15 | `Config migration necessary` | 1 | Top-level `"docker": {"enabled": true}` | Cosmetic, pre-existing | none (as before) |
**Checked and fine:**
- The 3 own-image digest updates were found in both runs. All 4 `# renovate-digest:` markers parse.
- No marker failed to parse.
- No package file is under `conformance/runs` or `tests`, nor is `DECISIONS-LOG.md`, `decisions/` or
any `*.sops.yaml` (77 package files checked).
- `kubevirt-cdi` grouping works.
- The schedule window behaved as designed.
- Vulnerability alerts: none in either log.
## What I verified
- I read both job logs in full (132 and 129 lines), and the router log for 10.0.0.38 during both
runs.
- **`renovate-config-validator` on 44.167.0**: `Config validated successfully`. The migration diff
is only the pre-existing `docker` block.
- **Local dry runs.** Pinned image, `--platform=local`, debug, origin/main snapshot. "Before" is
593d0fa and "after" is this config; the GitHub-hosted deps were skipped in both.
- **Extraction:** `regex` went from 6 files / 7 deps to **19 files / 20 deps**. Every other
manager is unchanged: flux 15, github-actions 24, kubernetes 131, kustomize 2.
- **Branches:** 33 before, **29 after**. Removed: gitea-28.x, qbittorrent-20.x, radarr-6.x,
nextcloud-35.x and postgresql-18.x. Added: velero `v1.14.3-rc.2 → v1.14.4` (patch).
- **Gitea, qbittorrent, radarr, sonarr, semaphore and seerr branches** now each carry
`catalog/<svc>.yaml` + `apps/wave-*/<svc>.yaml`.
- **The oauth2-proxy branch** carries `catalog/render.py` plus the 5 rendered apps and
`platform/crafty`.
- **The postgresql-16.x branch** carries all three Clusters.
- The 2 flux docker duplicates report `disabled`.
- **render==golden holds for the new branch shape.**
- In the toolbox, I applied the gitea branch edit (catalog + apps) and the oauth2-proxy branch
edit (`render.py` constant + 5 apps) textually. `./ci/render-golden.sh` reported
`all 10 onboarded service golden(s) reproduce`, rc 0.
- Negative control, the old shape with only `apps/wave-0/gitea.yaml` bumped: rc 1,
`GOLDEN DRIFT`.
- Both edits were reverted afterwards.
## What I did NOT verify
- **The effect of the automerge, label and dashboard-approval rules (3, 4, 5, 8). UNVERIFIED in a
run.** `--platform=local` stops before branch processing, so I saw only which updates exist, not
their merged flags. The matchers are standard (`matchPackageNames`, `matchManagers`,
`matchCurrentValue`, `matchUpdateTypes`). The next dispatch's debug log shows them per branch.
- **`allowedVersions` under docker versioning** used Renovate's "Falling back to npm semver
syntax" path (16 debug lines). That it filters is shown by the five removed branches.
- **render==golden for the other app branches** (qbittorrent, radarr, sonarr, semaphore, seerr). I
proved it only for gitea and oauth2-proxy. They have the same shape: one `source.image` line ↔
one rendered `image:` line.
- **The `allowedVersions` rules duplicate `catalog/*/update.range`** and can drift from it. Nothing
checks them against each other. A lint comparing the two is the obvious follow-up.
- **Any real (non-dry) run.** None has happened. PR creation, labels, auto-merge through Gitea's
API and the Dependency Dashboard issue are all unobserved.
- **Sensitive path?** `renovate.json5` is not in `infra/gitea/approval-policy.json`, as far as I
know. CI's `approval-policy` result on the PR is the measurement.
@@ -0,0 +1,113 @@
---
date: 2026-10-30
branch: feat/renovate-track-keycloak-operator
title: Renovate tracks the Keycloak operator (the real Keycloak version), never auto-merged; the catalog's phantom image is pinned as documentation and disabled
---
## What changed
- **`platform/keycloak/kustomization.yaml`**: each of the three `keycloak-k8s-resources/26.0.6`
release URLs (two CRDs and `kubernetes.yml`) now has a
`# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources` marker on the line
above it. That is the format the existing `custom.regex` marker manager parses (the kubevirt
pattern). A comment explains that these URLs are the Keycloak version. The `Keycloak` CR has no
`spec.image`, so the operator's `RELATED_IMAGE_KEYCLOAK` picks the server image.
- **`renovate.json5`** has two new packageRules:
- **`keycloak/keycloak-k8s-resources`**: `groupName: keycloak-operator`, `automerge: false`,
`addLabels: [needs-review, blast-radius]`. A comment explains why. The operator, the CRDs and
the server are released in lockstep. The IdP gates SSO for every app and for kubectl. And
`passkey-reconciler.yaml` encodes Keycloak 26.0.6 Admin API shapes that the static gate never
exercises. Majors still go through the existing dashboard-approval rule.
- **`quay.io/keycloak/keycloak`**: `enabled: false`. Its only occurrences are in files that
deploy nothing.
The "STILL CANNOT FIX" comment also gains a one-line correction for keycloak.
- **`catalog/keycloak.yaml`**: `source.image` changed from `quay.io/keycloak/keycloak` (no tag) to
`quay.io/keycloak/keycloak:26.0.6`. A comment says the image is DOCUMENTATION-ONLY and that the
operator chooses the real one. `apps/wave-1/keycloak.yaml` was re-rendered, which changes one
line.
- **`platform/keycloak/README.md`**:
- Corrects the false claim "Renovate maintains the version". Until now Renovate could not see
the version.
- Adds a new section, **"Upgrading Keycloak: the operator IS the version"**. It lists the 26.0.6
assumptions in the passkey reconciler to diff against a new tag, and the post-merge checks.
**The phantom: pin and disable, not delete.** `apps/wave-1/keycloak.yaml` is not listed in
`apps/wave-1/kustomization.yaml`, and render-golden excludes it (`RENDER_GOLDEN_NOT_ONBOARDED`).
Deleting it would still not be a no-op:
- It is the fixture for the keycloak cases in `apps/wave-1/tests/app-slis.test.yaml` (the
StatefulSet-based `keycloak-down` rule). Those cases are discovered by the
every-PrometheusRule-in-apps/wave-1 convention.
- Removing it means moving that rule and its tests to `platform/keycloak`. That is a separate
observability change, not a Renovate one.
Pinning plus `enabled: false` changes one line in a file nothing applies. It cannot affect the
cluster, and render-golden stays green.
## What I verified
- **Live state, prod, read-only kubectl, 2026-10-30.**
- Keycloak CR `keycloak` has an empty `spec.image`.
- The operator Deployment runs `quay.io/keycloak/keycloak-operator:26.0.6`, imageID
`sha256:8e3a1a56346f…`, with `RELATED_IMAGE_KEYCLOAK=quay.io/keycloak/keycloak:26.0.6`.
- `keycloak-0` and `keycloak-1` both run `quay.io/keycloak/keycloak:26.0.6`, imageID
`sha256:a93d22e13b86…`.
- There is no PrometheusRule containing "keycloak" in any namespace. This confirms the phantom
deploys nothing, including its `keycloak-down` rule (see the next section).
- **`renovate-config-validator`** on the pinned `ghcr.io/renovatebot/renovate:44.167.0`:
`Config validated successfully`. The only migration note is the pre-existing `docker` block.
- **Local dry run** (`RENOVATE_PLATFORM=local`, `RENOVATE_DRY_RUN=full`, debug, the same pinned
image, this worktree, with a GitHub token passed by environment variable; the token string
occurs 0 times in the log):
- The regex manager went to 21 files / 24 deps.
- `keycloak/keycloak-k8s-resources` was extracted three times from
`platform/keycloak/kustomization.yaml`, each at `currentValue 26.0.6`, datasource
`github-tags`, versioning `semver`.
- Each extraction has one update, `26.8.1`, updateType `minor`, and all three share **one**
`branchName: renovate/keycloak-operator`.
- `quay.io/keycloak/keycloak` was extracted from `catalog/keycloak.yaml` (regex) and from
`apps/wave-1/keycloak.yaml` (kubernetes) at 26.0.6, with `skipReason: disabled` and no
updates.
- **Rule effects.** `--platform=local` does not print each branch's merged flags. So I ran
Renovate's own `applyPackageRules` (from `dist/util/package-rules` in the same 44.167.0 image)
over this `renovate.json5` for each case:
| Update | Result |
|---|---|
| operator minor 26.0.6 → 26.8.1 | `automerge:false`, `groupName:keycloak-operator`, labels `dependencies, automated-update, needs-review, blast-radius` |
| operator patch | same as the minor |
| operator major | `automerge:false`, `dependencyDashboardApproval:true`, labels add `breaking` |
| `quay.io/keycloak/keycloak`, catalog and rendered | `enabled:false` |
| control: sonarr patch | `automerge:true`, so the rule did not disable auto-merge globally |
- **render-golden, host:** `render.py --selftest` passed, and `all 10 onboarded service golden(s)
reproduce`. Before the pin, the committed phantom was byte-identical to the canonical render. I
re-rendered it after the pin.
- **Toolbox gate** (commit 24db269, worktree with the ENGINE shim mounting `.git` and the spec
repo, `DEV_KUBECONFIG=/dev/null`): **rc 3**.
- The only stage that did not run is **`gitea-settings`**.
- spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all passed.
- L2/L3 `conformance` scaffold-skipped, as expected.
- **Not green.**
- **Sensitive paths:** none touched (checked against `infra/gitea/approval-policy.json`), so no
`/approve-override` is needed.
## What I did NOT verify
- **A real Renovate run.** No PR was opened by Renovate. The labels and `automerge:false` are
proven by Renovate's own matcher, not by an opened PR.
- **That 26.8.1 is safe to adopt.** I did not diff 26.0.6 against 26.8.1. That includes the
reconciler's assumptions: `recurseExecutions` displayName, `updateExecutions`, the
`RpEntityName` guard, `requiredActions` REPLACE, and config aliases. The README section lists
them for whoever reviews the first bump PR. It also says to take or confirm a fresh CNPG backup
first, because a server minor may migrate the schema.
- **FINDING, not fixed: Keycloak has no deployed "IdP down" alert.** The `keycloak-down` SLI rule
exists only inside the undeployed `apps/wave-1/keycloak.yaml`. promtool tests it, but it is not
on the cluster: there is no keycloak PrometheusRule live. Separately, because that file exists,
`render.py --emit-waves` would treat keycloak as onboarded. It would then add a Deployment named
`keycloak` beside the operator's StatefulSet (the known `--emit-waves` hazard). The proper fix is
a separate PR: move the rule and its tests into `platform/keycloak`, then delete the phantom.
- **The catalog tag can drift.** Nothing ties `catalog/keycloak.yaml`'s documentation tag to the
operator URLs. It must be updated by hand after an operator bump, and the README says so.
- **Concurrent branches** `fix/renovate-hold-auth-and-self` and `feat/pin-pihole-jackett-plex`
also edit `renovate.json5`. Neither had merged when I checked. A rebase may be needed.
+6 -4
View File
@@ -244,10 +244,12 @@ about runtime behaviour in this document is a design claim, not a measurement.
(ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with
the credential decision now being local Gitea accounts with PATs, that means a
`dockerconfigjson` for an account whose token carries `read:package`. Only then does the
digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both
placeholder digests. Expect the build itself to need iteration too: the nodejs module
stream and the npm install have never run against this base. The same two blockers sit in
front of `conformance-runner`, which shares the registry path and has never been pulled.
digest matter. **Superseded 2026-10-29:** the build-and-push is no longer an operator step.
`.gitea/workflows/images.yaml` builds and verifies the image on rootless BuildKit
(`platform/buildkit/`), publishes it on a push to `main`, and Renovate opens the one
PR that replaces both placeholder digests (`decisions/2026-10-29-image-pipeline.md`). The
same two blockers sit in front of `conformance-runner`, which shares the registry path and
has never been pulled.
2. **Create the credentials** per `pm-agent-credentials.sops.yaml.example`, and **verify the
bot's permission is `write`, not `admin`** — that measurement is the control that keeps
the agent out of its own approval gate.
+81 -1
View File
@@ -50,7 +50,10 @@
# default is deliberately EMPTY so a build that forgets it fails loudly instead of
# resolving some other image named `agent-base`.
ARG AGENT_BASE_IMAGE=
FROM ${AGENT_BASE_IMAGE}
# Stage `agent-runner` is the image that ships; stage `verify` (end of file) is a TEST stage built
# FROM it that only runs assertions and is never pushed. build-image.sh and the CI pipeline
# build `--target verify` to prove the image and `--target agent-runner` for the image itself.
FROM ${AGENT_BASE_IMAGE} AS agent-runner
# Re-declared AFTER the FROM: an ARG from before the first FROM is not in scope in the
# build stage, so the provenance layer below would otherwise record an empty base.
@@ -208,3 +211,80 @@ WORKDIR /workspace
# paths above; an ENTRYPOINT here would either be ignored or prepend something the policy
# cannot see. `bash` is the interactive fallback for a human debugging the image locally.
CMD ["/bin/bash"]
# =============================================================================
# verify — a TEST stage, never shipped. Every assertion infra/agent/build-image.sh used to make
# with `run --user 65532 --read-only --tmpfs /workspace --tmpfs /tmp --network none` now runs
# here, at build time, because CI builds on rootless BuildKit where no container runtime exists.
#
# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE, stated so nobody over-reads a green verify:
# * uid 65532 ............ exact: USER is inherited from the agent-runner stage.
# * --network none ....... exact: RUN --network=none, and the first check proves it.
# * --read-only rootfs ... NOT reproducible: a RUN step always has a writable layer. It is
# APPROXIMATED by permission: as uid 65532, every path the process can
# write is inventoried and must lie under /workspace, /tmp or /var/tmp
# (the pod mounts emptyDirs on the first two; /var/tmp is read-only
# there, and TMPDIR points into /workspace so nothing should use it).
# The pod's readOnlyRootFilesystem is enforced by admission
# (platform/kyverno/admission/pm-agent-job-shape.yaml), not by this.
# * --tmpfs /workspace ... approximated by the image's own /workspace (owned 65532), which is
# what the pod's emptyDir replaces.
# The probe script is COPYed into THIS stage only, so it never ships in the image.
# =============================================================================
FROM agent-runner AS verify
COPY probe-validation.sh /opt/verify/probe-validation.sh
RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
echo "--- no network (the build-time stand-in for run --network none) ---"
if getent hosts registry.npmjs.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then
echo "UNEXPECTED: network reachable in the verify stage"; exit 1
fi
echo " confirmed: no network"
echo "--- id ---"; id
echo "--- HOME ---"; echo "HOME=${HOME} CLAUDE_CONFIG_DIR=${CLAUDE_CONFIG_DIR}"
echo "--- baked versions ---"; cat /etc/agent-runner-versions
echo "--- the gate toolchain the worker needs (CLAUDE.md §3) ---"
for t in git python3 bash yamllint check-jsonschema kyverno promtool chainsaw flux sops age kubectl jq diff; do
if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t";
else printf " MISSING %s\n" "$t"; missing=1; fi
done
[ -z "${missing:-}" ] || { echo "the gate cannot run in this image"; exit 1; }
echo "--- python can import the renderer dependency ---"
python3 -c "import yaml; print('PyYAML', yaml.__version__)"
echo "--- claude ---"; claude --version
echo "--- the two entrypoints refuse arguments ---"
# Each must REFUSE. (The old docker-run version wrote `cmd | grep -q ... && echo`, which under
# `set -e` + pipefail could never fail the script; these are real assertions.)
for e in pm-worker pm-dispatch; do
out="$(/usr/local/bin/$e --anything 2>&1 || true)"
grep -q "takes no arguments" <<<"$out" || { echo "UNEXPECTED: $e accepted an argument: $out"; exit 1; }
echo " $e: rejected"
done
echo "--- the dispatcher selftest (branch vocabulary, result envelopes, Job template, persona) ---"
PM_SELFTEST=1 /usr/local/bin/pm-dispatch
echo "--- the entrypoints are root-owned and NOT writable by the runtime uid ---"
for f in /usr/local/bin/pm-worker /usr/local/bin/pm-dispatch /etc/pm-agent/worker-job-template.yaml; do
stat -c " %n %U:%G %a" "$f"
if [ -w "$f" ]; then echo "UNEXPECTED: uid $(id -u) can write $f"; exit 1; fi
done
echo " confirmed: not writable by uid $(id -u)"
echo "--- the root filesystem is not writable by the runtime uid (expected: a failure) ---"
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi
echo " confirmed: / is not writable"
echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---"
stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \
| grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)"
if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf ' %s\n' ${stray} | head -20; exit 1; fi
echo " confirmed: no writable path outside /workspace, /tmp, /var/tmp"
echo "--- the two writable paths really are writable ---"
touch "${HOME}/.probe" /tmp/.probe || { echo "UNEXPECTED: ${HOME} or /tmp not writable"; exit 1; }
echo " confirmed: ${HOME} and /tmp are writable"
echo "--- uid is non-zero ---"
[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; }
echo " confirmed: uid $(id -u), not root"
echo "--- no Anthropic API key is baked into the image (it would outrank the subscription token) ---"
[ -z "${ANTHROPIC_API_KEY:-}${ANTHROPIC_AUTH_TOKEN:-}" ] || { echo "UNEXPECTED: an Anthropic key is baked in"; exit 1; }
echo " confirmed: neither is set"
echo "--- pm-worker's input validation, with the interpreter and script that run in the cluster ---"
PM_WORKER_PATH=/usr/local/bin/pm-worker /bin/bash /opt/verify/probe-validation.sh
echo "AGENT-RUNNER VERIFIED"
VERIFY
+86 -140
View File
@@ -1,26 +1,36 @@
#!/usr/bin/env bash
###############################################################################
# infra/agent/build-image.sh — build the pinned `agent-runner` image and print its digest.
# infra/agent/build-image.sh — build the pinned `agent-runner` image and prove it works.
#
# ./build-image.sh # build + verify + print the digest and push command
# ./build-image.sh # build the verify target, then the image
# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes)
# ./build-image.sh --no-verify # build only (skip the in-image proof)
# ENGINE=docker ./build-image.sh # build with docker instead of podman
# AGENT_BASE_IMAGE=home-k3s-dev:local ./build-image.sh
# # build on a LOCAL toolbox instead of the pinned one
# # (`infra/devcontainer/dev.sh build` makes that tag)
# IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this)
#
# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is not
# an agent's to hold, and the pushed digest is what a fail-closed admission policy then
# trusts. So this builds locally, proves the toolchain and both entrypoints work INSIDE the
# image as the non-root uid with the read-only root filesystem the policy forces, and prints
# the exact push line for a human.
# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on
# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and
# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest
# in the job summary. Renovate then opens the pull request that moves platform/pm-agent/ onto
# that digest. This script is the LOCAL path (podman by default); its proof — the
# Containerfile's `verify` stage — is the same one CI runs.
#
# VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the four pins this image
# adds (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds
# the platform toolchain the base already carries. Nothing in this file invents a version.
# VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the pins this image adds
# (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds the
# platform toolchain the base already carries. Nothing in this file invents a version.
#
# THE DIGEST CAVEAT — read it before pasting a digest into anything. The digest printed here
# is the LOCAL manifest digest. It is normally identical to the registry digest, but that is
# a property of the push rather than a guarantee, so the push command below uses
# `--digestfile`: the digest the REGISTRY returns is the authoritative one, and that is the
# one that goes into platform/pm-agent/configmap.yaml.
# THE BASE. AGENT_BASE_IMAGE in versions.env is the PUBLISHED toolbox, pinned by digest and
# kept current by Renovate. An AGENT_BASE_IMAGE in the environment overrides it — CI passes
# the toolbox it built in the same run, and a developer can pass `home-k3s-dev:local`.
#
# NO `run` STEP. Verification used to `run` the image as uid 65532 with a read-only rootfs.
# CI has no container runtime (rootless BuildKit only), so the assertions moved into the
# Containerfile's `verify` stage and this script builds that target. `--print-build-args`
# prints the pins as KEY=VALUE for the CI pipeline; AGENT_BASE_IMAGE is the caller's choice.
###############################################################################
set -euo pipefail
@@ -29,9 +39,11 @@ REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
ENGINE="${ENGINE:-podman}"
VERIFY=1
PRINT_ARGS=0
for arg in "$@"; do
case "$arg" in
--no-verify) VERIFY=0 ;;
--print-build-args) PRINT_ARGS=1 ;;
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown argument: $arg" >&2; exit 2 ;;
esac
@@ -41,13 +53,14 @@ log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
info() { printf ' %s\n' "$*"; }
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
# --- pins: sourced, never invented -------------------------------------------
AGENT_ENV="${HERE}/versions.env"
[ -f "${AGENT_ENV}" ] || die "missing ${AGENT_ENV} — the pins live there, not here"
BASE_OVERRIDE="${AGENT_BASE_IMAGE:-}"
TAG_OVERRIDE="${IMAGE_TAG:-}"
# shellcheck disable=SC1090
source "${AGENT_ENV}"
[ -z "${BASE_OVERRIDE}" ] || AGENT_BASE_IMAGE="${BASE_OVERRIDE}"
: "${AGENT_BASE_IMAGE:?AGENT_BASE_IMAGE not set in ${AGENT_ENV}}"
: "${NODEJS_MAJOR:?NODEJS_MAJOR not set in ${AGENT_ENV}}"
: "${CLAUDE_CODE_VERSION:?CLAUDE_CODE_VERSION not set in ${AGENT_ENV}}"
@@ -55,16 +68,39 @@ source "${AGENT_ENV}"
: "${AGENT_MODEL:?AGENT_MODEL not set in ${AGENT_ENV}}"
: "${AGENT_IMAGE_REPO:?AGENT_IMAGE_REPO not set in ${AGENT_ENV}}"
# --- the base must already exist ---------------------------------------------
# Deliberately NOT built on demand. The toolbox build is its own reviewed artefact with its
# own pins, and silently building it here would mean this script could change the toolchain
# a worker gates with as a side effect of rebuilding the agent image.
if ! "${ENGINE}" image exists "${AGENT_BASE_IMAGE}" 2>/dev/null; then
die "base image '${AGENT_BASE_IMAGE}' is not present. Build the pinned toolbox first:
${REPO_ROOT}/infra/devcontainer/dev.sh build"
# THE build args, in ONE place; `--print-build-args` hands them to CI (which builds on rootless
# BuildKit and supplies its own base reference). AGENT_BASE_IMAGE is deliberately NOT in this
# list: the caller decides the base.
BUILD_ARGS=(
"NODEJS_MAJOR=${NODEJS_MAJOR}" "CLAUDE_CODE_VERSION=${CLAUDE_CODE_VERSION}"
"CLAUDE_CODE_MIN_VERSION=${CLAUDE_CODE_MIN_VERSION}" "AGENT_MODEL=${AGENT_MODEL}"
)
if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
# --- the base must exist, locally or in the registry ---------------------------
# Deliberately NOT built on demand. The toolbox is its own reviewed artefact with its own
# pins, and silently building it here would mean this script could change the toolchain a
# worker gates with as a side effect of rebuilding the agent image.
case "${AGENT_BASE_IMAGE}" in
*@sha256:0000000000000000000000000000000000000000000000000000000000000000)
die "AGENT_BASE_IMAGE is still the all-zero placeholder in ${AGENT_ENV}: the toolbox has
not been published yet. Either let CI publish it (.gitea/workflows/images.yaml) and
merge Renovate's digest bump, or build on a local toolbox:
${REPO_ROOT}/infra/devcontainer/dev.sh build
AGENT_BASE_IMAGE=home-k3s-dev:local $0" ;;
esac
if ! "${ENGINE}" image inspect "${AGENT_BASE_IMAGE}" >/dev/null 2>&1; then
info "base ${AGENT_BASE_IMAGE} is not local; pulling it"
"${ENGINE}" pull "${AGENT_BASE_IMAGE}" >/dev/null \
|| die "base image '${AGENT_BASE_IMAGE}' is neither local nor pullable (registry login?).
To build on a local toolbox instead:
${REPO_ROOT}/infra/devcontainer/dev.sh build
AGENT_BASE_IMAGE=home-k3s-dev:local $0"
fi
IMAGE_TAG="claude${CLAUDE_CODE_VERSION}-node${NODEJS_MAJOR}"
IMAGE_TAG="${TAG_OVERRIDE:-claude${CLAUDE_CODE_VERSION}-node${NODEJS_MAJOR}}"
IMAGE_REF="${AGENT_IMAGE_REPO}:${IMAGE_TAG}"
log "Building ${IMAGE_REF}"
@@ -75,126 +111,36 @@ info " node ${NODEJS_MAJOR}"
info " claude ${CLAUDE_CODE_VERSION} (floor ${CLAUDE_CODE_MIN_VERSION})"
info " model ${AGENT_MODEL}"
"${ENGINE}" build \
--file "${HERE}/Containerfile" \
--tag "${IMAGE_REF}" \
--build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}" \
--build-arg "NODEJS_MAJOR=${NODEJS_MAJOR}" \
--build-arg "CLAUDE_CODE_VERSION=${CLAUDE_CODE_VERSION}" \
--build-arg "CLAUDE_CODE_MIN_VERSION=${CLAUDE_CODE_MIN_VERSION}" \
--build-arg "AGENT_MODEL=${AGENT_MODEL}" \
"${HERE}"
# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
LABEL_ARGS=()
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
ARG_FLAGS=(--build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}")
for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
# --- verify: the point of the whole exercise ---------------------------------
# A build that merely succeeded is not evidence this image is usable. Prove the tools
# EXECUTE, as the NON-ROOT uid, with a READ-ONLY root filesystem and only the two volumes
# the admission policy admits — the exact shape a worker Job runs in. Anything that works
# here but not there is a 3am discovery.
# A build that merely succeeded is not evidence this image is usable. The proof is the
# Containerfile's `verify` stage: as the NON-ROOT uid, with no network, the toolchain executes,
# both entrypoints refuse arguments, the dispatcher selftest passes, the entrypoints are not
# writable, nothing outside /workspace and /tmp is writable, and pm-worker's input guards
# (probe-validation.sh) refuse every hostile input. It is a build target rather than a
# `run` so the SAME proof runs here and in CI, where no container runtime exists — see the
# stage's header for what a build step can and cannot reproduce of the pod's shape.
if [ "${VERIFY}" -eq 1 ]; then
log "Verifying inside the image (uid 65532, read-only rootfs, workspace+tmp only)"
"${ENGINE}" run --rm \
--user 65532:65532 \
--read-only \
--tmpfs /workspace:rw,mode=1777 \
--tmpfs /tmp:rw,mode=1777 \
--network none \
--entrypoint "" \
"${IMAGE_REF}" \
/bin/bash -euo pipefail -c '
echo "--- id ---"; id
echo "--- HOME ---"; echo "HOME=${HOME} CLAUDE_CONFIG_DIR=${CLAUDE_CONFIG_DIR}"
echo "--- baked versions ---"; cat /etc/agent-runner-versions
echo "--- the gate toolchain the worker needs (CLAUDE.md §3) ---"
for t in git python3 bash yamllint check-jsonschema kyverno promtool chainsaw flux sops age kubectl jq diff; do
if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t";
else printf " MISSING %s\n" "$t"; missing=1; fi
done
[ -z "${missing:-}" ] || { echo "the gate cannot run in this image"; exit 1; }
echo "--- python can import the renderer dependency ---"
python3 -c "import yaml; print(\"PyYAML\", yaml.__version__)"
echo "--- claude ---"; claude --version
echo "--- the two entrypoints refuse arguments ---"
/usr/local/bin/pm-worker --anything 2>&1 | grep -q "takes no arguments" && echo " pm-worker: rejected"
/usr/local/bin/pm-dispatch --anything 2>&1 | grep -q "takes no arguments" && echo " pm-dispatch: rejected"
echo "--- the dispatcher selftest (branch vocabulary, result envelopes, Job template, persona) ---"
PM_SELFTEST=1 /usr/local/bin/pm-dispatch
echo "--- the entrypoints are root-owned and NOT writable by the runtime uid ---"
for f in /usr/local/bin/pm-worker /usr/local/bin/pm-dispatch /etc/pm-agent/worker-job-template.yaml; do
stat -c " %n %U:%G %a" "$f"
if [ -w "$f" ]; then echo "UNEXPECTED: uid $(id -u) can write $f"; exit 1; fi
done
echo " confirmed: not writable by uid $(id -u)"
echo "--- rootfs is read-only (expected: a failure) ---"
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
echo " confirmed: / is read-only"
echo "--- the two writable paths really are writable ---"
touch "${HOME}/.probe" /tmp/.probe && echo " confirmed: ${HOME} and /tmp are writable"
echo "--- uid is non-zero ---"
[ "$(id -u)" -ne 0 ] && echo " confirmed: uid $(id -u), not root"
echo "--- no Anthropic API key is baked into the image (it would outrank the subscription token) ---"
[ -z "${ANTHROPIC_API_KEY:-}${ANTHROPIC_AUTH_TOKEN:-}" ] && echo " confirmed: neither is set"
'
# The worker's input guards, exercised in the image, with the interpreter and the script
# that will actually run in the cluster. The probe is mounted read-only rather than baked
# in: it is a build-time check, not something a worker Job has any business carrying.
log "Probing pm-worker's input validation inside the image"
"${ENGINE}" run --rm \
--user 65532:65532 \
--read-only \
--tmpfs /workspace:rw,mode=1777 \
--tmpfs /tmp:rw,mode=1777 \
--network none \
--volume "${HERE}/probe-validation.sh:/probe-validation.sh:ro,Z" \
--entrypoint "" \
"${IMAGE_REF}" \
/bin/bash /probe-validation.sh
log "Verifying (Containerfile target: verify)"
"${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}"
log "Image verified"
fi
# --- digest ------------------------------------------------------------------
DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target agent-runner \
--file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}"
# --- result ------------------------------------------------------------------
# No digest is printed: a manifest digest does not exist until a registry has the manifest,
# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at
# push time and prints it in the job summary (.gitea/workflows/scripts/images.sh).
SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
log "Built ${IMAGE_REF}"
info "local manifest digest ${DIGEST}"
info "size $(( SIZE / 1024 / 1024 )) MiB"
cat <<EOF
================================================================================
NOT PUSHED. Publishing is an operator action — this script holds no registry
credential and will never acquire one.
OPERATOR: run these, then put the digest from the digestfile into
platform/pm-agent/configmap.yaml (PM_WORKER_IMAGE) via a pull request:
${ENGINE} login gitea.home.keithjackson.dev
${ENGINE} push --digestfile /tmp/agent-runner.digest ${IMAGE_REF}
cat /tmp/agent-runner.digest
# -> sha256:.... THIS is the authoritative digest, not the local one above.
# platform/pm-agent/configmap.yaml
PM_WORKER_IMAGE: "${AGENT_IMAGE_REPO}@\$(cat /tmp/agent-runner.digest)"
WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/pm-agent-job-shape.yaml is a
fail-closed Enforce policy. It DENIES any worker Job whose image is not
${AGENT_IMAGE_REPO}@sha256:<digest> — a tag reference is denied,
and so is any other repository path. The dispatcher also refuses to start if
PM_WORKER_IMAGE carries no digest, so the failure names its own cause.
THE SAME IMAGE RUNS THE DISPATCHER. platform/pm-agent/deployment.yaml pins it too, so a
new digest is a two-line change in one pull request: the Deployment's image and
PM_WORKER_IMAGE. They must be the same digest — the dispatcher and its workers share the
committed Job template and the env-name contract, and a mismatched pair would disagree
about it.
TIGHTEN THE BOUNDARY AFTERWARDS: the policy admits the wildcard
\`${AGENT_IMAGE_REPO}@sha256:*\`, so anyone who can push to that registry path can publish
a different digest. Replacing the wildcard with the LITERAL digest closes that gap; it is a
one-line change, reviewed in a pull request, which is exactly where a change to the
security boundary belongs.
================================================================================
EOF
info "image id $("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
info "size $(( SIZE / 1024 / 1024 )) MiB"
info "Not pushed. CI publishes this image from main (.gitea/workflows/images.yaml); see"
info "platform/pm-agent/README.md for how a new digest reaches the cluster."
+6 -4
View File
@@ -16,13 +16,15 @@
# scrubbed environment and fake credentials, and if a guard ever stopped firing, the case
# would fail here rather than by really cloning something.
#
# Run offline, no cluster, no Gitea, no model. infra/agent/build-image.sh runs it inside the
# built image as well, where the interpreter and the script are the ones that will actually
# run in the cluster.
# Run offline, no cluster, no Gitea, no model. The Containerfile's `verify` stage runs it inside
# the built image as well (as uid 65532, no network), where the interpreter and the script are
# the ones that will actually run in the cluster — locally via build-image.sh and in CI.
###############################################################################
set -uo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# `:-$0` keeps this safe if it is ever fed over stdin (`bash -s`), where BASH_SOURCE is empty
# and `set -u` would abort on it.
HERE="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)"
WORKER="${PM_WORKER_PATH:-${HERE}/pm-worker}"
[ -r "$WORKER" ] || { echo "cannot read ${WORKER}" >&2; exit 2; }
+10 -4
View File
@@ -9,10 +9,16 @@
# truth and, the first time the two drifted, the worker's "green" would stop meaning the
# gate's green. So this file pins ONLY what the toolbox does not carry.
# The toolbox image the runner is layered on. Built by `infra/devcontainer/dev.sh build`
# (tag `home-k3s-dev:local`); infra/agent/build-image.sh asserts it exists rather than
# silently building a different base.
AGENT_BASE_IMAGE=home-k3s-dev:local
# The toolbox image the runner is layered on: the PUBLISHED toolbox, by digest. CI builds and
# publishes it (.gitea/workflows/images.yaml) and Renovate moves this line onto each new digest
# of its `main` tag via the marker below, in a pull request — nobody edits it by hand.
#
# ALL-ZERO UNTIL THE FIRST PUBLISH. The first run of the image pipeline on main sees this
# placeholder, builds and publishes the toolbox first, and builds agent-runner FROM that fresh
# digest. For a LOCAL build on a local toolbox, override it:
# infra/devcontainer/dev.sh build && AGENT_BASE_IMAGE=home-k3s-dev:local infra/agent/build-image.sh
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/toolbox tag=main
AGENT_BASE_IMAGE=gitea.home.keithjackson.dev/home-network/toolbox@sha256:0000000000000000000000000000000000000000000000000000000000000000
# Node.js — the runtime for the Claude Code CLI, installed from the Fedora base's module
# stream. A major version, not a patch: the CLI supports the active LTS line and the
+6
View File
@@ -80,6 +80,12 @@ spec:
# Replaced by the dispatcher with the digest-pinned image. A tag here would be
# DENIED by admission; the dispatcher refuses a non-digest PM_WORKER_IMAGE at
# startup so the error names the real cause.
#
# DELIBERATELY ALL-ZERO, AND DELIBERATELY NOT TRACKED BY RENOVATE. This file is
# baked INTO agent-runner, so it cannot carry agent-runner's own digest: bumping it
# would change the image's inputs, CI would publish a new digest, Renovate would bump
# it again — forever. pm-dispatch overwrites this field from PM_WORKER_IMAGE
# (platform/pm-agent/configmap.yaml, which IS tracked) on every Job it creates.
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000
imagePullPolicy: IfNotPresent
# The pinned entrypoint, with NO args. Args on a pinned command are
+49 -1
View File
@@ -8,7 +8,10 @@
# packages are retained as harmless disk-image utilities. Fedora base is pinned for
# reproducibility.
ARG FEDORA_VERSION=44
FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION}
# Stage `toolbox` is the image that ships. Stage `verify` (end of file) is a TEST stage: it is
# built FROM the finished image and only runs assertions, and it is never pushed. Build with
# `--target toolbox` for the image and `--target verify` to prove it (build-image.sh does both).
FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} AS toolbox
ARG OPENTOFU_VERSION
ARG AGE_VERSION
@@ -95,3 +98,48 @@ RUN { echo "opentofu=${OPENTOFU_VERSION}"; echo "age=${AGE_VERSION}"; echo "sops
WORKDIR /work
CMD ["/bin/bash"]
# =============================================================================
# verify — a TEST stage, never shipped. It runs every assertion against the finished image AT
# BUILD TIME, so CI needs no container runtime to prove the image: the image pipeline builds
# this target on rootless BuildKit (platform/buildkit/), where `docker run` does not exist.
# `RUN --network=none` reproduces the old `run --network none` exactly, and the first check
# below proves it (it fails if any network is reachable).
# =============================================================================
FROM toolbox AS verify
ARG PROMTOOL_VERSION
ARG KUBECTL_VERSION
ARG CHAINSAW_VERSION
ARG KYVERNO_VERSION
ARG SOPS_VERSION
RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
echo "--- no network (the build-time stand-in for run --network none) ---"
if getent hosts registry.fedoraproject.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then
echo "UNEXPECTED: network reachable in the verify stage"; exit 1
fi
echo " confirmed: no network"
echo "--- baked versions ---"; cat /etc/dev-toolchain-versions
echo "--- every tool is on PATH ---"
for t in tofu age age-keygen sops flux kubectl helm kyverno kyverno-json chainsaw yq \
yamllint check-jsonschema promtool ansible-playbook git make jq diff python3 \
wg qrencode; do
if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t";
else printf " MISSING %s\n" "$t"; missing=1; fi
done
[ -z "${missing:-}" ] || { echo "toolbox is missing tools"; exit 1; }
echo "--- version-sensitive tools report their pins ---"
: "${PROMTOOL_VERSION:?}" "${KUBECTL_VERSION:?}" "${CHAINSAW_VERSION:?}" "${KYVERNO_VERSION:?}" "${SOPS_VERSION:?}"
check() { # name pin actual-output
if printf "%s" "$3" | grep -qF "$2"; then printf " ok %-10s %s\n" "$1" "$2";
else printf " DRIFT %-10s want %s, got: %s\n" "$1" "$2" "$(printf "%s" "$3" | head -1)"; drift=1; fi
}
check promtool "${PROMTOOL_VERSION}" "$(promtool --version 2>&1)"
check kubectl "${KUBECTL_VERSION}" "$(kubectl version --client 2>&1)"
check chainsaw "${CHAINSAW_VERSION}" "$(chainsaw version 2>&1)"
check kyverno "${KYVERNO_VERSION}" "$(kyverno version 2>&1)"
check sops "${SOPS_VERSION}" "$(sops --version --disable-version-check 2>&1 || sops --version 2>&1)"
[ -z "${drift:-}" ] || { echo "a tool does not report its pinned version"; exit 1; }
echo "--- python can import what the renderer and the gate need ---"
python3 -c "import yaml; print(' PyYAML', yaml.__version__)"
echo "TOOLBOX VERIFIED"
VERIFY
+18
View File
@@ -34,6 +34,24 @@ cd infra/devcontainer
```
`ENGINE=docker ./dev.sh` to use docker instead of rootless podman.
### The published toolbox
`./dev.sh build` delegates to [`build-image.sh`](build-image.sh). The recipe is the
`Containerfile` plus that script's build-arg list, and the proof is the Containerfile's `verify`
stage (every tool on `PATH`, the version-sensitive ones report their `versions.env` pins, no
network). CI builds the same Containerfile on rootless BuildKit with the args from
`build-image.sh --print-build-args`, and, on a push to `main` that changes
`Containerfile`, `versions.env` or `build-image.sh`, **publishes** the result as
`gitea.home.keithjackson.dev/home-network/toolbox` (tags `sha-<commit>` and `main`) —
see [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). `agent-runner` is
built `FROM` that published digest (`infra/agent/versions.env`, kept current by Renovate).
Nobody pushes this image from a workstation. To use the published one locally instead of
building, after `podman login gitea.home.keithjackson.dev` (a token with `read:package`):
```bash
IMAGE=gitea.home.keithjackson.dev/home-network/toolbox:main ./dev.sh run make -C ci gate
```
## Choosing a cluster — `CLUSTER=dev|prod`
```bash
+84
View File
@@ -0,0 +1,84 @@
#!/usr/bin/env bash
###############################################################################
# infra/devcontainer/build-image.sh — build the pinned dev `toolbox` image and prove it works.
#
# ./build-image.sh # build + verify (tag: home-k3s-dev:local)
# ./build-image.sh --no-verify # build only
# IMAGE=foo:bar ./build-image.sh # build under another local name
# ENGINE=docker ./build-image.sh # build with docker instead of podman
# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes)
#
# ONE BUILD DEFINITION, TWO CALLERS. `dev.sh build` calls this for local development. The CI
# image pipeline (.gitea/workflows/images.yaml -> .gitea/workflows/scripts/images.sh) builds the
# SAME Containerfile on rootless BuildKit (platform/buildkit/) — no container engine exists in a
# CI job — taking its build args from `--print-build-args`, and PUBLISHES the result as
# gitea.home.keithjackson.dev/home-network/toolbox. The Containerfile (including its `verify`
# test stage) and the arg list here are the one recipe both use.
#
# THIS SCRIPT NEVER PUSHES. Publishing is CI's job (images.yaml, on push to main), not a
# workstation's: an image that exists only because someone ran `podman push` from a laptop has
# no build log, no recorded inputs and no verification anyone else can read.
#
# VERSIONS ARE NOT DEFINED HERE. infra/devcontainer/versions.env is the single source of truth.
###############################################################################
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENGINE="${ENGINE:-podman}"
IMAGE="${IMAGE:-home-k3s-dev:local}"
VERIFY=1
PRINT_ARGS=0
for arg in "$@"; do
case "$arg" in
--no-verify) VERIFY=0 ;;
--print-build-args) PRINT_ARGS=1 ;;
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown argument: $arg" >&2; exit 2 ;;
esac
done
log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
# shellcheck disable=SC1091
source "${HERE}/versions.env"
# THE build args, in ONE place. `--print-build-args` hands exactly this list to the CI pipeline
# (.gitea/workflows/scripts/images.sh), which builds on rootless BuildKit rather than calling
# "$ENGINE build" — so CI and a local build cannot drift apart on a pin.
BUILD_ARGS=(
"FEDORA_VERSION=$FEDORA_VERSION" "OPENTOFU_VERSION=$OPENTOFU_VERSION" "AGE_VERSION=$AGE_VERSION"
"SOPS_VERSION=$SOPS_VERSION" "FLUX_VERSION=$FLUX_VERSION" "KUBECTL_VERSION=$KUBECTL_VERSION"
"HELM_VERSION=$HELM_VERSION" "KYVERNO_VERSION=$KYVERNO_VERSION"
"KYVERNO_JSON_VERSION=$KYVERNO_JSON_VERSION" "CHAINSAW_VERSION=$CHAINSAW_VERSION"
"YAMLLINT_VERSION=$YAMLLINT_VERSION" "YQ_VERSION=$YQ_VERSION"
"CHECK_JSONSCHEMA_VERSION=$CHECK_JSONSCHEMA_VERSION" "PROMTOOL_VERSION=$PROMTOOL_VERSION"
)
if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
ARG_FLAGS=()
for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
LABEL_ARGS=()
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
# --- verify --------------------------------------------------------------------
# The assertions live in the Containerfile's `verify` stage (every tool on PATH, the
# version-sensitive ones report their versions.env pins — promtool above all, because
# ci/versions.env asserts a promtool/Prometheus lockstep — and no network), so the SAME proof
# runs here under podman and in CI under rootless BuildKit. A failing assertion fails the build.
if [ "${VERIFY}" -eq 1 ]; then
log "Verifying (Containerfile target: verify)"
"$ENGINE" build "${ARG_FLAGS[@]}" --target verify -f "${HERE}/Containerfile" "${HERE}"
log "Toolbox verified"
fi
log "Building ${IMAGE}"
"$ENGINE" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target toolbox \
-t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}"
log "Built ${IMAGE}"
printf ' image id %s\n' "$("$ENGINE" image inspect --format '{{.Id}}' "$IMAGE")"
+4 -16
View File
@@ -72,22 +72,10 @@ esac
CLUSTER_KUBECONFIG="${REPO_ROOT}/infra/ansible/.cache/${CLUSTER}.kubeconfig"
build() {
"$ENGINE" build \
--build-arg FEDORA_VERSION="$FEDORA_VERSION" \
--build-arg OPENTOFU_VERSION="$OPENTOFU_VERSION" \
--build-arg AGE_VERSION="$AGE_VERSION" \
--build-arg SOPS_VERSION="$SOPS_VERSION" \
--build-arg FLUX_VERSION="$FLUX_VERSION" \
--build-arg KUBECTL_VERSION="$KUBECTL_VERSION" \
--build-arg HELM_VERSION="$HELM_VERSION" \
--build-arg KYVERNO_VERSION="$KYVERNO_VERSION" \
--build-arg KYVERNO_JSON_VERSION="$KYVERNO_JSON_VERSION" \
--build-arg CHAINSAW_VERSION="$CHAINSAW_VERSION" \
--build-arg YAMLLINT_VERSION="$YAMLLINT_VERSION" \
--build-arg YQ_VERSION="$YQ_VERSION" \
--build-arg CHECK_JSONSCHEMA_VERSION="$CHECK_JSONSCHEMA_VERSION" \
--build-arg PROMTOOL_VERSION="$PROMTOOL_VERSION" \
-t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}"
# The build recipe lives in build-image.sh so that CI (.gitea/workflows/images.yaml), which
# builds and PUBLISHES this image, and a local `dev.sh build` cannot drift apart. --no-verify
# keeps the local loop as fast as it always was; CI runs the verification.
IMAGE="$IMAGE" ENGINE="$ENGINE" "${HERE}/build-image.sh" --no-verify
}
run() {
+104
View File
@@ -0,0 +1,104 @@
#!/usr/bin/env bash
# Copy the image builder's CLIENT certificate from the cluster into the three repository Actions
# secrets the image pipeline reads: BUILDKIT_CA, BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY.
#
# ./infra/gitea/buildkit-client-secret.sh # copy and verify
# ./infra/gitea/buildkit-client-secret.sh --check # compare only; changes nothing
#
# RUNNING IT IS AN OPERATOR ACTION. It writes repository secrets.
#
# WHERE THE CERTIFICATE COMES FROM. platform/buildkit/certificates.yaml has cert-manager issue
# `buildkit-client-ci` from a CA that exists only for the builder; buildkitd admits a connection
# only with a certificate signed by that CA. CI jobs hold NO cluster credential (R-SEC-02), so they
# cannot read the Secret themselves: this script is the one bridge, run by a human who has both
# a kube context and the Gitea credential.
#
# WHEN TO RUN IT
# 1. once, after Flux has applied platform/buildkit (Kustomization `buildkit` Ready, and
# `kubectl -n buildkit get certificate buildkit-client-ci` READY=True);
# 2. after every renewal of that certificate (duration 1y, renewed 30 days early). Until then
# the image pipeline fails LOUDLY at its builder preflight — it never skips.
# `--check` tells you whether the copy is stale.
#
# NOTHING SECRET IS PRINTED. The key goes from `kubectl get secret` into the Gitea secrets API
# through shell variables and nowhere else. The script prints only names, sizes and the
# certificate's subject and expiry (public data).
#
# NEEDS: kubectl with a context for the cluster (default k3s-prod), jq, curl, openssl, base64,
# and the Gitea credential `git` already has stored for the host — the same requirements as
# infra/gitea/conformance-trigger-kubeconfig.sh, run from the same place.
set -euo pipefail
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
NS=buildkit; SECRET=buildkit-client-ci; KCTX="${KCTX:-k3s-prod}"
CHECK=0
for a in "$@"; do
case "$a" in
--check) CHECK=1 ;;
-h|--help) sed -n '2,/^set -euo/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown argument: $a" >&2; exit 2 ;;
esac
done
die() { echo "ERROR: $*" >&2; exit 1; }
for t in kubectl jq curl openssl base64 git; do command -v "$t" >/dev/null || die "missing required tool: $t"; done
echo "==> reading Secret ${NS}/${SECRET} (context ${KCTX}; values not shown)"
js="$(kubectl --context "$KCTX" -n "$NS" get secret "$SECRET" -o json)" \
|| die "cannot read ${NS}/${SECRET} — is platform/buildkit deployed and the Certificate Ready?"
CA="$(jq -r '.data["ca.crt"] // empty' <<<"$js" | base64 -d)"
CRT="$(jq -r '.data["tls.crt"] // empty' <<<"$js" | base64 -d)"
KEY="$(jq -r '.data["tls.key"] // empty' <<<"$js" | base64 -d)"
unset js
[ -n "$CA" ] && [ -n "$CRT" ] && [ -n "$KEY" ] || die "${NS}/${SECRET} lacks ca.crt/tls.crt/tls.key"
# Sanity before anything is written: the leaf is a CLIENT cert issued by this CA, and the key
# belongs to it. A wrong pairing would otherwise surface only as a TLS failure in CI.
openssl verify -CAfile <(printf '%s\n' "$CA") <(printf '%s\n' "$CRT") >/dev/null 2>&1 \
|| die "tls.crt does not verify against ca.crt"
openssl x509 -noout -ext extendedKeyUsage <<<"$CRT" 2>/dev/null | grep -q "TLS Web Client Authentication" \
|| die "tls.crt is not a client-auth certificate"
[ "$(openssl x509 -noout -pubkey <<<"$CRT" | openssl sha256)" = "$(openssl pkey -pubout <<<"$KEY" 2>/dev/null | openssl sha256)" ] \
|| die "tls.key does not match tls.crt"
echo " $(openssl x509 -noout -subject <<<"$CRT") / $(openssl x509 -noout -enddate <<<"$CRT")"
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"; unset creds
[ -n "$U" ] && [ -n "$P" ] || die "no stored credential for $HOST"
echo "==> authenticating to Gitea as '${U}' (secret not shown)"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
API="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets"
if [ "$CHECK" -eq 1 ]; then
# Gitea never returns a secret's value, only its name and creation time. "Stale" therefore
# means: the secret was created before the certificate's current notBefore.
names="$(api "$API" | jq -c '[.[]? | {name, created_at}]')"
nb="$(date -u -d "$(openssl x509 -noout -startdate <<<"$CRT" | cut -d= -f2)" +%s)"
rc=0
for n in BUILDKIT_CA BUILDKIT_CLIENT_CERT BUILDKIT_CLIENT_KEY; do
c="$(jq -r --arg n "$n" '.[] | select(.name==$n) | .created_at' <<<"$names")"
if [ -z "$c" ]; then echo " MISSING $n"; rc=1
elif [ "$(date -u -d "$c" +%s)" -lt "$nb" ]; then echo " STALE $n (stored $c, certificate re-issued since)"; rc=1
else echo " current $n (stored $c)"; fi
done
exit "$rc"
fi
put() { # name value
local code
code="$(api -o /dev/null -w '%{http_code}' -X PUT -H 'Content-Type: application/json' \
-d "$(jq -n --arg d "$2" '{data:$d}')" "${API}/$1")"
[ "$code" = 201 ] || [ "$code" = 204 ] || die "PUT $1 failed: HTTP ${code}"
echo " stored $1 (${#2} bytes, value not shown)"
}
echo "==> storing repository Actions secrets"
put BUILDKIT_CA "$CA"
put BUILDKIT_CLIENT_CERT "$CRT"
put BUILDKIT_CLIENT_KEY "$KEY"
unset CA CRT KEY
# VERIFY BY READING BACK (names only — Gitea never returns values). curl -sS exits 0 on an HTTP
# error, which is how set-ci-token.sh once claimed a store that never happened.
names="$(api "$API" | jq -r '[.[]?.name] | join(" ")')"
for n in BUILDKIT_CA BUILDKIT_CLIENT_CERT BUILDKIT_CLIENT_KEY; do
grep -qw "$n" <<<"$names" || die "stored $n but it is NOT listed (${names}) — refusing to claim success"
done
echo "==> stored and VERIFIED present. Re-run the images workflow (workflow_dispatch) to use them."
@@ -252,11 +252,16 @@ step "preconditions — repo side (these block ONLY arming ${VARIABLE}; the secr
# R1: the runner image is built and digest-pinned. Without it the Job cannot run run.sh at
# all (the stock chainsaw image has no bash/python3/kubectl/git) — plan §3 item 1.
# EVALUATED, not grepped: the line is `CONFORMANCE_RUNNER_IMAGE="${..._REPO}@${..._DIGEST}"`, so
# its literal text never contains 64 hex digits and the grep this replaced could never pass. The
# all-zero digest is the "not published yet" placeholder (Renovate replaces it after CI's first
# publish, .gitea/workflows/images.yaml) and must not count as pinned.
VERSIONS="${REPO_ROOT}/conformance/versions.env"
if grep -Eq '^[[:space:]]*CONFORMANCE_RUNNER_IMAGE=.*@sha256:[0-9a-f]{64}' "$VERSIONS" 2>/dev/null; then
pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env"
runner_img="$( (set +u; . "$VERSIONS" >/dev/null 2>&1; printf '%s' "${CONFORMANCE_RUNNER_IMAGE:-}") )"
if [[ "$runner_img" =~ @sha256:[0-9a-f]{64}$ ]] && [[ ! "$runner_img" =~ @sha256:0{64}$ ]]; then
pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env (${runner_img##*@})"
else
fail_arm "conformance/versions.env has no digest-pinned CONFORMANCE_RUNNER_IMAGE (plan §3 item 1) — the Job cannot run ./run.sh"
fail_arm "conformance/versions.env has no PUBLISHED digest for CONFORMANCE_RUNNER_IMAGE (got '${runner_img}') — the image pipeline has not published it yet, or Renovate's bump is unmerged (plan §3 item 1)"
fi
# R2: the Job template is the real thing, not the skeleton: the admitted image AND an
+130
View File
@@ -0,0 +1,130 @@
#!/usr/bin/env bash
# Registry retention for this repo's OWN images (toolbox, agent-runner, conformance-runner):
# keep what is deployed and what is recent, delete the rest. DRY RUN unless --apply.
#
# ./infra/gitea/registry-retention.sh # show what would be deleted
# ./infra/gitea/registry-retention.sh --apply # delete it
# KEEP=15 ./infra/gitea/registry-retention.sh # keep the 15 newest sha- tags (default 10)
#
# RUNNING IT IS AN OPERATOR ACTION. It deletes registry content, so it is not wired into CI and
# no agent runs it with --apply.
#
# WHY A SCRIPT, AND NOT GITEA'S BUILT-IN "CLEANUP RULE" — measured against Gitea 1.24.3's source:
#
# 1. There is NO API for cleanup rules. routers/api/v1/api.go has no such route; the rules live
# only behind the web UI's session+CSRF forms (routers/web/web.go, org settings
# `/packages/rules/add`). A rule therefore cannot be created, read or drift-checked as code —
# which is also why ci/gitea-settings.sh cannot assert one exists.
# 2. A built-in rule would DELETE WHAT IS DEPLOYED. Every reference to these images is
# digest-only (`<repo>@sha256:...`, required by admission). Gitea stores a manifest as a
# file OF THE TAG VERSION it was pushed under (routers/api/packages/container/manifest.go
# createPackageAndVersion), and resolves a by-digest pull by searching those files
# (models/packages/container/search.go). So deleting the tag a digest was pushed under makes
# that digest unpullable — MANIFEST_UNKNOWN, i.e. ImagePullBackOff on the next reschedule.
# The built-in rule keeps "the newest N" and a name pattern, and knows nothing about which
# digest git currently pins. Renovate's bump PRs can lag, so "newest N" is not a guarantee.
#
# WHAT IS KEPT, per image:
# * the `main` and `latest` tags;
# * the KEEP newest `sha-<commit>` tags (by creation time);
# * EVERY tag whose manifest digest is referenced anywhere in the tracked tree at REF
# (default origin/main) — i.e. whatever Flux is deploying, and the toolbox digest
# agent-runner builds FROM;
# * every version whose NAME is a digest (`sha256:...`): those are index children, and Gitea's
# own cleanup handles them (services/packages/container/cleanup.go ShouldBeSkipped).
# Everything else is deleted. Gitea's `cleanup_packages` cron then reclaims the unreferenced
# blobs (default: daily, blobs older than 24h) — space comes back the day after, not instantly.
#
# THE STORAGE IT PROTECTS. Packages live on the SAME 10Gi Longhorn PVC as every repository and
# Gitea's database (catalog/gitea.yaml). The toolbox is ~1.4 GB uncompressed and every main push
# that touches its inputs adds a complete, non-reproducible copy. Unpruned, the registry is what
# fills the volume the git host lives on.
#
# CREDENTIAL: the one `git` has stored for the host (as the other scripts here). It needs
# `read:package` to list and `write:package` to --apply, and the account needs package write on
# the org. The secret is never printed.
set -euo pipefail
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; NS=home-network
IMAGES=(toolbox agent-runner conformance-runner)
KEEP="${KEEP:-10}"
REF="${REF:-origin/main}"
APPLY=0
for a in "$@"; do
case "$a" in
--apply) APPLY=1 ;;
-h|--help) sed -n '2,/^set -euo/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown argument: $a" >&2; exit 2 ;;
esac
done
[[ "$KEEP" =~ ^[0-9]+$ ]] && [ "$KEEP" -ge 1 ] || { echo "KEEP must be a positive integer" >&2; exit 2; }
command -v jq >/dev/null || { echo "jq is required" >&2; exit 1; }
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
git -C "$REPO_ROOT" fetch -q origin main 2>/dev/null || echo " (could not fetch origin/main; using the local ${REF})" >&2
git -C "$REPO_ROOT" rev-parse -q --verify "${REF}^{commit}" >/dev/null || { echo "no such ref: ${REF}" >&2; exit 1; }
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
echo "authenticating as '${U}' (secret not shown)"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
REG_TOKEN="$(api "https://${HOST}/v2/token?service=container_registry" | jq -r '.token // empty')"
[ -n "$REG_TOKEN" ] || { echo "registry token exchange failed (does the credential have read:package?)" >&2; exit 1; }
manifest_digest() { # <image> <tag> -> digest, or nothing
curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "$REG_TOKEN") \
-H 'Accept: application/vnd.oci.image.manifest.v1+json' \
-H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
-H 'Accept: application/vnd.oci.image.index.v1+json' \
-H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
"https://${HOST}/v2/${NS}/$1/manifests/$2" \
| tr -d '\r' | awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' | tail -1
}
TOTAL_DEL=0
for img in "${IMAGES[@]}"; do
echo
echo "==> ${NS}/${img}"
# Digests git pins at REF. The marker convention puts the digest on the line after
# `# renovate-digest: image=<repo>`; the plain `<repo>@sha256:` grep catches any other use.
repo="${HOST}/${NS}/${img}"
pinned="$( { git -C "$REPO_ROOT" grep -h -A1 -F "renovate-digest: image=${repo} " "$REF" -- . 2>/dev/null || true
git -C "$REPO_ROOT" grep -h -F "${repo}@sha256:" "$REF" -- . 2>/dev/null || true; } \
| grep -oE 'sha256:[0-9a-f]{64}' | grep -v 'sha256:0\{64\}' | sort -u || true)"
echo " pinned at ${REF}: $(wc -w <<<"$pinned") digest(s)"
versions="[]"; page=1
while :; do
chunk="$(api "https://${HOST}/api/v1/packages/${OWNER}?type=container&q=${img}&limit=50&page=${page}")"
jq -e 'type=="array"' >/dev/null <<<"$chunk" || { echo " package list failed: $(head -c 200 <<<"$chunk")" >&2; exit 1; }
[ "$(jq length <<<"$chunk")" -gt 0 ] || break
versions="$(jq -c --argjson a "$versions" --arg n "$img" '$a + [ .[] | select(.name==$n) | {version, created_at} ]' <<<"$chunk")"
page=$((page + 1))
done
echo " versions: $(jq length <<<"$versions")"
keep_sha="$(jq -r '[ .[] | select(.version|startswith("sha-")) ] | sort_by(.created_at) | reverse | .[:'"$KEEP"'][] .version' <<<"$versions")"
while IFS= read -r v; do
[ -n "$v" ] || continue
case "$v" in
main|latest) echo " keep ${v} (moving tag)"; continue ;;
sha256:*) echo " keep ${v} (digest-named version; Gitea's cleanup owns those)"; continue ;;
esac
if grep -qxF "$v" <<<"$keep_sha"; then echo " keep ${v} (among the ${KEEP} newest)"; continue; fi
d="$(manifest_digest "$img" "$v")"
if [ -z "$d" ]; then echo " keep ${v} (could not read its digest — never delete what cannot be checked)"; continue; fi
if grep -qxF "$d" <<<"$pinned"; then echo " keep ${v} (${d} is pinned at ${REF})"; continue; fi
if [ "$APPLY" -eq 1 ]; then
code="$(api -o /dev/null -w '%{http_code}' -X DELETE \
"https://${HOST}/api/v1/packages/${OWNER}/container/${img}/$(jq -rn --arg v "$v" '$v|@uri')")"
if [ "$code" = 204 ]; then echo " DELETED ${v} (${d})"; TOTAL_DEL=$((TOTAL_DEL + 1));
else echo " FAILED ${v} HTTP ${code}" >&2; fi
else
echo " would delete ${v} (${d})"; TOTAL_DEL=$((TOTAL_DEL + 1))
fi
done < <(jq -r '.[].version' <<<"$versions")
done
echo
if [ "$APPLY" -eq 1 ]; then echo "deleted ${TOTAL_DEL} version(s). Blobs are reclaimed by Gitea's cleanup_packages cron."
else echo "DRY RUN: ${TOTAL_DEL} version(s) would be deleted. Re-run with --apply to delete them."; fi
+74
View File
@@ -0,0 +1,74 @@
# platform/buildkit — rootless BuildKit, the CI image builder
The builder behind [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). It
builds `toolbox`, `agent-runner` and `conformance-runner`, and on `main` pushes them to the
Gitea registry. Flux reconciles it as the Kustomization `buildkit`
([`clusters/base/buildkit.yaml`](../../clusters/base/buildkit.yaml), `dependsOn: platform`).
## Why it exists (operator decision on PR #81, 2026-10-29)
CI jobs run as containers on the act runner's privileged DinD sidecar, and **cannot reach any
Docker daemon**. act_runner 0.2.11 forces `docker_host: "-"` for a `tcp://` DinD, and gitea/act
strips every bind mount when `valid_volumes` is empty. Measured on PR #81's first run. Giving jobs
that daemon would make every branch push root on the node. So the rule is: **no job may reach a
privileged daemon.** Builds go to this rootless daemon over mTLS instead.
## Shape
| Piece | What it is |
|---|---|
| `deployment.yaml` | `moby/buildkit:v0.34.1-rootless`, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The `securityContext` is commented line by line with how each item was measured |
| `certificates.yaml` | cert-manager: a self-signed CA used only here, plus the server cert `buildkitd-tls` and the CI client cert `buildkit-client-ci` |
| `service.yaml` | ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`, mTLS only. Never on the Gateway |
| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and, for the registry push, the Gateway (`ingress` entity) **plus** Gitea's backend identity on :3000 — Cilium's Gateway re-checks this pod's egress against the backend it proxies to, and without that rule answers `403 Access denied` itself (measured 2026-10-30, `decisions/2026-10-30-gitea-admit-buildkit.md`) |
| `configmap.yaml` | `buildkitd.toml`: cache GC capped at 4 GB, because the cache is an `emptyDir` on node disk |
| `versions.env` | The CI side: pinned `buildx` (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource |
## Admission
**No Kyverno exception is needed.**
- Every ClusterPolicy that is actually applied (the list in
[`platform-config/kustomization.yaml`](../../platform-config/kustomization.yaml)) matches only
namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`. This namespace, like
`gitea-actions`, does not carry that label.
- The job-shape policies (`pm-agent-job-shape`, `conformance-trigger-job-shape`) match only their
own namespaces.
- There is no cluster-wide Pod Security default. This namespace is deliberately left without a
`pod-security` label, because PSS `baseline` forbids the `Unconfined` seccomp and AppArmor
profiles that rootless BuildKit needs.
## Prerequisites (operator, in order)
1. **Unprivileged user namespaces on the nodes.** Already true. Measured 2026-10-29 on all three
Debian 13.7 nodes (kernel 7.1.8): `kernel.unprivileged_userns_clone = 1`,
`user.max_user_namespaces = 127206`, and no Ubuntu-style AppArmor userns restriction.
2. **Merge.** Flux applies this directory. Check that the Kustomization `buildkit` is Ready and
that all three Certificates in namespace `buildkit` are READY.
3. **Copy the client certificate to CI.** Run
[`infra/gitea/buildkit-client-secret.sh`](../../infra/gitea/buildkit-client-secret.sh).
Re-run it yearly, after renewal; `--check` reports whether the copy is stale.
4. **Restart buildkitd after the server certificate renews.** It reads the certificate at start,
and cert-manager renews it 30 days before expiry, so a restart inside that window is needed.
Until step 3 is done, the `images` workflow fails at its builder preflight with a message naming
the cause. It never skips.
## What a build step can do
The daemon runs with `--oci-worker-no-process-sandbox`, so build steps share its PID namespace.
buildkitd warns about this itself at startup.
- **A build step CAN** signal or ptrace the daemon and read what it can read, including this
pod's TLS key and the build cache.
- **A build step CANNOT** reach the node. Everything runs as uid 1000 in a user namespace,
unprivileged, with no host mounts.
- **Who can run one:** anyone holding the client certificate, i.e. every CI job, including pull
requests.
## Local equivalent
The pinned image runs under rootless podman with the same restrictions:
`--cap-drop ALL --cap-add SETUID --cap-add SETGID --security-opt seccomp=unconfined --read-only`,
plus tmpfs at the same five paths. That is how the `securityContext` was measured. Rootless podman
maps only 65,537 uids, so the image's `/etc/subuid` (`user:100000:65536`) must be overridden with
two ranges that fit: `user:1:999` and `user:1001:64535`.
+115
View File
@@ -0,0 +1,115 @@
---
# mTLS for buildkitd, issued by cert-manager from a CA that exists ONLY for this purpose.
#
# buildkit-selfsigned (Issuer) --signs--> buildkit-ca (CA Certificate, 10y)
# buildkit-ca (Issuer) --signs--> buildkitd-tls (server, the daemon presents it)
# --signs--> buildkit-client-ci (client, CI presents it)
#
# buildkitd verifies EVERY client certificate against this CA (--tlscacert), so holding a client
# certificate signed by it is what "may build" means. Nothing else in the cluster trusts this CA,
# and it trusts nothing else.
#
# THE CLIENT CERTIFICATE REACHES CI AS ACTIONS SECRETS, not through the cluster: CI jobs hold no
# cluster credential (R-SEC-02). infra/gitea/buildkit-client-secret.sh copies ca.crt/tls.crt/
# tls.key from the Secret below into the repository secrets BUILDKIT_CA, BUILDKIT_CLIENT_CERT and
# BUILDKIT_CLIENT_KEY without printing them. Re-run it after every renewal (see renewBefore).
#
# WHY A PRIVATE CA AND NOT LET'S ENCRYPT. The platform's TLS rule (R-TLS-01/02, docs/08,
# ADR-0006: ACME DNS-01) is about EXTERNALLY REACHABLE services; ADR-0006 itself says "mTLS between
# workloads is a separate concern … not covered here", and a public CA cannot issue for a
# `*.svc.cluster.local` name or a client-auth certificate anyway. buildkitd is ClusterIP-only. Each
# Certificate below is therefore annotated `homelab.keithjackson.dev/tls-scope: workload-mtls`,
# which the L0 check tls-lifecycle-automated accepts ONLY if the certificate proves it: no
# Gateway serves its Secret, every name is cluster-internal, and the issuer is a namespaced
# selfSigned/CA Issuer whose CA is itself issued here. Issuance and renewal stay unattended.
#
# ROTATION. cert-manager renews both leaves `renewBefore` their expiry and rewrites the Secrets.
# * server: buildkitd reads its certificate at START, so the pod must restart inside the 30-day
# renewal window (a node reboot by kured, a Deployment rollout, or a manual restart).
# * client: the Actions secrets are a COPY; re-run the script. Until then CI fails loudly at
# its TLS preflight (.gitea/workflows/scripts/images.sh), never silently.
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: buildkit-selfsigned
namespace: buildkit
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: buildkit-ca
namespace: buildkit
annotations:
homelab.keithjackson.dev/tls-scope: workload-mtls # see the header: R-TLS-01 is about external TLS
spec:
isCA: true
commonName: buildkit-ca
secretName: buildkit-ca
duration: 87600h # 10 years: rotating the CA means re-issuing everything and re-copying
privateKey:
algorithm: ECDSA
size: 256
issuerRef:
name: buildkit-selfsigned
kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: buildkit-ca
namespace: buildkit
spec:
ca:
secretName: buildkit-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: buildkitd-tls
namespace: buildkit
annotations:
homelab.keithjackson.dev/tls-scope: workload-mtls
spec:
secretName: buildkitd-tls
commonName: buildkitd.buildkit.svc.cluster.local
dnsNames:
- buildkitd.buildkit.svc.cluster.local # what CI dials (platform/buildkit/versions.env)
- buildkitd.buildkit.svc
- buildkitd
duration: 8760h
renewBefore: 720h
usages:
- server auth
- digital signature
privateKey:
algorithm: ECDSA
size: 256
rotationPolicy: Always
issuerRef:
name: buildkit-ca
kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: buildkit-client-ci
namespace: buildkit
annotations:
homelab.keithjackson.dev/tls-scope: workload-mtls
spec:
secretName: buildkit-client-ci
commonName: gitea-actions-image-pipeline
duration: 8760h
renewBefore: 720h
usages:
- client auth
- digital signature
privateKey:
algorithm: ECDSA
size: 256
rotationPolicy: Always
issuerRef:
name: buildkit-ca
kind: Issuer
+24
View File
@@ -0,0 +1,24 @@
---
# buildkitd.toml. Validated against the pinned image (v0.34.1-rootless): `buildctl debug workers
# -v` reports "Maximum used space: 4.29GB / Reserved space: 1.07GB" for the default policy.
#
# WHY 4GB. The cache lives in an emptyDir on the NODE's disk (deployment.yaml), and the nodes had
# 7-9 GB free when this was written. The largest image (toolbox) is ~1.4 GB uncompressed; the
# three images share their Fedora base. The emptyDir's sizeLimit sits above this cap, so GC
# trims the cache well before the kubelet would evict the pod for exceeding it.
apiVersion: v1
kind: ConfigMap
metadata:
name: buildkitd-config
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
data:
buildkitd.toml: |
[worker.oci]
enabled = true
gc = true
maxUsedSpace = "4GB"
reservedSpace = "1GB"
[worker.containerd]
enabled = false
+142
View File
@@ -0,0 +1,142 @@
---
# buildkitd, ROOTLESS. The builder behind .gitea/workflows/images.yaml; see README.md.
#
# THE SECURITY CONTEXT IS THE BOUNDARY, AND EVERY LINE OF IT WAS MEASURED (2026-10-29, the pinned
# image under podman with the same restrictions; decisions/2026-10-29-image-pipeline.md):
# * privileged: false — never. This is the whole point of the operator's decision on PR #81:
# the alternative was handing CI the privileged DinD daemon, i.e. root on the node.
# * runAsUser/Group 1000 — the image's `user`; rootlesskit maps it to uid 0 INSIDE a user
# namespace only. The nodes allow that (measured: kernel.unprivileged_userns_clone = 1,
# user.max_user_namespaces = 127206 on all three Debian 13 nodes).
# * capabilities: drop ALL, add SETUID + SETGID — the image's newuidmap/newgidmap are NOT
# setuid; they carry FILE capabilities, which need SETUID/SETGID in the bounding set.
# Measured: with drop ALL and nothing added, rootlesskit fails "newuidmap ... operation not
# permitted" and the daemon never starts.
# * allowPrivilegeEscalation: true — required for the same reason: no_new_privs (what `false`
# sets) stops a file capability from ever taking effect. It does NOT grant privileges the
# bounding set above does not contain.
# * seccompProfile: Unconfined — measured: with the default seccomp profile the daemon starts
# but every RUN step fails (the runtime cannot create the build step's namespaces/mounts).
# * appArmorProfile: Unconfined — upstream's documented requirement (buildkit
# examples/kubernetes, *.rootless.yaml); the nodes run AppArmor (enabled=Y) and containerd's
# default profile denies mount. NOT measured here: the test host runs SELinux, not AppArmor.
# * readOnlyRootFilesystem: true — measured to work with the five emptyDirs below.
#
# --oci-worker-no-process-sandbox: build steps share the daemon's PID namespace instead of getting
# their own (creating one needs an unmasked /proc, i.e. more privilege). Consequence, stated by
# buildkitd itself at startup: a build step can signal or ptrace the daemon and read what the
# daemon can read — including this pod's TLS key and the build cache. So a build step is trusted
# at the level of "may impersonate this builder", which is the level anyone holding the client
# certificate already has. It can NOT reach the node: everything still runs as uid 1000 in a user
# namespace, unprivileged, with no host mounts.
apiVersion: apps/v1
kind: Deployment
metadata:
name: buildkitd
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
spec:
replicas: 1
strategy:
type: Recreate # one builder, one cache; never two daemons fighting over a node's disk
selector:
matchLabels:
app.kubernetes.io/name: buildkitd
template:
metadata:
labels:
app.kubernetes.io/name: buildkitd
spec:
automountServiceAccountToken: false # the builder needs no Kubernetes API at all
enableServiceLinks: false
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000 # makes the 0440 TLS Secret below readable by uid 1000, and nobody else
containers:
- name: buildkitd
# Pinned by digest (the multi-arch index). The kubernetes manager in renovate.json5
# tracks platform/**, so Renovate proposes bumps; the CI client (buildx, pinned in
# versions.env) talks to any recent daemon.
image: docker.io/moby/buildkit:v0.34.1-rootless@sha256:f68752a5a94acfcdad2d07867b7075601a421638e17fc5f28efc1c814d0c4f89
args:
- --addr
- unix:///run/user/1000/buildkit/buildkitd.sock # local only: the probes below
- --addr
- tcp://0.0.0.0:1234 # CI, mTLS only
- --tlscacert
- /certs/ca.crt # client certs are VERIFIED against this CA: no cert, no build
- --tlscert
- /certs/tls.crt
- --tlskey
- /certs/tls.key
- --config
- /etc/buildkit/buildkitd.toml
- --oci-worker-no-process-sandbox
ports:
- name: buildkit
containerPort: 1234
protocol: TCP
readinessProbe:
exec:
command: ["buildctl", "debug", "workers"]
initialDelaySeconds: 5
periodSeconds: 30
livenessProbe:
exec:
command: ["buildctl", "debug", "workers"]
initialDelaySeconds: 30
periodSeconds: 60
timeoutSeconds: 10
securityContext:
privileged: false
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
allowPrivilegeEscalation: true
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
add: ["SETUID", "SETGID"]
seccompProfile:
type: Unconfined
appArmorProfile:
type: Unconfined
resources:
# Build steps (dnf, npm, pip) run INSIDE this container's cgroup, so the limit covers
# the daemon and the build together.
requests:
cpu: 250m
memory: 512Mi
ephemeral-storage: 2Gi
limits:
memory: 4Gi
volumeMounts:
- { name: certs, mountPath: /certs, readOnly: true }
- { name: config, mountPath: /etc/buildkit, readOnly: true }
- { name: cache, mountPath: /home/user/.local/share/buildkit }
- { name: run, mountPath: /run/user }
- { name: tmpdir, mountPath: /home/user/.local/tmp } # the image's TMPDIR
- { name: tmp, mountPath: /tmp }
- { name: clicfg, mountPath: /home/user/.docker } # buildctl (the probes) writes here
volumes:
- name: certs
secret:
secretName: buildkitd-tls # cert-manager: ca.crt + tls.crt + tls.key
defaultMode: 0440
- name: config
configMap:
name: buildkitd-config
- name: cache
emptyDir:
sizeLimit: 6Gi # GC caps use at 4GB (configmap.yaml); this is the backstop
- name: run
emptyDir: {}
- name: tmpdir
emptyDir: {}
- name: tmp
emptyDir: {}
- name: clicfg
emptyDir: {}
+13
View File
@@ -0,0 +1,13 @@
---
# platform/buildkit — rootless BuildKit, the CI image builder. Reconciled by the Flux
# Kustomization `buildkit` (clusters/base/buildkit.yaml). See README.md.
# versions.env is NOT a resource: it pins the CI-side client (buildx) and the address CI dials.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- certificates.yaml
- configmap.yaml
- deployment.yaml
- service.yaml
- networkpolicy.yaml
+26
View File
@@ -0,0 +1,26 @@
---
# buildkit — the image builder for CI (.gitea/workflows/images.yaml). Operator decision on PR #81
# (2026-10-29): images are built by a ROOTLESS BuildKit daemon reached over mTLS, and NO CI job
# may reach a privileged daemon. See README.md for the whole design.
#
# A NAMESPACE OF ITS OWN, deliberately NOT `gitea-actions`:
# * The act-runner namespace exists to hold a PRIVILEGED DinD sidecar. Putting the builder
# beside it would put a TLS server key next to a privileged container for no benefit.
# * Its NetworkPolicies, Secrets and quota then belong to one component, and the ingress rule
# ("only the act-runner pod may connect") can be written from the outside.
#
# NO `app-namespace` LABEL, like gitea-actions: this is CI plumbing, not a user-facing app. That
# is also why NO Kyverno exception is needed — every applied ClusterPolicy
# (platform-config/kustomization.yaml) matches only namespaces labelled
# homelab.keithjackson.dev/app-namespace: "true", and none of the job-shape policies match here.
#
# NO pod-security.kubernetes.io LABEL, on purpose. Rootless BuildKit needs seccomp AND AppArmor
# `Unconfined` (measured — see deployment.yaml), which PSS `baseline` forbids; there is no
# cluster-wide PSA default here, so an unlabelled namespace admits it. It is NOT privileged and
# the deployment's securityContext is the boundary — read it before relaxing anything.
apiVersion: v1
kind: Namespace
metadata:
name: buildkit
labels:
homelab.keithjackson.dev/ci: buildkit
+109
View File
@@ -0,0 +1,109 @@
---
# Default-deny for the namespace, then two narrow allows.
#
# WHO MAY CONNECT — AND WHY THE ACT-RUNNER POD IS THE RIGHT SOURCE (measured 2026-10-29).
# CI jobs do not run as pods. act_runner starts each job as a container on its DinD sidecar, on a
# docker bridge network INSIDE the act-runner pod's network namespace, and dockerd masquerades it:
# kubectl -n gitea-actions exec deploy/act-runner -c dind -- iptables -t nat -S POSTROUTING
# -A POSTROUTING -s 172.18.0.0/16 ! -o br-c0f13d041ad6 -j MASQUERADE
# -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
# So a job's connection leaves with the act-runner POD's IP and therefore Cilium's identity for
# that pod (namespace gitea-actions, app.kubernetes.io/name=act-runner). That is exactly what the
# ingress rule names. It cannot distinguish one job from another, or a job from act_runner
# itself — per-job authentication is the mTLS client certificate, not this policy. What the
# policy adds is that NOTHING ELSE in the cluster can even open the port.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: buildkit
labels:
homelab.keithjackson.dev/baseline: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress: []
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: buildkitd-allow-ci-ingress
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: buildkitd
ingress:
- fromEndpoints:
- matchLabels:
io.kubernetes.pod.namespace: gitea-actions
app.kubernetes.io/name: act-runner
toPorts:
- ports:
- { port: "1234", protocol: TCP }
---
# WHAT A BUILD NEEDS TO REACH. Build steps download (dnf mirrors, GitHub releases, dl.k8s.io,
# npm, PyPI, Fedora/Docker registries) and the result is pushed to the Gitea registry.
# * world 443/80: the downloads. 80 because Fedora's mirror list includes plain-http mirrors
# (dnf verifies package GPG signatures, not transport).
# * ingress (the Gateway's Envoy, reserved:ingress): the push. gitea.home.keithjackson.dev
# resolves in-cluster to the Gateway VIP (split-horizon CoreDNS), and Cilium hands that
# traffic to the Gateway, whose identity is `ingress` — the same identity gitea's own
# ingress allow admits.
# * gitea's BACKEND identity, :3000 only: ALSO the push, and the half that `ingress` alone does
# not cover. Cilium >= 1.16.8/1.17.2 (CVE-2025-30162; this cluster runs 1.20.1) makes the
# Gateway's Envoy re-evaluate THIS pod's egress policy against the identity of the backend it
# resolves the request to (enforce_policy_on_l7lb). `toEntities: ingress` gets the TCP
# connection to Envoy; without this rule Envoy then refuses the HTTP request itself with
# `403 Access denied`, and Gitea never sees it. Measured 2026-10-30 (images run 337 failed
# this way): Hubble on buildkitd's node logged `buildkit/buildkitd (ingress) ->
# gateway/cilium-gateway-shared:443 http-request DROPPED`, traffic_direction EGRESS, and
# Gitea's router log had no entry for it — while the same unauthenticated GET /v2/ from the
# act-runner pod (egress unrestricted) reached Gitea and got its normal 401. It is the same
# fix catalog/render.py applies for every SSO app's hop to Keycloak (commit 68690e5).
# Gitea's INGRESS needs nothing: the request arrives there as `reserved:ingress`, already
# admitted. Do not add buildkit to catalog/gitea.yaml's inClusterClients — that would open
# the ClusterIP path from build code to Gitea, which nothing here uses.
# Nothing else in the cluster: not the API server, not other pods. A build step is code from a
# pull request.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: buildkitd-allow-egress
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: buildkitd
egress:
- toEntities:
- world
toPorts:
- ports:
- { port: "443", protocol: TCP }
- { port: "80", protocol: TCP }
- toEntities:
- ingress
- toEndpoints:
- matchLabels:
io.kubernetes.pod.namespace: gitea
app.kubernetes.io/name: gitea
toPorts:
- ports:
- { port: "3000", protocol: TCP }
+19
View File
@@ -0,0 +1,19 @@
---
# The address CI dials: tcp://buildkitd.buildkit.svc.cluster.local:1234 (platform/buildkit/
# versions.env), mTLS only. ClusterIP — never exposed on the Gateway.
apiVersion: v1
kind: Service
metadata:
name: buildkitd
namespace: buildkit
labels:
app.kubernetes.io/name: buildkitd
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: buildkitd
ports:
- name: buildkit
port: 1234
targetPort: buildkit
protocol: TCP
+15
View File
@@ -0,0 +1,15 @@
# platform/buildkit/versions.env — the CI side of the builder. Sourced by
# .gitea/workflows/scripts/images.sh. NOT a Kubernetes resource (kustomization.yaml omits it).
#
# The CLIENT. CI drives the daemon with `buildx` (remote driver) rather than bare `buildctl`
# because the image pipeline needs `bake`'s `target:` contexts: on a pull request agent-runner is
# built FROM the toolbox built in the same run, which never exists in any registry.
# Pinned with the sha256 from the release's own checksums.txt
# (https://github.com/docker/buildx/releases/download/v0.38.1/checksums.txt); the pipeline
# refuses a download that does not match. Bump BOTH lines together.
BUILDX_VERSION=0.38.1
BUILDX_SHA256=e25d5cd675092bb5d1d03632fdc46cdf3fede7b514dc5f023a105e00d3ce5ad4
# The DAEMON's address and the name its certificate carries (platform/buildkit/certificates.yaml).
BUILDKIT_HOST=tcp://buildkitd.buildkit.svc.cluster.local:1234
BUILDKIT_SERVERNAME=buildkitd.buildkit.svc.cluster.local
+35 -1
View File
@@ -19,7 +19,9 @@ Docker Hub images were purged (DECISIONS-LOG **F-KY1**) → that path ImagePullB
3-replica **`longhorn`** StorageClass (DR-critical, R-IDP-05), with declarative backup hooks.
quay.io + ghcr.io are unaffected by the Bitnami purge. Operator manifests are pinned upstream release
files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate maintains the version.
files pulled as remote kustomize bases (the kubevirt-stub pattern). Renovate tracks the version via
`# renovate:` markers on those URLs (since 2026-10-30 — before that it could not see them) and never
auto-merges it; see [Upgrading Keycloak](#upgrading-keycloak-the-operator-is-the-version).
## Layout (controllers / configs split — F-MON1)
@@ -360,6 +362,38 @@ it on the BOUND flow, after go-live, in order:
A failed passkey ceremony is usually silent in the server log — the browser refuses before Keycloak
is involved — so step 2 failing with no log entry is the expected signature of a wrong RP ID.
## Upgrading Keycloak: the operator IS the version
The `Keycloak` CR has no `spec.image`, so the server image is whatever the operator's
`RELATED_IMAGE_KEYCLOAK` names — the same tag as the operator (prod, 2026-10-30: operator
`26.0.6`, `keycloak-0/1` on `quay.io/keycloak/keycloak:26.0.6`). Operator, CRDs and server are
released in lockstep from `keycloak/keycloak-k8s-resources`, and the three URLs in
`kustomization.yaml` are the one knob. Renovate raises them as ONE PR (`keycloak-operator` group,
labels `needs-review` + `blast-radius`), **never auto-merged**; majors wait for a dashboard tick.
`catalog/keycloak.yaml`'s image tag is documentation only (its render deploys nothing) and
Renovate is disabled for it — update it by hand to match after an operator bump merges.
**An operator bump requires re-checking the passkey reconciler's assumptions.**
`passkey-reconciler.yaml` (and `ci/render-keycloak-passkey-policy.py`, which generates its desired
state) is written against **Keycloak 26.0.6 Admin API behaviour read from that tag's source** —
grep it for `26.0.6`. The gate cannot catch a break: it is static and never runs the reconciler
against a server. Before merging a bump, diff the new tag against 26.0.6 for at least:
- `AuthenticationManagementResource.recurseExecutions` — does a sub-flow row still carry its name
in `displayName` (not `alias`)? The reconciler's flow matching depends on it.
- `updateExecutions` / execution-requirement updates — same request shape and id resolution?
- the WebAuthn Passwordless policy — is it still gated all-or-nothing on a non-empty
`webAuthnPolicyPasswordlessRpEntityName`, and does a realm `PUT` still REPLACE rather than merge
`requiredActions`? (The strict read-modify-write exists because of both.)
- authenticator-config alias handling for copied flows.
Then, after the merge reconciles (operator actions): watch the `Keycloak` CR reach Ready and both
pods roll; run the reconciler once **with `--audit`** (see "Passkeys" above for the one-off Job)
and confirm it reports convergence with no write; confirm the client-secret and client-URL
reconcilers' next runs succeed; log in to one SSO app and run one `kubectl --context k3s-prod`
OIDC command. Roll back by reverting the PR — but a server minor may migrate the database
schema on start, so take/confirm a fresh CNPG backup before merging.
## Exceptions that keep native auth (do not force SSO) — ADR-0014
Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).
+9
View File
@@ -21,8 +21,17 @@ namespace: keycloak
resources:
- namespace.yaml
# --- Keycloak Operator v26.0.6 (pinned upstream release manifests) ---
# The operator, its CRDs and the SERVER IMAGE move in lockstep: the Keycloak CR has no
# spec.image, so the operator's RELATED_IMAGE_KEYCLOAK (quay.io/keycloak/keycloak:<same tag>)
# is what runs. These three URLs are therefore THE Keycloak version. Each carries a Renovate
# marker (custom.regex manager, renovate.json5) under ONE depName, so all three bump in ONE
# branch, never auto-merged (packageRule "keycloak-operator"). Before merging a bump, re-check
# passkey-reconciler.yaml's 26.0.6 Admin API assumptions — see README.md "Upgrading Keycloak".
# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.0.6/kubernetes/keycloaks.k8s.keycloak.org-v1.yml
# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.0.6/kubernetes/keycloakrealmimports.k8s.keycloak.org-v1.yml
# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.0.6/kubernetes/kubernetes.yml
# --- Admin (break-glass) credential — SOPS-encrypted, referenced by name (ADR-0007, R-IDP-04) ---
- keycloak-admin-credentials.sops.yaml
+14 -7
View File
@@ -45,11 +45,18 @@ waive the sensitive-path human review on its own pull request.
Both are deliberate, and both fail **visibly** rather than silently:
1. **The image does not exist.** `platform/pm-agent/deployment.yaml` and
`PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder. Build it with
`infra/agent/build-image.sh` (which verifies it and refuses to push), push it as the
operator, then update **both** places with the digest `podman push --digestfile` returned,
in one pull request. Until then: `ImagePullBackOff`.
1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
`PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder until the first
publish. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
(`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that
`infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to
`gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
Renovate then follows the `main` tag through the `# renovate-digest:` marker above **both**
lines and moves them together in one pull request. Nobody builds-and-pushes this image from
a workstation, and nobody pastes a digest. Until the first publish, and until that PR
merges: `ImagePullBackOff`. The one operator prerequisite on the publishing side is the
`REGISTRY_PUSH_TOKEN` Actions secret (see the header of `.gitea/workflows/images.yaml`);
without it the publish job fails loudly rather than skipping.
**Two things have to be true before any digest will pull, and neither is declared in this
repository.** Both were measured after PR #57 merged — this service is the first workload
@@ -78,8 +85,8 @@ Both are deliberate, and both fail **visibly** rather than silently:
account whose token carries `read:package`, referenced from the Deployment and the
committed worker Job template — not built yet, because minting that token is yours.
The same two apply to `conformance-runner`: same registry path, same push-then-pin model,
never pulled either.
The same two apply to `conformance-runner`: same registry path, same publish-then-Renovate
model, never pulled either.
2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the
repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off —
which Gitea account to create, which scopes, and why it must *not* be an administrator —
+7 -6
View File
@@ -52,12 +52,13 @@ data:
# contract. A tag here is refused at startup (and would be DENIED by admission anyway).
#
# PLACEHOLDER UNTIL THE IMAGE IS PUBLISHED. The all-zero digest below is deliberately
# impossible: `infra/agent/build-image.sh` builds and verifies the image, the OPERATOR
# pushes it, and the authoritative digest from `--digestfile` replaces this value in a
# reviewed pull request (the same two-step the conformance-runner image follows, and for
# the same reason: a registry credential is not an agent's to hold). Until then the
# dispatcher starts, fails to pull, and says so — which is the visible failure, not a
# silent one.
# impossible. CI builds, verifies and — on main — publishes agent-runner
# (.gitea/workflows/images.yaml), and Renovate replaces this value with the digest the
# REGISTRY returned for the image's `main` tag, in a pull request that also moves the
# Deployment's image (same marker, same branch). Nobody pushes this image from a workstation.
# Until the first publish the dispatcher starts, fails to pull, and says so — which is the
# visible failure, not a silent one.
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000"
# --- the model -------------------------------------------------------------
+5 -2
View File
@@ -66,9 +66,12 @@ spec:
- name: dispatch
# THE SAME IMAGE THE WORKERS RUN, at the same digest. They share the committed Job
# template and the env-name contract baked into it, so a mismatched pair would
# disagree about the shape the admission policy enforces. Replaced together, in one
# pull request, with the digest `infra/agent/build-image.sh` printed — the all-zero
# disagree about the shape the admission policy enforces. Replaced together, in ONE
# Renovate pull request: both lines carry the same marker, so Renovate treats them as
# one dependency and bumps them in the same branch. The digest comes from CI's publish
# of agent-runner's `main` tag (.gitea/workflows/images.yaml) — the all-zero
# placeholder below is impossible on purpose (see configmap.yaml PM_WORKER_IMAGE).
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000
imagePullPolicy: IfNotPresent
command:
+240 -10
View File
@@ -22,9 +22,29 @@
// podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \
// ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json5
//
// NOTE on what was NOT wrong: `enabledManagers` containing bare `regex` was suspected of needing to
// be `custom.regex`. The validator accepts `regex`, so that suspicion was unfounded and nothing
// here changes it.
// CORRECTION (2026-10-29): THE VALIDATOR ABOVE WAS NOT THE RENOVATE THE WORKFLOW RUNS. The bare
// `ghcr.io/renovatebot/renovate` tag is `latest` (44.167.0 when this was measured), while
// renovatebot/[email protected] defaults `renovate-version` to `39` (its action.yml), i.e.
// 39.264.0. Validated against THAT, this file failed with five errors — every
// `managerFilePatterns` key ("Custom Manager contains disallowed fields", "Invalid configuration
// option: flux.managerFilePatterns", ...), because 39.x only knows `fileMatch`; the rename landed
// in 40.2.0. So even after the workflow found this file, every run would have stopped at
// CONFIG_VALIDATION. renovate.yaml now pins the exact release this file is validated against —
// since 2026-10-30 as the job's `container.image` tag (the action is gone: it needed a Docker
// daemon jobs here do not get). VALIDATE WITH THAT SAME TAG:
// podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \
// ghcr.io/renovatebot/renovate:<container.image tag in renovate.yaml> \
// renovate-config-validator /tmp/renovate.json5
// (Not `--strict`: that also fails on a pending migration of the pre-existing top-level
// `"docker": {"enabled": true}` block at the bottom, which is a warning, not an error.)
//
// `regex` vs `custom.regex` in enabledManagers, RESOLVED FROM SOURCE (Renovate 39.264.0 and
// later): both work. lib/config/migrations/custom/enabled-managers-migration.ts rewrites
// `regex` -> `custom.regex`, and that migration runs on every real repository run
// (lib/workers/repository/init/merge.ts -> migrateAndValidate), not only in the validator;
// lib/modules/manager/index.ts getEnabledManagersList accepts either spelling. `custom.regex` is
// the canonical name (introduced in 37.51.1, renovatebot/renovate#24079), so it is used below and
// nothing depends on the migration any more.
{ "$schema": "https://docs.renovatebot.com/renovate-schema.json",
// Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17
// RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags.
@@ -51,7 +71,7 @@
"labels": ["dependencies", "automated-update"],
"ignoreUnstable": true,
"respectLatest": true,
"enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
"enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "custom.regex", "github-actions"],
"flux": {
"managerFilePatterns": [
"/(^|/)platform/.+\\.ya?ml$/",
@@ -93,6 +113,86 @@
"#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n.*?(?<currentValue>v?[0-9]+\\.[0-9]+\\.[0-9]+)"
],
"versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
},
// ADDED 2026-10-29 — THIS REPO'S OWN IMAGES (toolbox, agent-runner, conformance-runner).
// CI publishes them (.gitea/workflows/images.yaml) under an immutable `sha-<commit>` tag and a
// moving `main` tag, and never writes to git. Every reference to them is DIGEST-ONLY
// (`<repo>@sha256:<hex>`, no tag), because the admission policies
// (platform/kyverno/admission/pm-agent-job-shape.yaml, conformance-trigger-job-shape.yaml)
// admit exactly that form. A digest alone gives Renovate nothing to follow, so the tag to
// follow rides in a marker comment on the line ABOVE the reference:
//
// # renovate-digest: image=<registry>/<namespace>/<image> tag=main
// <anything>sha256:<64 hex>
//
// `main` is not a version, so Renovate skips the release lookup and proposes a `digest`
// update whenever the tag's digest moves (lib/workers/repository/process/lookup/index.ts:
// an unversioned currentValue WITH a currentDigest yields updateType `digest`). The
// `sha256:` prefix is INSIDE currentDigest on purpose: Renovate replaces that capture
// verbatim with the new digest, which carries the prefix.
//
// DELIBERATELY NOT TRACKED: infra/agent/worker-job-template.yaml. It is baked INTO
// agent-runner, so bumping it would rebuild agent-runner and loop forever; pm-dispatch
// overwrites that field from PM_WORKER_IMAGE at runtime anyway.
// REGISTRY AUTH: the Gitea registry answers 401 to anonymous reads, and the platform token is
// scoped to hostType `gitea`, which a docker lookup never matches
// (lib/util/host-rules.ts find). renovate.yaml supplies a hostType `docker` rule.
{
"customType": "regex",
"description": "This repo's own CI-published images, pinned by digest, following the tag named in a `# renovate-digest:` marker on the previous line.",
"managerFilePatterns": [
"/(^|/)versions\\.env$/",
"/(^|/)platform/pm-agent/.+\\.ya?ml$/"
],
"matchStrings": [
"#\\s*renovate-digest:\\s*image=(?<depName>\\S+)\\s+tag=(?<currentValue>\\S+)\\s*\\n[^\\n]*?(?<currentDigest>sha256:[a-f0-9]{64})"
],
"datasourceTemplate": "docker",
"versioningTemplate": "docker"
},
// ADDED 2026-10-30 — THE CATALOG IS THE SOURCE; apps/wave-*/*.yaml IS GENERATED FROM IT.
// First real dry run (dispatch runs 353/354): EVERY app-image update edited only the rendered
// apps/wave-*/*.yaml and NONE edited catalog/. The `kubernetes` manager lists catalog/ above
// but extracts nothing there: a ServiceDefinition has no `apiVersion`, so it is not a
// Kubernetes manifest to that manager (measured: 0 deps from catalog/). Every such PR would
// fail ci/render-golden.sh (the canonical render reverts the bump), sit red and hold a
// prConcurrentLimit slot. These two managers read the SOURCES, under the SAME depName as the
// rendered line, so both land in ONE branch and render==golden holds: `source.image` in each
// ServiceDefinition, and render.py's OAUTH2_PROXY_IMAGE (the six auth sidecars).
{
"customType": "regex",
"description": "ServiceDefinition `source.image: <repo>:<tag>` (catalog/*.yaml) — the source of the rendered apps/wave-* image.",
"managerFilePatterns": ["/(^|/)catalog/[^/]+\\.ya?ml$/"],
"matchStrings": [
"\\n\\s+image:\\s*(?<depName>[^\\s:@#\"']+):(?<currentValue>[^\\s@#\"']+)"
],
"datasourceTemplate": "docker"
},
{
"customType": "regex",
"description": "catalog/render.py OAUTH2_PROXY_IMAGE — the per-app oauth2-proxy sidecar every forward-auth app is rendered with.",
"managerFilePatterns": ["/(^|/)catalog/render\\.py$/"],
"matchStrings": [
"\\nOAUTH2_PROXY_IMAGE\\s*=\\s*\"(?<depName>[^\":@]+):(?<currentValue>[^\"@]+)\""
],
"datasourceTemplate": "docker"
},
// ADDED 2026-10-30 — CNPG `Cluster.spec.imageName`. The `kubernetes` manager reads `image:` only,
// so the three database Clusters (platform/keycloak/postgres-cluster.yaml, apps/wave-0/
// nextcloud-db.yaml, apps/wave-0/semaphore-db.yaml) were untracked, while the grant Job in
// nextcloud-db.yaml — whose comment says "keep the tag in step with spec.imageName" — WAS. A real
// run would have bumped the Job and left the database behind. Same depName, so one branch.
{
"customType": "regex",
"description": "CloudNativePG Cluster `imageName: <repo>:<tag>`.",
"managerFilePatterns": [
"/(^|/)platform/.+\\.ya?ml$/",
"/(^|/)apps/.+\\.ya?ml$/"
],
"matchStrings": [
"\\n\\s+imageName:\\s*(?<depName>[^\\s:@#\"']+):(?<currentValue>[^\\s@#\"']+)"
],
"datasourceTemplate": "docker"
}
],
"packageRules": [
@@ -144,13 +244,129 @@
"longhorn",
"cilium",
"cloudnative-pg",
"kube-prometheus-stack"
"kube-prometheus-stack",
// ADDED 2026-10-30. gitea and act_runner ARE the merge gate: a bad auto-merged bump takes
// down the very check that would have to approve its revert (the dry run proposed gitea
// 1.24.3 -> 1.27.3 and act_runner 0.2.11 -> 0.6.1, both as auto-mergeable "minor"; a 0.x
// minor is a breaking line by semver). The postgresql image restarts every CNPG database,
// Keycloak's (the IdP) included, now that imageName is tracked.
"gitea/gitea",
"gitea/act_runner",
"ghcr.io/cloudnative-pg/postgresql"
],
"automerge": false,
"addLabels": [
"needs-review",
"blast-radius"
]
},
// ADDED 2026-10-30 — THE KEYCLOAK OPERATOR IS THE KEYCLOAK VERSION. The operator, its CRDs
// and the server image are released in lockstep from keycloak/keycloak-k8s-resources, and the
// Keycloak CR carries no spec.image, so the operator's RELATED_IMAGE_KEYCLOAK picks the server
// (measured on prod 2026-10-30: operator 26.0.6 -> keycloak-0/1 run quay.io/keycloak/keycloak:
// 26.0.6). The three release URLs in platform/keycloak/kustomization.yaml carry `# renovate:`
// markers under this ONE depName, so they move together in one branch (groupName makes that
// explicit). NEVER AUTO-MERGED, not even a patch: Keycloak is the IdP that gates SSO for every
// app AND kubectl OIDC, and platform/keycloak/passkey-reconciler.yaml encodes Keycloak 26.0.6
// Admin API shapes read from that tag's source (sub-flow rows named by displayName, the
// RpEntityName all-or-nothing guard, requiredActions REPLACE-not-merge, ...). The gate is
// static and does not run that reconciler against a server, so a green check says nothing
// about whether a bump breaks it. Majors still wait on the dashboard (rule at the bottom).
{
"description": "Keycloak operator + CRDs + server (lockstep, one branch). IdP blast radius; passkey-reconciler.yaml encodes 26.0.6 Admin API shapes — re-check it against the new tag before merging. Never auto-merge.",
"matchDepNames": ["keycloak/keycloak-k8s-resources"],
"groupName": "keycloak-operator",
"automerge": false,
"addLabels": ["needs-review", "blast-radius"]
},
// ADDED 2026-10-30 — quay.io/keycloak/keycloak IS NOT A DEPENDENCY RENOVATE MAY MOVE. Its only
// occurrences are catalog/keycloak.yaml `source.image` and the apps/wave-1/keycloak.yaml it
// renders, which is NOT listed in apps/wave-1/kustomization.yaml and deploys nothing (the
// operator above runs the server). A PR bumping it would change no running pod while looking
// like a Keycloak upgrade, and could drift from the operator's real version.
{
"description": "quay.io/keycloak/keycloak appears only in the documentation-only catalog/keycloak.yaml and its undeployed render; the operator (keycloak/keycloak-k8s-resources) chooses the real server image.",
"matchPackageNames": ["quay.io/keycloak/keycloak"],
"enabled": false
},
{
"description": "ADDED 2026-10-29. This repo's own images are followed ONLY through the `# renovate-digest:` markers above. The `kubernetes` manager also sees platform/pm-agent/deployment.yaml and infra/agent/worker-job-template.yaml, where the image has a digest and no tag; left on, it would look those up as `latest` (a tag CI never pushes) and, for the template, re-open the self-referential loop the custom manager deliberately avoids.",
"matchManagers": ["kubernetes"],
"matchPackageNames": [
"gitea.home.keithjackson.dev/home-network/toolbox",
"gitea.home.keithjackson.dev/home-network/agent-runner",
"gitea.home.keithjackson.dev/home-network/conformance-runner"
],
"enabled": false
},
{
"description": "ADDED 2026-10-29. Digest bumps for this repo's own images: the image was built and verified by CI before it was published, so these auto-merge on a green gate like any digest (rule 1). Labelled so they are easy to find. NOTE: the conformance-runner bump edits conformance/versions.env, a sensitive path (infra/gitea/approval-policy.json), so it will wait for an administrator's /approve-override regardless of this rule.",
"matchManagers": ["custom.regex"],
"matchPackageNames": [
"gitea.home.keithjackson.dev/home-network/toolbox",
"gitea.home.keithjackson.dev/home-network/agent-runner",
"gitea.home.keithjackson.dev/home-network/conformance-runner"
],
"addLabels": ["own-image"]
},
{
"description": "ADDED 2026-10-30. The `flux` manager also extracts `image:` lines inside HelmRelease values, which the `kubernetes` manager already owns. Measured: exactly two (alpine in platform/oauth2-proxy/helmrelease.yaml, velero-plugin-for-aws in platform/velero/helmrelease.yaml), each extracted twice; the second rewrite of the same line logged `Cannot find replaceString in current file content` in runs 353/354. Flux keeps its charts (datasource helm).",
"matchManagers": ["flux"],
"matchDatasources": ["docker"],
"enabled": false
},
{
"description": "ADDED 2026-10-30. MIRRORS catalog/<svc>.yaml `update.range` — the operator's declared per-service policy (schemas/servicedefinition.schema.json), which Renovate does not read. Without it the dry run proposed gitea 28.2.0, qbittorrent 20.04.1 (both stale/odd tags, not real successors), radarr 6.x and nextcloud 35 (four majors past 31; Nextcloud cannot skip majors). KEEP IN STEP with the catalog ranges.",
"matchPackageNames": ["gitea/gitea"],
"allowedVersions": "<2.0.0"
},
{ "matchPackageNames": ["nextcloud"], "allowedVersions": "<32.0.0" },
{ "matchPackageNames": ["lscr.io/linuxserver/qbittorrent"], "allowedVersions": "<6.0.0" },
{ "matchPackageNames": ["lscr.io/linuxserver/radarr"], "allowedVersions": "<6.0.0" },
{ "matchPackageNames": ["lscr.io/linuxserver/sonarr"], "allowedVersions": "<5.0.0" },
{ "matchPackageNames": ["semaphoreui/semaphore"], "allowedVersions": "<3.0.0" },
{
"description": "ADDED 2026-10-30. A CNPG Postgres MAJOR is a migration (new cluster + import), not an image bump; the dry run proposed 16.4 -> 18.6. Minors stay (and are blast-radius, above).",
"matchPackageNames": ["ghcr.io/cloudnative-pg/postgresql"],
"allowedVersions": "<17.0.0"
},
{
"description": "ADDED 2026-10-30. A HelmRelease semver RANGE is a policy ceiling, and Renovate does not respect it — it WIDENS it (dry run: external-dns `<1.22.0` -> `<1.24.0`, headlamp `<0.42.0` -> `<0.47.0`), and the default rule would then auto-merge that as a 'minor', silently letting Flux adopt every new minor. Raising a ceiling is deliberate per-component work: propose it, never auto-merge it.",
"matchManagers": ["flux"],
"matchCurrentValue": "/^[<>=~^]|\\.x$/",
"automerge": false,
"addLabels": ["needs-review", "range-widen"]
},
{
"description": "ADDED 2026-10-30. velero-plugin-for-aws is pinned to a RELEASE CANDIDATE (v1.14.3-rc.2) and Renovate proposed NOTHING for it, although v1.14.3 and v1.14.4 are stable on Docker Hub. Docker versioning treats everything after the first `-` as a compatibility SUFFIX, so only other `*-rc.2` tags qualify and the RC is a fossil forever. Semver versioning reads it as a prerelease, and a prerelease upgrades to the stable release.",
"matchPackageNames": ["velero/velero-plugin-for-aws"],
"versioning": "semver"
},
{
"description": "ADDED 2026-10-30. A python MINOR is a language-runtime release (the dry run proposed 3.13 -> 3.15 for the Keycloak reconcilers), and the gate does not execute those scripts. Review it; patches still auto-merge.",
"matchPackageNames": ["python"],
"matchUpdateTypes": ["minor"],
"automerge": false,
"addLabels": ["needs-review"]
},
// OPERATOR DECISION 2026-10-30: these two never auto-merge, at ANY update type. Both sit after
// the auto-merge rule (rule 1) so they override it; `needs-review` is added, not replaced.
{
"description": "ADDED 2026-10-30 (operator decision). The oauth2-proxy image, under the ONE depName every source of it emits: catalog/render.py OAUTH2_PROXY_IMAGE (custom regex manager), the rendered apps/wave-* sidecars and platform/crafty/deployment.yaml (kubernetes manager). WHY: since v7.13.0 oauth2-proxy validates REFRESHED OIDC sessions against the ACCESS token instead of the id_token (upstream CHANGELOG, v7.13.0), every sidecar sets --cookie-refresh, and that refresh path has never run against this Keycloak (the platform oauth2-proxy runs v7.15.5 WITHOUT --cookie-refresh). render-golden and render.py --selftest pass on the bump, but both check rendered text, not the binary, so a green gate is not evidence here.",
"matchPackageNames": ["quay.io/oauth2-proxy/oauth2-proxy"],
"automerge": false,
"addLabels": ["needs-review"]
},
{
"description": "ADDED 2026-10-30 (operator decision). Renovate's OWN version: the job `container.image` pin in .gitea/workflows/renovate.yaml (ghcr.io/renovatebot/renovate), and renovatebot/github-action should it ever be reintroduced (ci/lint.sh rejects it today). WHY: nothing re-validates renovate.json5 against the NEW version before it takes effect. The gate does not run renovate-config-validator, and this exact gap has bitten before: the action's default 39.x rejected this file (managerFilePatterns needs >= 40.2.0) and every run stopped at CONFIG_VALIDATION. Run the validator from the new tag, then merge.",
"matchPackageNames": ["ghcr.io/renovatebot/renovate", "renovatebot/github-action"],
"automerge": false,
"addLabels": ["needs-review"]
},
{
"description": "ADDED 2026-10-30. MAJORS WAIT FOR A TICK ON THE DEPENDENCY DASHBOARD instead of opening a PR. They are never auto-merged anyway, and with prConcurrentLimit 5 the dry run's 10 majors would hold every PR slot open indefinitely, starving the patch/minor/digest PRs that can merge on green.",
"matchUpdateTypes": ["major"],
"dependencyDashboardApproval": true
}
],
// WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A
@@ -159,19 +375,33 @@
// lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak
// (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete
// version before any automation can maintain them, and until then they drift on every pull with
// nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12,
// e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it
// has been told not to leave; raising those ceilings is deliberate per-component work and is
// NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.
// nothing recording what is running. CORRECTED 2026-10-30: nextcloud:31-apache and docker:27-dind
// ARE maintained (the first dry run proposed 35-apache and 29-dind; the suffix is kept).
// CORRECTED 2026-10-30: quay.io/keycloak/keycloak was never a running reference — the operator
// picks the server image — and the Keycloak version is now tracked via the operator URLs.
// Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on
// kube-prometheus-stack): THIS FILE USED TO SAY they cap Renovate at patches. They do not —
// Renovate WIDENS the range (runs 353/354). The `range-widen` rule above keeps that a reviewed
// PR rather than an auto-merge. Both lists are in decisions/2026-10-17-automated-upgrades.md.
// platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest`
// and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed
// shape, not software to keep current. `conformance/**` was already ignored for the same
// reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise
// have opened.
// conformance/** NARROWED 2026-10-29 to its subdirectories: ignorePaths applies to custom
// managers too (lib/workers/repository/extract/file-match.ts getMatchingFiles ->
// filterIgnoredFiles), so the blanket glob also hid conformance/versions.env, which now carries
// the conformance-runner digest Renovate must maintain. Everything else under conformance/
// stays ignored, for the original reason (test fixtures with deliberately fixed shapes).
"ignorePaths": [
"**/*.example",
"**/*.sops.yaml",
"conformance/**",
"conformance/config/**",
"conformance/incluster/**",
"conformance/lib/**",
"conformance/requirements/**",
"conformance/runs/**",
"conformance/tests/**",
"platform/image-automation/per-service/**",
"platform/kyverno/tests/**"
],
+2 -2
View File
@@ -67,10 +67,10 @@ Work through these in order; the first three are by far the most likely.
| Symptom in the log | Cause | Fix |
|---|---|---|
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first |
| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been published; the committed digest is the deliberate placeholder | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | restore the digest form; the digest comes from CI's publish (`.gitea/workflows/images.yaml` job summary) and Renovate's bump PR, never from a hand push |
| `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
| `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |
| `at the worker cap` / `worker limit reached` | the per-cycle cap, draining a backlog one issue at a time | expected; it will continue next cycle |