home-network-k3s-impl/decisions/2026-10-29-image-pipeline.md

---
date: 2026-10-29
branch: feat/image-pipeline
title: "Images are built, verified and published by CI on rootless BuildKit; Renovate carries the digests into git"
---

## What changed

Operator rule (2026-10-29): container image creation, hosting and lifecycle belong to CI/CD and
the codebase, never to a one-off workstation build/push. Before this branch the Home-Network org's
package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and
both `build-image.sh` scripts said "publishing is an operator action".

### The builder: rootless BuildKit — DECISION MADE (operator, on PR #81)

**The problem it answers, measured on this PR's first CI run (images run 325, failed in 6 s):** a
CI job on this runner gets **no Docker daemon at all**, for two reasons, both in the deployed
versions' source:
- act_runner v0.2.11 `internal/app/cmd/daemon.go` sets `container.docker_host` to `-` ("never
  mount into jobs") whenever `DOCKER_HOST` is `tcp://`, which it is here (the DinD sidecar);
- gitea/act v0.261.2 `pkg/container/docker_run.go` `sanitizeConfig` drops **every** bind,
  socket included, when `valid_volumes` is empty, which it is (the runner has no config file).

Giving jobs the DinD daemon would have made every branch push root on the node, because that
daemon is `privileged: true`.

**The decision:** no job may reach a privileged daemon, and images are built by a **rootless
BuildKit** daemon over mTLS. The runner (`apps/wave-1/gitea-actions.yaml`, DOCKER_HOST,
valid_volumes) is **untouched**.

- **`platform/buildkit/`** — a new hand-managed component in its own namespace `buildkit`. The
  runner manifest is renderer-generated (`catalog/render.py` EXTRA_UNITS), and the builder has no
  business sitting next to a privileged container. Flux reconciles it as the new Kustomization
  `buildkit` (`clusters/base/buildkit.yaml`, `dependsOn: platform`).
  - **Deployment:** `moby/buildkit:v0.34.1-rootless@sha256:f68752a5…0c4f89`, pinned by index
    digest. The securityContext, every line measured (see *What I verified*):
    - `privileged: false`;
    - uid/gid 1000, `runAsNonRoot`;
    - `capabilities: drop ALL, add SETUID, SETGID`;
    - `allowPrivilegeEscalation: true`;
    - `seccompProfile: Unconfined` and `appArmorProfile: Unconfined`;
    - `readOnlyRootFilesystem: true`, with five emptyDirs.

    Other settings:
    - `--oci-worker-no-process-sandbox`;
    - no ServiceAccount token;
    - memory limit 4Gi, which covers build steps too;
    - cache emptyDir with `sizeLimit: 6Gi`, and GC capped at 4 GB in `buildkitd.toml`
      (ConfigMap).
  - **mTLS via cert-manager** (`certificates.yaml`): a self-signed CA used only here, the server
    cert `buildkitd-tls`, and the client cert `buildkit-client-ci`. buildkitd verifies every
    client against that CA.
  - **Service:** ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`.
  - **NetworkPolicies:**
    - default-deny;
    - ingress on 1234 **only from the act-runner pod** (`gitea-actions`,
      `app.kubernetes.io/name=act-runner`);
    - egress: DNS, world :443/:80 for downloads, and the `ingress` entity (the Gateway) for the
      registry push.
  - `versions.env`: the CI-side client, `buildx` v0.38.1 with the sha256 from its
    `checksums.txt`, plus the address and certificate name CI dials.
- **How DinD job containers appear to Cilium (measured, read-only).**
  `iptables -t nat -S POSTROUTING` in the dind container shows
  `-A POSTROUTING -s 172.18.0.0/16 ! -o br-… -j MASQUERADE` (and the same for docker0). So a job's
  connection leaves with the **act-runner pod's IP, and therefore that pod's Cilium identity**.
  That makes "only the act-runner pod" expressible and correct, but it cannot tell one job from
  another, or a job from act_runner itself. Per-job authentication is the client certificate.
- **Admission: no Kyverno exception needed.**
  - Every ClusterPolicy actually applied (`platform-config/kustomization.yaml`; live:
    `conformance-trigger-job-shape`, `label-cdi-worker-pods`, `pm-agent-job-shape`,
    `require-auth-mode-label`, `require-default-deny-baseline`, `require-egress-class-label`,
    `restrict-host-namespaces`, `secret-provenance`, `validate-default-deny-shape`) matches only
    namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`, or only its own
    namespace in the job-shape policies' case. `buildkit` carries no such label.
  - There are no namespaced Kyverno Policies and no cluster-wide PSA default (both measured).
  - The namespace is deliberately left without a PSA label, because PSS `baseline` forbids
    `Unconfined` seccomp and AppArmor.
  - `validate-default-deny-shape` does apply to the namespace's default-deny NetworkPolicy, and
    its shape (both directions) satisfies it.
- **Unprivileged user namespaces on the nodes (measured, read-only, via Ansible `sysctl`).** On
  all three Debian 13.7 nodes, kernel `7.1.8+deb13-amd64`:
  - `kernel.unprivileged_userns_clone = 1`;
  - `user.max_user_namespaces = 127206`;
  - no `kernel.apparmor_restrict_unprivileged_userns` (an Ubuntu-only key);
  - AppArmor enabled (`Y`).

  So this is a satisfied prerequisite, not an open one.
- **The TLS conformance check needed a narrow, self-proving exception** (`conformance/**`, so this
  changes what green means; flagged for the reviewer).
  - The first gate run failed L0 `tls-lifecycle-automated` clause (5): "every Certificate's
    issuer must be ACME". That is broader than the spec it serves:
    - R-TLS-01 and docs/08 concern *externally reachable* services;
    - ADR-0006 says "mTLS between workloads is a separate concern … not covered here";
    - a public CA cannot issue for a `*.svc.cluster.local` name or a client-auth certificate.
  - The new clause (5b) exempts a Certificate annotated
    `homelab.keithjackson.dev/tls-scope: workload-mtls` **only if it proves it**:
    - no Gateway listener serves its Secret;
    - every DNS name and the commonName are a bare label or `*.svc[.cluster.local]`;
    - it has no IP, URI or email SANs;
    - its issuer is a namespaced `selfSigned`/`ca` Issuer in its own namespace;
    - a `ca` issuer's Secret is produced by a likewise-exempt tracked Certificate.
  - Exemptions are printed, and clause (9), the renewal arithmetic, still applies.
  - **Negative-tested:** a public DNS name, and a Gateway serving `buildkitd-tls`, each fail the
    check with a specific message.
- **`infra/gitea/buildkit-client-secret.sh`** (operator action): copies `ca.crt`, `tls.crt` and
  `tls.key` from the Secret `buildkit/buildkit-client-ci` into the Actions secrets `BUILDKIT_CA`,
  `BUILDKIT_CLIENT_CERT` and `BUILDKIT_CLIENT_KEY`. Before writing anything it checks that:
  - the cert verifies against the CA;
  - the cert is a client-auth cert;
  - the key matches the cert.

  It never prints a value. `--check` reports whether the stored copy is older than the current
  certificate.

### Verification moved into the Containerfiles

`docker run` cannot exist on BuildKit, so each Containerfile now names its shipped stage after the
image (`toolbox`, `agent-runner`, `conformance-runner`) and ends in a **`verify` test stage**,
`FROM <image> AS verify`. That stage runs every previous assertion at build time, as the image's
non-root uid (65532 for agent-runner and conformance-runner), in `RUN --network=none`. It is never
pushed.
- **Every existing assertion is preserved**, and the toolbox gained one it never had: every tool
  on PATH, and five tools report their pinned versions.
- `probe-validation.sh` is `COPY`ed into the verify stage only, so it never ships.
- **Bug fixed on the way:** in the old `docker run` block, `cmd | grep -q … && echo ok` and
  `[ … ] && echo` could never fail the script under `set -e` + `pipefail`. The "entrypoints refuse
  arguments", "not root", "no API key baked in" and "HOME writable" checks asserted nothing. They
  are real assertions now.
- **What a build step cannot reproduce, and how it is approximated** (also written in each stage's
  header):
  - **uid:** exact.
  - **`--network none`:** exact, and each stage *proves* it. Its first check fails if DNS or a
    TCP connection works. A negative control without `--network=none` fails that check.
  - **`--read-only` rootfs:** not reproducible, because a RUN step always has a writable layer.
    It is approximated by permission:
    - `touch /should-fail` must fail;
    - a `find / -xdev -writable ! -type l` inventory, run as the uid, must lie entirely under
      `/workspace`, `/tmp` or `/var/tmp`. `! -type l` was needed: Fedora's `/usr/tmp` symlink
      to `/var/tmp` tripped it.

    The real control remains the pods' `readOnlyRootFilesystem`, which the admission policies
    enforce.
  - **`--tmpfs /workspace`:** approximated by the image's own `/workspace`, owned by 65532, which
    is exactly what the pod's emptyDir replaces.

  The opt-in `--verify-clone` still `run`s the image, so it remains local-only.
- **`build-image.sh` (all three):**
  - verify is `build --target verify` and the image is `build --target <image>`;
  - a new `--print-build-args` prints the pins as `KEY=VALUE`, which is what CI consumes. The
    arg list therefore exists once;
  - the local podman path still works.
- The L0 check `alert-rules-unit-tested` now looks for `"PROMTOOL_VERSION=$PROMTOOL_VERSION"` in
  the toolbox's build-arg list.

### The pipeline (`.gitea/workflows/images.yaml` + `scripts/images.sh`)

- **The builder.** The driver downloads the pinned `buildx` and refuses a checksum mismatch. It
  writes the client cert from the `BUILDKIT_*` secrets to a 0600 temp dir that is removed on exit,
  creates a `remote` builder, and runs `buildx inspect --bootstrap`.
- **It fails loudly, never skips.** It stops with a clear message if:
  - the secrets are absent;
  - buildkitd is unreachable. The message lists the causes in order: not deployed yet (Flux
    applies it only after merge), stale certificate, NetworkPolicy.
- **Pull request:** `buildx bake <image>-verify` for changed images. Nothing is pushed and no
  registry credential is referenced. The `BUILDKIT_*` builder secrets *are* given to PR jobs,
  because they must build.
  - agent-runner is built FROM the toolbox built **in the same run** through a bake `target:`
    context. That toolbox exists in no registry, so the toolbox is built as its base even when
    its own inputs did not change.
- **Push to main / dispatch on main:** for each image, the verify target first, and only then the
  image itself with `push=true`. Tags are `sha-<12 hex>` (immutable) and `main` (moving).
  - The digest is read back from the registry and compared with the client's metadata.
  - agent-runner builds FROM the toolbox digest just pushed, or else from the pinned
    `AGENT_BASE_IMAGE` (the zero placeholder bootstraps).
  - `BUILDX_NO_DEFAULT_ATTESTATIONS=1` keeps one manifest per tag, which is what admission,
    Renovate and the retention script assume.
- **Unchanged from the first iteration:** change detection, publish idempotence,
  `REGISTRY_PUSH_TOKEN` and its preflight, the Renovate markers, and `registry-retention.sh`. The
  DinD prune step is gone, because nothing is built in DinD now.

### Unchanged from the first iteration (still true)

- **Digest references** carry `# renovate-digest: image=<repo> tag=main` markers:
  - `platform/pm-agent/deployment.yaml` and `configmap.yaml`;
  - `infra/agent/versions.env` `AGENT_BASE_IMAGE`, now the published toolbox by digest, zero
    until the first publish;
  - `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to zero.

  The admission policies are untouched.
- **`renovate.json5`**:
  - the `custom.regex` marker manager;
  - `enabledManagers` `regex` → `custom.regex`;
  - the `kubernetes` manager disabled for these three repos;
  - `ignorePaths` narrowed so `conformance/versions.env` is visible.

  `renovate.yaml` pins `renovate-version: 44.167.0` and adds a docker `RENOVATE_HOST_RULES` entry
  using `RENOVATE_TOKEN`, which now needs `read:package`.
- **`infra/gitea/registry-retention.sh`** (dry run by default; operator action); the
  `conformance-trigger-kubeconfig.sh` precondition fix; and the docs (pm-agent README, runbook,
  plan, conformance README, devcontainer README, workflows README). The docs are updated again
  for BuildKit, and there is a new `platform/buildkit/README.md`.

### Decisions and pushback

- **The push credential is a PAT secret, `REGISTRY_PUSH_TOKEN`; the Actions token cannot push.**
  Checked against Gitea **v1.24.3** source:
  - `services/auth/basic.go` `Basic.Verify` turns a task token into the pseudo-user
    `gitea-actions` (ID −2) and sets no token scope;
  - `routers/api/packages/container/container.go` `Authenticate` (the `/v2/token` exchange every
    docker client performs) requires `read:package`/`write:package`/`all` scope, so it answers
    401. The Actions token cannot even *pull*;
  - `services/context/package.go` `determineAccessMode` has `// TODO: ActionUser permission check`
    and no branch for that user. Even past login it would get at most read on a public org;
  - `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3.

  The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight
  pattern. With BuildKit the credential reaches the daemon through the buildx session (a
  0600 `config.json` in the job's temp dir), never through the cluster.
- **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate:
  - `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` →
    `custom.regex`;
  - that migration runs on every repository run (`lib/workers/repository/init/merge.ts` →
    `migrateAndValidate`), not only in the validator;
  - `lib/modules/manager/index.ts` `getEnabledManagersList` accepts either spelling;
  - `custom.regex` exists since 37.51.1 (renovatebot/renovate#24079).
- **The open question hid a bigger bug: the Renovate config on main is invalid for the Renovate
  the workflow actually runs.**
  - `renovatebot/[email protected]` defaults `renovate-version` to `39`
    (`action.yml`, `src/docker.ts`), which is 39.264.0.
  - The validator used in #79 was `ghcr.io/renovatebot/renovate` with no tag, i.e. `latest`,
    44.167.0.
  - **Measured:** `renovate:39.264.0 renovate-config-validator` on main's `renovate.json5` gives
    `ERROR: Found errors in configuration` with five errors, all `managerFilePatterns`.
    `fileMatch` was renamed in 40.2.0.
  - So once #80 fixes the filename, every run would still die at `CONFIG_VALIDATION`.
  - Fixed by pinning `renovate-version: 44.167.0`. This branch's config validates clean on it.
    The only `--strict` complaint is a pending migration of the pre-existing top-level
    `"docker": {"enabled": true}` block, which I left alone.
- **Unworkable as briefed: tracking the worker Job template's digest.**
  - `infra/agent/worker-job-template.yaml` is `COPY`ed into agent-runner. A Renovate bump there
    changes agent-runner's inputs, CI publishes a new digest, Renovate bumps it again, and so on
    forever.
  - `pm-dispatch` overwrites that field from `PM_WORKER_IMAGE` at runtime
    (`pm-dispatch.py:649`), so nothing is lost by leaving it all-zero. It is commented as
    deliberately untracked.
  - For the same loop reason, `conformance/versions.env` is **not** an input of the
    conformance-runner build, and a pipeline-only change does not republish on main.
- **Changed from the brief: retention is a script, not a Gitea cleanup rule.**
  - Gitea 1.24.3 has **no API** for cleanup rules (none in `routers/api/v1/api.go`; web
    session+CSRF forms only, in `routers/web/web.go`). So a rule cannot be created, or
    drift-asserted, as code. **`ci/gitea-settings.sh` was therefore not extended.** A cheaper
    proxy (count versions per image) would need `read:package` on `SETTINGS_API_TOKEN`, which
    it lacks.
  - A built-in rule is also **unsafe here**. Gitea stores a manifest as a file of the TAG
    version it was pushed under (`routers/api/packages/container/manifest.go`
    `createPackageAndVersion`), and resolves a by-digest pull from those files
    (`models/packages/container/search.go`). Deleting the tag a deployed digest was pushed under
    breaks the `@sha256:` pull, and the rule's "newest N" knows nothing about what git pins. The
    script keeps every digest git pins.
- **Auto-merge.** Digest bumps of these images auto-merge on a green gate (the existing
  `digest` rule). The exception is the conformance-runner bump: `conformance/versions.env` is a
  sensitive path, so it waits for an administrator's `/approve-override` every time.

## What I verified

**Proved locally, against the exact pinned images:**

- **Rootless buildkitd, configured as the manifest configures it.**
  - Setup: the pinned `moby/buildkit:v0.34.1-rootless` digest, under rootless podman, with
    `--cap-drop ALL --cap-add SETUID --cap-add SETGID`, seccomp unconfined and `--read-only`;
    tmpfs at the same five paths; the ConfigMap's `buildkitd.toml`; mTLS on :1234.
  - The image's `/etc/subuid` was overridden with `user:1:999` and `user:1001:64535`, because
    rootless podman maps only 65,537 uids. rootlesskit's real `newuidmap` user-namespace path
    still ran.
  - It started, `buildctl debug workers` listed the OCI worker, and `debug workers -v` showed the
    GC config applied (max 4.29 GB, reserved 1.07 GB).
- **The securityContext, item by item.** Each requirement was measured by taking it away:
  - **default seccomp:** daemon starts, every RUN step fails → `Unconfined` is required;
  - **no `SETUID`/`SETGID`:** rootlesskit dies with
    `newuidmap … operation not permitted` → both capabilities are required;
  - **`--read-only` rootfs:** works once the five emptyDirs exist. Its only failure was the
    probe client's `~/.docker`, which is now an emptyDir;
  - **minimal caps + Unconfined seccomp:** works, and a RUN step executes.
- **The full CI path, in the actual CI job image.** Setup: `ghcr.io/catthehacker/ubuntu:act-22.04`
  running `.gitea/workflows/scripts/images.sh` in verify mode. It downloaded and sha256-checked
  buildx v0.38.1, connected over mTLS, and bake-built all three `verify` targets from an empty
  cache. rc 0, 1m32s total:
  - `TOOLBOX VERIFIED` (42 s);
  - `AGENT-RUNNER VERIFIED` (26 s): selftest OK, probe "20 refusal(s), 0 failure(s)", and built
    FROM the in-run toolbox through the `target:` context, with the log showing
    `[agent-runner-verify toolbox 1/6]` and `base=toolbox-in-this-run`;
  - `CONFORMANCE-RUNNER VERIFIED` (21 s): all seven refusals;
  - every stage printed `confirmed: no network`.
- **mTLS rejects what it should.** Tested with a real BuildKit API call (`buildctl debug workers`):
  - no client cert → `remote error: tls: certificate required`;
  - a client cert from another CA → rejected the same way;
  - the valid cert → accepted.

  Through the driver:
  - another CA's client cert → rc 1;
  - the wrong CA for the server → rc 1;
  - an unreachable host → rc 1 in about 20 s, with the explanatory message;
  - missing secrets → rc 1, naming the three secrets.
- **The local podman path**, `build-image.sh` for all three (rc 0).
  - Negative control: a verify stage expecting a different promtool version printed
    `DRIFT promtool want 9.9.9` and failed the build.
  - Negative control: the `--network=none` check fails when the flag is removed.
- **`buildkit-client-secret.sh`'s sanity checks**, run offline on local certs. They tell apart:
  - a good client cert;
  - a server cert (not client-auth);
  - a mismatched key;
  - a cert from another CA.

**Measured live, read-only:**
- node user namespaces, Debian 13.7 and AppArmor (via an Ansible `sysctl`/`cat`);
- DinD's MASQUERADE rules;
- the live ClusterPolicies, namespaced Policies, PSA labels, and cert-manager v1.21.3 with its
  ClusterIssuers;
- the act-runner pod labels.

**Renovate (unchanged since the first iteration):**
- the validator with 44.167.0: exit 0;
- `extract` finds all four markers;
- the `kubernetes` duplicates are `skipReason: disabled`;
- 39.264.0 rejects `main`'s config.

**Local gate, CI results:** see the PR body. They are recorded there because this entry is part of
the tree being gated.

## What I did NOT verify

- **platform/buildkit has never run in the cluster.** It can't until it merges: agents may not
  `kubectl apply`, and Flux reconciles only `main`. So this PR's own `images` job **cannot build
  and is expected to fail at the builder preflight**, which is the designed loud failure. The
  following are therefore all unproven in-cluster:
  - kubelet admitting the pod;
  - containerd honouring `appArmorProfile: Unconfined`;
  - the five emptyDirs;
  - cert-manager issuing the three certs;
  - the CiliumNetworkPolicies.
- **AppArmor `Unconfined` is reasoned, not measured.** My test host runs SELinux, so the AppArmor
  requirement comes from upstream's example plus the nodes having AppArmor enabled. If the pod
  were to work under `RuntimeDefault` AppArmor, that line could be tightened; I did not test it.
- **The local userns mapping is narrower than the cluster's.** Locally it was 65,535 ids across
  two ranges; in the cluster it is the image's own `100000:65536` against real host subuids. That
  is the normal case, but it is the cluster's first test.
- **The NetworkPolicies are untested.** In particular:
  - that `toEntities: [ingress]` admits the push to the Gateway VIP. The comment explains the
    reasoning, but it is unmeasured;
  - that world :443/:80 covers every download the builds make. A dnf mirror on another port
    would fail a build visibly.
- **No push, anywhere.** The whole publish path is unexercised:
  - `REGISTRY_PUSH_TOKEN` reaching buildkitd through the buildx session;
  - `push=true`;
  - the metadata digest compared with the registry read-back;
  - agent-runner FROM `toolbox@sha256:`;
  - the immutable-tag guard;
  - idempotence.

  So are the 15 s Envoy route-timeout risk on blob finalisation, and the Gitea PVC (10Gi, shared
  with repositories) absorbing ~2 GB per toolbox publish until retention runs.
- **Node disk under real builds.** The cache emptyDir (6Gi limit, 4 GB GC cap) sits on nodes that
  had 7–9 GB free. It is untested under concurrent pressure.
- **Certificate rotation is designed, not exercised.** The server cert renews 30 days early but
  buildkitd only reads it at start; the client copy needs the script re-run.
- **`--oci-worker-no-process-sandbox` is an accepted risk, not a mitigated one.** A build step can
  ptrace the daemon and read its TLS key and cache. Anyone with the client cert, which means every
  PR job, can run build steps.
- **Renovate has still never completed a run.** It is blocked on #80, on `RENOVATE_TOKEN` gaining
  `read:package`, and on this branch. The `[email protected]` + 44.x image pairing is
  untested.
- **Pods cannot pull these images until PR #61** (the kubelet pull secret) merges.
- **`registry-retention.sh --apply` has never run**; its keep logic has only run against an empty
  registry.