home-network-k3s-impl/decisions/2026-10-29-image-pipeline.md

date, branch, title

date	branch	title
2026-10-29	feat/image-pipeline	Images are built, verified and published by CI on rootless BuildKit; Renovate carries the digests into git

What changed

Operator rule (2026-10-29): container image creation, hosting and lifecycle belong to CI/CD and the codebase, never to a one-off workstation build/push. Before this branch the Home-Network org's package list was empty, platform/pm-agent/ pinned an all-zero digest (ImagePullBackOff), and both build-image.sh scripts said "publishing is an operator action".

The builder: rootless BuildKit — DECISION MADE (operator, on PR #81)

The problem it answers, measured on this PR's first CI run (images run 325, failed in 6 s): a CI job on this runner gets no Docker daemon at all, for two reasons, both in the deployed versions' source:

• act_runner v0.2.11 internal/app/cmd/daemon.go sets container.docker_host to - ("never mount into jobs") whenever DOCKER_HOST is tcp://, which it is here (the DinD sidecar);
• gitea/act v0.261.2 pkg/container/docker_run.go sanitizeConfig drops every bind, socket included, when valid_volumes is empty, which it is (the runner has no config file).

Giving jobs the DinD daemon would have made every branch push root on the node, because that daemon is privileged: true.

The decision: no job may reach a privileged daemon, and images are built by a rootless BuildKit daemon over mTLS. The runner (apps/wave-1/gitea-actions.yaml, DOCKER_HOST, valid_volumes) is untouched.

• platform/buildkit/ — a new hand-managed component in its own namespace buildkit. The runner manifest is renderer-generated (catalog/render.py EXTRA_UNITS), and the builder has no business sitting next to a privileged container. Flux reconciles it as the new Kustomization buildkit (clusters/base/buildkit.yaml, dependsOn: platform).

  • Deployment: moby/buildkit:v0.34.1-rootless@sha256:f68752a5…0c4f89, pinned by index digest. The securityContext, every line measured (see What I verified):

    • privileged: false;
    • uid/gid 1000, runAsNonRoot;
    • capabilities: drop ALL, add SETUID, SETGID;
    • allowPrivilegeEscalation: true;
    • seccompProfile: Unconfined and appArmorProfile: Unconfined;
    • readOnlyRootFilesystem: true, with five emptyDirs.

    Other settings:

    • --oci-worker-no-process-sandbox;
    • no ServiceAccount token;
    • memory limit 4Gi, which covers build steps too;
    • cache emptyDir with sizeLimit: 6Gi, and GC capped at 4 GB in buildkitd.toml (ConfigMap).

  • mTLS via cert-manager (certificates.yaml): a self-signed CA used only here, the server cert buildkitd-tls, and the client cert buildkit-client-ci. buildkitd verifies every client against that CA.

  • Service: ClusterIP buildkitd.buildkit.svc.cluster.local:1234.

  • NetworkPolicies:

    • default-deny;
    • ingress on 1234 only from the act-runner pod (gitea-actions, app.kubernetes.io/name=act-runner);
    • egress: DNS, world :443/:80 for downloads, and the ingress entity (the Gateway) for the registry push.

  • versions.env: the CI-side client, buildx v0.38.1 with the sha256 from its checksums.txt, plus the address and certificate name CI dials.

• How DinD job containers appear to Cilium (measured, read-only). iptables -t nat -S POSTROUTING in the dind container shows -A POSTROUTING -s 172.18.0.0/16 ! -o br-… -j MASQUERADE (and the same for docker0). So a job's connection leaves with the act-runner pod's IP, and therefore that pod's Cilium identity. That makes "only the act-runner pod" expressible and correct, but it cannot tell one job from another, or a job from act_runner itself. Per-job authentication is the client certificate.

• Admission: no Kyverno exception needed.

  • Every ClusterPolicy actually applied (platform-config/kustomization.yaml; live: conformance-trigger-job-shape, label-cdi-worker-pods, pm-agent-job-shape, require-auth-mode-label, require-default-deny-baseline, require-egress-class-label, restrict-host-namespaces, secret-provenance, validate-default-deny-shape) matches only namespaces labelled homelab.keithjackson.dev/app-namespace: "true", or only its own namespace in the job-shape policies' case. buildkit carries no such label.
  • There are no namespaced Kyverno Policies and no cluster-wide PSA default (both measured).
  • The namespace is deliberately left without a PSA label, because PSS baseline forbids Unconfined seccomp and AppArmor.
  • validate-default-deny-shape does apply to the namespace's default-deny NetworkPolicy, and its shape (both directions) satisfies it.

• Unprivileged user namespaces on the nodes (measured, read-only, via Ansible sysctl). On all three Debian 13.7 nodes, kernel 7.1.8+deb13-amd64:

  • kernel.unprivileged_userns_clone = 1;
  • user.max_user_namespaces = 127206;
  • no kernel.apparmor_restrict_unprivileged_userns (an Ubuntu-only key);
  • AppArmor enabled (Y).

  So this is a satisfied prerequisite, not an open one.

• The TLS conformance check needed a narrow, self-proving exception (conformance/**, so this changes what green means; flagged for the reviewer).

  • The first gate run failed L0 tls-lifecycle-automated clause (5): "every Certificate's issuer must be ACME". That is broader than the spec it serves:
    • R-TLS-01 and docs/08 concern externally reachable services;
    • ADR-0006 says "mTLS between workloads is a separate concern … not covered here";
    • a public CA cannot issue for a *.svc.cluster.local name or a client-auth certificate.
  • The new clause (5b) exempts a Certificate annotated homelab.keithjackson.dev/tls-scope: workload-mtls only if it proves it:
    • no Gateway listener serves its Secret;
    • every DNS name and the commonName are a bare label or *.svc[.cluster.local];
    • it has no IP, URI or email SANs;
    • its issuer is a namespaced selfSigned/ca Issuer in its own namespace;
    • a ca issuer's Secret is produced by a likewise-exempt tracked Certificate.
  • Exemptions are printed, and clause (9), the renewal arithmetic, still applies.
  • Negative-tested: a public DNS name, and a Gateway serving buildkitd-tls, each fail the check with a specific message.

• infra/gitea/buildkit-client-secret.sh (operator action): copies ca.crt, tls.crt and tls.key from the Secret buildkit/buildkit-client-ci into the Actions secrets BUILDKIT_CA, BUILDKIT_CLIENT_CERT and BUILDKIT_CLIENT_KEY. Before writing anything it checks that:

  • the cert verifies against the CA;
  • the cert is a client-auth cert;
  • the key matches the cert.

  It never prints a value. --check reports whether the stored copy is older than the current certificate.

Verification moved into the Containerfiles

docker run cannot exist on BuildKit, so each Containerfile now names its shipped stage after the image (toolbox, agent-runner, conformance-runner) and ends in a verify test stage, FROM <image> AS verify. That stage runs every previous assertion at build time, as the image's non-root uid (65532 for agent-runner and conformance-runner), in RUN --network=none. It is never pushed.

• Every existing assertion is preserved, and the toolbox gained one it never had: every tool on PATH, and five tools report their pinned versions.

• probe-validation.sh is COPYed into the verify stage only, so it never ships.

• Bug fixed on the way: in the old docker run block, cmd | grep -q … && echo ok and [ … ] && echo could never fail the script under set -e + pipefail. The "entrypoints refuse arguments", "not root", "no API key baked in" and "HOME writable" checks asserted nothing. They are real assertions now.

• What a build step cannot reproduce, and how it is approximated (also written in each stage's header):

  • uid: exact.

  • --network none: exact, and each stage proves it. Its first check fails if DNS or a TCP connection works. A negative control without --network=none fails that check.

  • --read-only rootfs: not reproducible, because a RUN step always has a writable layer. It is approximated by permission:

    • touch /should-fail must fail;
    • a find / -xdev -writable ! -type l inventory, run as the uid, must lie entirely under /workspace, /tmp or /var/tmp. ! -type l was needed: Fedora's /usr/tmp symlink to /var/tmp tripped it.

    The real control remains the pods' readOnlyRootFilesystem, which the admission policies enforce.

  • --tmpfs /workspace: approximated by the image's own /workspace, owned by 65532, which is exactly what the pod's emptyDir replaces.

  The opt-in --verify-clone still runs the image, so it remains local-only.

• build-image.sh (all three):

  • verify is build --target verify and the image is build --target <image>;
  • a new --print-build-args prints the pins as KEY=VALUE, which is what CI consumes. The arg list therefore exists once;
  • the local podman path still works.

• The L0 check alert-rules-unit-tested now looks for "PROMTOOL_VERSION=$PROMTOOL_VERSION" in the toolbox's build-arg list.

The pipeline (.gitea/workflows/images.yaml + scripts/images.sh)

• The builder. The driver downloads the pinned buildx and refuses a checksum mismatch. It writes the client cert from the BUILDKIT_* secrets to a 0600 temp dir that is removed on exit, creates a remote builder, and runs buildx inspect --bootstrap.
• It fails loudly, never skips. It stops with a clear message if:
  • the secrets are absent;
  • buildkitd is unreachable. The message lists the causes in order: not deployed yet (Flux applies it only after merge), stale certificate, NetworkPolicy.
• Pull request: buildx bake <image>-verify for changed images. Nothing is pushed and no registry credential is referenced. The BUILDKIT_* builder secrets are given to PR jobs, because they must build.
  • agent-runner is built FROM the toolbox built in the same run through a bake target: context. That toolbox exists in no registry, so the toolbox is built as its base even when its own inputs did not change.
• Push to main / dispatch on main: for each image, the verify target first, and only then the image itself with push=true. Tags are sha-<12 hex> (immutable) and main (moving).
  • The digest is read back from the registry and compared with the client's metadata.
  • agent-runner builds FROM the toolbox digest just pushed, or else from the pinned AGENT_BASE_IMAGE (the zero placeholder bootstraps).
  • BUILDX_NO_DEFAULT_ATTESTATIONS=1 keeps one manifest per tag, which is what admission, Renovate and the retention script assume.
• Unchanged from the first iteration: change detection, publish idempotence, REGISTRY_PUSH_TOKEN and its preflight, the Renovate markers, and registry-retention.sh. The DinD prune step is gone, because nothing is built in DinD now.

Unchanged from the first iteration (still true)

• Digest references carry # renovate-digest: image=<repo> tag=main markers:

  • platform/pm-agent/deployment.yaml and configmap.yaml;
  • infra/agent/versions.env AGENT_BASE_IMAGE, now the published toolbox by digest, zero until the first publish;
  • conformance/versions.env CONFORMANCE_RUNNER_IMAGE_DIGEST, which went from "" to zero.

  The admission policies are untouched.

• renovate.json5:

  • the custom.regex marker manager;
  • enabledManagers regex → custom.regex;
  • the kubernetes manager disabled for these three repos;
  • ignorePaths narrowed so conformance/versions.env is visible.

  renovate.yaml pins renovate-version: 44.167.0 and adds a docker RENOVATE_HOST_RULES entry using RENOVATE_TOKEN, which now needs read:package.

• infra/gitea/registry-retention.sh (dry run by default; operator action); the conformance-trigger-kubeconfig.sh precondition fix; and the docs (pm-agent README, runbook, plan, conformance README, devcontainer README, workflows README). The docs are updated again for BuildKit, and there is a new platform/buildkit/README.md.

Decisions and pushback

• The push credential is a PAT secret, REGISTRY_PUSH_TOKEN; the Actions token cannot push. Checked against Gitea v1.24.3 source:

  • services/auth/basic.go Basic.Verify turns a task token into the pseudo-user gitea-actions (ID −2) and sets no token scope;
  • routers/api/packages/container/container.go Authenticate (the /v2/token exchange every docker client performs) requires read:package/write:package/all scope, so it answers 401. The Actions token cannot even pull;
  • services/context/package.go determineAccessMode has // TODO: ActionUser permission check and no branch for that user. Even past login it would get at most read on a public org;
  • models/repo/repo_unit.go ActionsConfig has no token-permissions setting in 1.24.3.

  The workflow fails loudly when the secret is absent, copying the RENOVATE_TOKEN preflight pattern. With BuildKit the credential reaches the daemon through the buildx session (a 0600 config.json in the job's temp dir), never through the cluster.

• regex vs custom.regex: both work. Switched to the canonical custom.regex. In Renovate:

  • lib/config/migrations/custom/enabled-managers-migration.ts rewrites regex → custom.regex;
  • that migration runs on every repository run (lib/workers/repository/init/merge.ts → migrateAndValidate), not only in the validator;
  • lib/modules/manager/index.ts getEnabledManagersList accepts either spelling;
  • custom.regex exists since 37.51.1 (renovatebot/renovate#24079).

• The open question hid a bigger bug: the Renovate config on main is invalid for the Renovate the workflow actually runs.

  • renovatebot/[email protected] defaults renovate-version to 39 (action.yml, src/docker.ts), which is 39.264.0.
  • The validator used in #79 was ghcr.io/renovatebot/renovate with no tag, i.e. latest, 44.167.0.
  • Measured: renovate:39.264.0 renovate-config-validator on main's renovate.json5 gives ERROR: Found errors in configuration with five errors, all managerFilePatterns. fileMatch was renamed in 40.2.0.
  • So once #80 fixes the filename, every run would still die at CONFIG_VALIDATION.
  • Fixed by pinning renovate-version: 44.167.0. This branch's config validates clean on it. The only --strict complaint is a pending migration of the pre-existing top-level "docker": {"enabled": true} block, which I left alone.

• Unworkable as briefed: tracking the worker Job template's digest.

  • infra/agent/worker-job-template.yaml is COPYed into agent-runner. A Renovate bump there changes agent-runner's inputs, CI publishes a new digest, Renovate bumps it again, and so on forever.
  • pm-dispatch overwrites that field from PM_WORKER_IMAGE at runtime (pm-dispatch.py:649), so nothing is lost by leaving it all-zero. It is commented as deliberately untracked.
  • For the same loop reason, conformance/versions.env is not an input of the conformance-runner build, and a pipeline-only change does not republish on main.

• Changed from the brief: retention is a script, not a Gitea cleanup rule.

  • Gitea 1.24.3 has no API for cleanup rules (none in routers/api/v1/api.go; web session+CSRF forms only, in routers/web/web.go). So a rule cannot be created, or drift-asserted, as code. ci/gitea-settings.sh was therefore not extended. A cheaper proxy (count versions per image) would need read:package on SETTINGS_API_TOKEN, which it lacks.
  • A built-in rule is also unsafe here. Gitea stores a manifest as a file of the TAG version it was pushed under (routers/api/packages/container/manifest.go createPackageAndVersion), and resolves a by-digest pull from those files (models/packages/container/search.go). Deleting the tag a deployed digest was pushed under breaks the @sha256: pull, and the rule's "newest N" knows nothing about what git pins. The script keeps every digest git pins.

• Auto-merge. Digest bumps of these images auto-merge on a green gate (the existing digest rule). The exception is the conformance-runner bump: conformance/versions.env is a sensitive path, so it waits for an administrator's /approve-override every time.

What I verified

Proved locally, against the exact pinned images:

• Rootless buildkitd, configured as the manifest configures it.

  • Setup: the pinned moby/buildkit:v0.34.1-rootless digest, under rootless podman, with --cap-drop ALL --cap-add SETUID --cap-add SETGID, seccomp unconfined and --read-only; tmpfs at the same five paths; the ConfigMap's buildkitd.toml; mTLS on :1234.
  • The image's /etc/subuid was overridden with user:1:999 and user:1001:64535, because rootless podman maps only 65,537 uids. rootlesskit's real newuidmap user-namespace path still ran.
  • It started, buildctl debug workers listed the OCI worker, and debug workers -v showed the GC config applied (max 4.29 GB, reserved 1.07 GB).

• The securityContext, item by item. Each requirement was measured by taking it away:

  • default seccomp: daemon starts, every RUN step fails → Unconfined is required;
  • no SETUID/SETGID: rootlesskit dies with newuidmap … operation not permitted → both capabilities are required;
  • --read-only rootfs: works once the five emptyDirs exist. Its only failure was the probe client's ~/.docker, which is now an emptyDir;
  • minimal caps + Unconfined seccomp: works, and a RUN step executes.

• The full CI path, in the actual CI job image. Setup: ghcr.io/catthehacker/ubuntu:act-22.04 running .gitea/workflows/scripts/images.sh in verify mode. It downloaded and sha256-checked buildx v0.38.1, connected over mTLS, and bake-built all three verify targets from an empty cache. rc 0, 1m32s total:

  • TOOLBOX VERIFIED (42 s);
  • AGENT-RUNNER VERIFIED (26 s): selftest OK, probe "20 refusal(s), 0 failure(s)", and built FROM the in-run toolbox through the target: context, with the log showing [agent-runner-verify toolbox 1/6] and base=toolbox-in-this-run;
  • CONFORMANCE-RUNNER VERIFIED (21 s): all seven refusals;
  • every stage printed confirmed: no network.

• mTLS rejects what it should. Tested with a real BuildKit API call (buildctl debug workers):

  • no client cert → remote error: tls: certificate required;
  • a client cert from another CA → rejected the same way;
  • the valid cert → accepted.

  Through the driver:

  • another CA's client cert → rc 1;
  • the wrong CA for the server → rc 1;
  • an unreachable host → rc 1 in about 20 s, with the explanatory message;
  • missing secrets → rc 1, naming the three secrets.

• The local podman path, build-image.sh for all three (rc 0).

  • Negative control: a verify stage expecting a different promtool version printed DRIFT promtool want 9.9.9 and failed the build.
  • Negative control: the --network=none check fails when the flag is removed.

• buildkit-client-secret.sh's sanity checks, run offline on local certs. They tell apart:

  • a good client cert;
  • a server cert (not client-auth);
  • a mismatched key;
  • a cert from another CA.

Measured live, read-only:

• node user namespaces, Debian 13.7 and AppArmor (via an Ansible sysctl/cat);
• DinD's MASQUERADE rules;
• the live ClusterPolicies, namespaced Policies, PSA labels, and cert-manager v1.21.3 with its ClusterIssuers;
• the act-runner pod labels.

Renovate (unchanged since the first iteration):

• the validator with 44.167.0: exit 0;
• extract finds all four markers;
• the kubernetes duplicates are skipReason: disabled;
• 39.264.0 rejects main's config.

Local gate, CI results: see the PR body. They are recorded there because this entry is part of the tree being gated.

What I did NOT verify

• platform/buildkit has never run in the cluster. It can't until it merges: agents may not kubectl apply, and Flux reconciles only main. So this PR's own images job cannot build and is expected to fail at the builder preflight, which is the designed loud failure. The following are therefore all unproven in-cluster:

  • kubelet admitting the pod;
  • containerd honouring appArmorProfile: Unconfined;
  • the five emptyDirs;
  • cert-manager issuing the three certs;
  • the CiliumNetworkPolicies.

• AppArmor Unconfined is reasoned, not measured. My test host runs SELinux, so the AppArmor requirement comes from upstream's example plus the nodes having AppArmor enabled. If the pod were to work under RuntimeDefault AppArmor, that line could be tightened; I did not test it.

• The local userns mapping is narrower than the cluster's. Locally it was 65,535 ids across two ranges; in the cluster it is the image's own 100000:65536 against real host subuids. That is the normal case, but it is the cluster's first test.

• The NetworkPolicies are untested. In particular:

  • that toEntities: [ingress] admits the push to the Gateway VIP. The comment explains the reasoning, but it is unmeasured;
  • that world :443/:80 covers every download the builds make. A dnf mirror on another port would fail a build visibly.

• No push, anywhere. The whole publish path is unexercised:

  • REGISTRY_PUSH_TOKEN reaching buildkitd through the buildx session;
  • push=true;
  • the metadata digest compared with the registry read-back;
  • agent-runner FROM toolbox@sha256:;
  • the immutable-tag guard;
  • idempotence.

  So are the 15 s Envoy route-timeout risk on blob finalisation, and the Gitea PVC (10Gi, shared with repositories) absorbing ~2 GB per toolbox publish until retention runs.

• Node disk under real builds. The cache emptyDir (6Gi limit, 4 GB GC cap) sits on nodes that had 7–9 GB free. It is untested under concurrent pressure.

• Certificate rotation is designed, not exercised. The server cert renews 30 days early but buildkitd only reads it at start; the client copy needs the script re-run.

• --oci-worker-no-process-sandbox is an accepted risk, not a mitigated one. A build step can ptrace the daemon and read its TLS key and cache. Anyone with the client cert, which means every PR job, can run build steps.

• Renovate has still never completed a run. It is blocked on #80, on RENOVATE_TOKEN gaining read:package, and on this branch. The [email protected] + 44.x image pairing is untested.

• Pods cannot pull these images until PR #61 (the kubelet pull secret) merges.

• registry-retention.sh --apply has never run; its keep logic has only run against an empty registry.