Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

feat(images): build, verify and publish the three images in CI on rootless BuildKit; Renovate carries the digests #81

Merged
jakman85 merged 4 commits from feat/image-pipeline into main 2026-10-30 05:15:16 +00:00
Conversation 2 Commits 4 Files Changed 40
+2482 -359
gitea-admin commented 2026-10-30 02:36:25 +00:00
Owner
Copy link
Copy source

Operator rule (2026-10-29): image creation, hosting and lifecycle belong to CI/CD, never to a workstation build/push. Until now nothing had ever been pushed to this Gitea's registry, and pm-agent sits in ImagePullBackOff on an all-zero digest.

Operator decision on this PR: the builder is rootless BuildKit. No CI job may reach a privileged daemon. The act runner is untouched.

What this does

  • platform/buildkit/, a new component in its own namespace, reconciled by the Flux Kustomization buildkit.
    • Image: moby/buildkit:v0.34.1-rootless, pinned by digest.
    • Not privileged. Runs as uid 1000, with drop ALL + SETUID/SETGID, seccomp and AppArmor Unconfined, and a read-only rootfs. Each line is commented with how it was measured.
    • mTLS from a cert-manager CA that exists only for this, behind a ClusterIP.
    • NetworkPolicy: ingress only from the act-runner pod, the identity Cilium sees for DinD job traffic (measured: dockerd masquerades jobs to the pod IP).
  • Verification moved into the Containerfiles. Each now ends in a verify stage that holds every former docker run assertion, run as uid 65532 with RUN --network=none.
    • A read-only rootfs can't be reproduced in a build step, so it is approximated by a writable-path inventory; the stage headers say so.
    • Three old && echo checks could never fail. They are real assertions now.
  • .gitea/workflows/images.yaml + scripts/images.sh drive buildkitd with a pinned, sha256-checked buildx (bake).
    • PR: builds the verify targets only. agent-runner is built FROM the same-run toolbox via a target: context.
    • main: verify first, then push sha-<commit> and main, then read the digest back from the registry.
    • The job fails loudly if the builder secrets are missing or buildkitd is unreachable.
  • infra/gitea/buildkit-client-secret.sh copies the client certificate into the Actions secrets BUILDKIT_CA, BUILDKIT_CLIENT_CERT and BUILDKIT_CLIENT_KEY. It never prints them.
  • Unchanged from before:
    • Renovate # renovate-digest: markers and the custom.regex manager;
    • renovate-version: 44.167.0, because 39.264.0 rejects main's config;
    • the REGISTRY_PUSH_TOKEN preflight (the Actions token has no package scope in Gitea 1.24.3);
    • registry-retention.sh.

For the reviewer: a conformance check changed

L0 tls-lifecycle-automated required every Certificate to come from an ACME issuer. The spec it serves is about externally reachable TLS (R-TLS-01; ADR-0006: "mTLS between workloads is a separate concern … not covered here"), and no public CA will issue for *.svc.cluster.local or client-auth certificates.

New clause (5b) exempts a Certificate annotated homelab.keithjackson.dev/tls-scope: workload-mtls only if it proves it is internal:

  • no Gateway serves its Secret;
  • its names are internal only;
  • its issuer is a namespaced selfSigned/CA Issuer backed by a likewise-exempt CA.

Negative-tested: a public DNS name fails, and a Gateway serving the Secret fails.

Admission

No Kyverno exception needed. Every applied ClusterPolicy (the list in platform-config/kustomization.yaml, plus a live listing) matches only app-namespace: "true" namespaces, or only its own namespace for the job-shape policies. buildkit carries neither. There are no namespaced Policies and no cluster PSA default; the namespace is deliberately unlabelled for PSA, because baseline forbids Unconfined.

Proven locally vs reasoned

  • Proven, using the exact pinned images:
    • Rootless buildkitd ran under podman with the manifest's restrictions, and each requirement was confirmed by removing it: default seccomp makes RUN steps fail, and without SETUID/SETGID newuidmap fails.
    • Full CI path in the real job image (catthehacker/ubuntu:act-22.04): pinned buildx → mTLS → bake built all three verify targets from an empty cache. rc 0, 1m32s. TOOLBOX/AGENT-RUNNER/CONFORMANCE-RUNNER VERIFIED; probe 20/20; all 7 refusals; agent-runner on the in-run toolbox.
    • mTLS: no client cert is rejected (tls: certificate required), another CA's cert is rejected, the valid cert is accepted.
    • Loud failure when buildkitd is unreachable (about 20s) and when the secrets are missing.
    • Negative controls: a wrong pin fails the verify stage; removing --network=none fails the network check.
  • Measured live, read-only:
    • all 3 Debian 13.7 nodes: kernel.unprivileged_userns_clone=1, user.max_user_namespaces=127206, AppArmor enabled;
    • DinD MASQUERADE rules;
    • the live ClusterPolicies and PSA labels.
  • Reasoned, not measured:
    • that AppArmor Unconfined is needed (my test host is SELinux);
    • that the egress ingress-entity rule admits the registry push;
    • everything in-cluster: kubelet admission, cert-manager issuance, the NetworkPolicies, node disk;
    • the entire publish path (nothing has been pushed anywhere).

Gate (pinned toolbox, worktree, committed head 82235dd)

rc 3, MERGE GATE: UNDETERMINED. Nothing failed.

  • Passed: spec-immutability, decisions-log (56 entries), lint, l0 (82 Kyverno tests, plus the TLS check with 3 proven exemptions), render-golden, l1, approval-policy (offline N/A).
  • Not run: gitea-settings (CI-only) and L2/L3 conformance.
  • The first run on this iteration was rc 1, on the TLS check above and a yamllint comma error; both are fixed.

CI on this PR: images WILL FAIL, and that's by design

buildkitd does not exist until this merges and Flux applies platform/buildkit; I can't apply it. The job stops at its builder preflight with a message naming the cause. gate fails only on approval-policy (sensitive paths) until an administrator posts /approve-override <reason> as a review of type Comment. I have not posted it and will not.

Operator prerequisites, in order

  1. Post /approve-override <reason> on this PR, after reviewing the TLS-check change and platform/buildkit/deployment.yaml's securityContext. Then merge.
  2. Confirm Flux Kustomization buildkit is Ready and the 3 Certificates in buildkit are READY. Read-only check: kubectl --context k3s-prod -n buildkit get certificate,pod.
  3. Run infra/gitea/buildkit-client-secret.sh, which needs kubectl and the Gitea credential. It writes BUILDKIT_CA, BUILDKIT_CLIENT_CERT and BUILDKIT_CLIENT_KEY.
  4. Create REGISTRY_PUSH_TOKEN: scope write:package, from an account with package write on Home-Network. Optionally set the variable REGISTRY_PUSH_USER.
  5. Re-run images with workflow_dispatch on main, images: all. This is the first real build and publish.
  6. Re-mint RENOVATE_TOKEN with read:package, and merge #80. On rebase, take #80's renovate-version lines and keep only this PR's read:package text and RENOVATE_HOST_RULES.
  7. Merge #61 (the kubelet pull secret), then merge Renovate's digest PRs. The conformance one needs an override each time.
  8. Yearly: re-run step 3 after the client cert renews, and restart buildkitd inside the server cert's 30-day renewal window. Periodically run registry-retention.sh (dry run first).

Full detail, including what I did NOT verify: decisions/2026-10-29-image-pipeline.md and platform/buildkit/README.md.

🤖 Generated with Claude Code

Operator rule (2026-10-29): image creation, hosting and lifecycle belong to CI/CD, never to a workstation build/push. Until now nothing had ever been pushed to this Gitea's registry, and `pm-agent` sits in ImagePullBackOff on an all-zero digest. **Operator decision on this PR: the builder is rootless BuildKit. No CI job may reach a privileged daemon.** The act runner is untouched. ## What this does - **`platform/buildkit/`**, a new component in its own namespace, reconciled by the Flux Kustomization `buildkit`. - Image: `moby/buildkit:v0.34.1-rootless`, pinned by digest. - Not privileged. Runs as uid 1000, with `drop ALL` + `SETUID`/`SETGID`, seccomp and AppArmor `Unconfined`, and a read-only rootfs. Each line is commented with how it was measured. - mTLS from a cert-manager CA that exists only for this, behind a ClusterIP. - NetworkPolicy: ingress only from the act-runner pod, the identity Cilium sees for DinD job traffic (measured: dockerd masquerades jobs to the pod IP). - **Verification moved into the Containerfiles.** Each now ends in a `verify` stage that holds every former `docker run` assertion, run as uid 65532 with `RUN --network=none`. - A read-only rootfs can't be reproduced in a build step, so it is approximated by a writable-path inventory; the stage headers say so. - Three old `&& echo` checks could never fail. They are real assertions now. - **`.gitea/workflows/images.yaml` + `scripts/images.sh`** drive buildkitd with a pinned, sha256-checked `buildx` (`bake`). - **PR:** builds the verify targets only. agent-runner is built FROM the same-run toolbox via a `target:` context. - **main:** verify first, then push `sha-<commit>` and `main`, then read the digest back from the registry. - The job **fails loudly** if the builder secrets are missing or buildkitd is unreachable. - **`infra/gitea/buildkit-client-secret.sh`** copies the client certificate into the Actions secrets `BUILDKIT_CA`, `BUILDKIT_CLIENT_CERT` and `BUILDKIT_CLIENT_KEY`. It never prints them. - **Unchanged from before:** - Renovate `# renovate-digest:` markers and the `custom.regex` manager; - `renovate-version: 44.167.0`, because 39.264.0 rejects main's config; - the `REGISTRY_PUSH_TOKEN` preflight (the Actions token has no package scope in Gitea 1.24.3); - `registry-retention.sh`. ## For the reviewer: a conformance check changed L0 `tls-lifecycle-automated` required **every** Certificate to come from an ACME issuer. The spec it serves is about externally reachable TLS (R-TLS-01; ADR-0006: "mTLS between workloads is a separate concern … not covered here"), and no public CA will issue for `*.svc.cluster.local` or client-auth certificates. New clause (5b) exempts a Certificate annotated `homelab.keithjackson.dev/tls-scope: workload-mtls` **only if it proves it is internal**: - no Gateway serves its Secret; - its names are internal only; - its issuer is a namespaced selfSigned/CA Issuer backed by a likewise-exempt CA. Negative-tested: a public DNS name fails, and a Gateway serving the Secret fails. ## Admission **No Kyverno exception needed.** Every applied ClusterPolicy (the list in `platform-config/kustomization.yaml`, plus a live listing) matches only `app-namespace: "true"` namespaces, or only its own namespace for the job-shape policies. `buildkit` carries neither. There are no namespaced Policies and no cluster PSA default; the namespace is deliberately unlabelled for PSA, because `baseline` forbids `Unconfined`. ## Proven locally vs reasoned - **Proven, using the exact pinned images:** - **Rootless buildkitd** ran under podman with the manifest's restrictions, and each requirement was confirmed by removing it: default seccomp makes RUN steps fail, and without SETUID/SETGID `newuidmap` fails. - **Full CI path** in the real job image (`catthehacker/ubuntu:act-22.04`): pinned buildx → mTLS → bake built all three verify targets from an empty cache. rc 0, 1m32s. `TOOLBOX`/`AGENT-RUNNER`/`CONFORMANCE-RUNNER VERIFIED`; probe 20/20; all 7 refusals; agent-runner on the in-run toolbox. - **mTLS:** no client cert is rejected (`tls: certificate required`), another CA's cert is rejected, the valid cert is accepted. - **Loud failure** when buildkitd is unreachable (about 20s) and when the secrets are missing. - **Negative controls:** a wrong pin fails the verify stage; removing `--network=none` fails the network check. - **Measured live, read-only:** - all 3 Debian 13.7 nodes: `kernel.unprivileged_userns_clone=1`, `user.max_user_namespaces=127206`, AppArmor enabled; - DinD MASQUERADE rules; - the live ClusterPolicies and PSA labels. - **Reasoned, not measured:** - that AppArmor `Unconfined` is needed (my test host is SELinux); - that the egress `ingress`-entity rule admits the registry push; - everything in-cluster: kubelet admission, cert-manager issuance, the NetworkPolicies, node disk; - the entire publish path (nothing has been pushed anywhere). ## Gate (pinned toolbox, worktree, committed head `82235dd`) **rc 3, MERGE GATE: UNDETERMINED.** Nothing failed. - Passed: `spec-immutability`, `decisions-log` (56 entries), `lint`, `l0` (82 Kyverno tests, plus the TLS check with 3 proven exemptions), `render-golden`, `l1`, `approval-policy` (offline N/A). - Not run: `gitea-settings` (CI-only) and L2/L3 `conformance`. - The first run on this iteration was rc 1, on the TLS check above and a yamllint comma error; both are fixed. ## CI on this PR: `images` WILL FAIL, and that's by design buildkitd does not exist until this merges and Flux applies `platform/buildkit`; I can't apply it. The job stops at its builder preflight with a message naming the cause. `gate` fails only on `approval-policy` (sensitive paths) until an administrator posts `/approve-override <reason>` as a review of type Comment. I have not posted it and will not. ## Operator prerequisites, in order 1. Post `/approve-override <reason>` on this PR, after reviewing the TLS-check change and `platform/buildkit/deployment.yaml`'s securityContext. Then merge. 2. Confirm Flux Kustomization `buildkit` is Ready and the 3 Certificates in `buildkit` are READY. Read-only check: `kubectl --context k3s-prod -n buildkit get certificate,pod`. 3. Run `infra/gitea/buildkit-client-secret.sh`, which needs kubectl and the Gitea credential. It writes `BUILDKIT_CA`, `BUILDKIT_CLIENT_CERT` and `BUILDKIT_CLIENT_KEY`. 4. Create **`REGISTRY_PUSH_TOKEN`**: scope `write:package`, from an account with package write on Home-Network. Optionally set the variable `REGISTRY_PUSH_USER`. 5. Re-run `images` with `workflow_dispatch` on `main`, `images: all`. This is the first real build and publish. 6. Re-mint **`RENOVATE_TOKEN`** with `read:package`, and merge **#80**. On rebase, take #80's `renovate-version` lines and keep only this PR's `read:package` text and `RENOVATE_HOST_RULES`. 7. Merge **#61** (the kubelet pull secret), then merge Renovate's digest PRs. The conformance one needs an override each time. 8. Yearly: re-run step 3 after the client cert renews, and restart buildkitd inside the server cert's 30-day renewal window. Periodically run `registry-retention.sh` (dry run first). Full detail, including what I did NOT verify: `decisions/2026-10-29-image-pipeline.md` and `platform/buildkit/README.md`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
gitea-admin added 1 commit 2026-10-30 02:36:25 +00:00
feat(images): build, verify and publish the three images in CI; Renovate carries the digests
gate / gate (pull_request) Failing after 1m56s
Details
images / images (pull_request) Failing after 6s
Details
gate / conformance (pull_request) Has been skipped
Details
0da34ddbc6 
Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a
workstation build/push. Nothing had ever been pushed to the Gitea registry.

- .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main
  builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as
  sha-<commit> + main, reading the digest back from the registry. Only changed images build.
- Each image's own build-image.sh does the build and in-image verification, now engine-neutral
  (DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh.
- Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex
  manager for them. The worker Job template is deliberately untracked (it would loop).
- renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the
  current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION.
- Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in
  Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
gitea-admin added 1 commit 2026-10-30 02:46:26 +00:00
fix(images): name the measured runner blocker; the job container gets no Docker daemon
gate / gate (pull_request) Failing after 1m57s
Details
images / images (pull_request) Failing after 6s
Details
gate / conformance (pull_request) Has been skipped
Details
65a7769f55 
PR #81's first images run failed in 6s: act_runner 0.2.11 forces docker_host '-' for a tcp://
DOCKER_HOST, and gitea/act drops every bind when valid_volumes is empty. Giving jobs the
privileged DinD daemon is node-root for any workflow author, so it is left to the operator;
options are in the handoff entry. The driver's failure now names the cause.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
gitea-admin added 2 commits 2026-10-30 03:27:32 +00:00
feat(images): build on rootless BuildKit over mTLS; verification moves into Containerfile verify stages 310deb0f7d 
Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched.

- platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by
  digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined,
  read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP
  :1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it).
  No Kyverno exception: every applied policy matches app-namespace=true only.
- Each Containerfile names its shipped stage and ends in a `verify` stage holding every former
  `docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for
  the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now.
- images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify
  targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes.
  Fails loudly when the builder secrets are absent or buildkitd is unreachable.
- infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
fix(conformance): workload-mTLS certificates may use a private CA, if they prove they are workload-internal
images / images (pull_request) Failing after 7s
Details
gate / gate (pull_request) Successful in 1m58s
Details
gate / conformance (pull_request) Has been skipped
Details
82235dde30 
R-TLS-01/docs/08 govern externally reachable TLS and ADR-0006 puts workload mTLS out of scope, but
tls-lifecycle-automated required ACME for every Certificate, which platform/buildkit's internal CA
cannot satisfy. Clause (5b) accepts the tls-scope=workload-mtls annotation only when the
certificate proves it: no Gateway serves it, internal names only, namespaced selfSigned/ca issuer
backed by an exempt CA certificate. Also fixes yamllint comma spacing in the buildkitd mounts.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
gitea-admin changed title from feat(images): build, verify and publish the three images in CI; Renovate carries the digests to feat(images): build, verify and publish the three images in CI on rootless BuildKit; Renovate carries the digests 2026-10-30 03:28:06 +00:00
gitea-admin commented 2026-10-30 03:30:11 +00:00
Author
Owner
Copy link
Copy source

CI on 82235dd, read from the job logs:

  • images (run 331): failed as designed. In the real runner it downloaded buildx v0.38.1 and its sha256 check passed. It planned toolbox, agent-runner and conformance-runner (inputs changed), then stopped at the first preflight with: the builder's client certificate is missing: BUILDKIT_CA BUILDKIT_CLIENT_CERT BUILDKIT_CLIENT_KEY not set. Nothing was built, because buildkitd can't exist until this merges and Flux applies platform/buildkit.
  • gate (run 330): failed only on approval-policy, which is expected for sensitive paths. decisions-log, lint, l0, render-golden, l1 and gitea-settings passed. spec-immutability and L2/L3 conformance were skipped, as on every CI run here.

No /approve-override has been posted by me.

CI on `82235dd`, read from the job logs: - **images (run 331): failed as designed.** In the real runner it downloaded `buildx v0.38.1` and its sha256 check passed. It planned toolbox, agent-runner and conformance-runner (inputs changed), then stopped at the first preflight with: `the builder's client certificate is missing: BUILDKIT_CA BUILDKIT_CLIENT_CERT BUILDKIT_CLIENT_KEY not set`. Nothing was built, because buildkitd can't exist until this merges and Flux applies `platform/buildkit`. - **gate (run 330): failed only on `approval-policy`**, which is expected for sensitive paths. `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `gitea-settings` passed. `spec-immutability` and L2/L3 `conformance` were skipped, as on every CI run here. No `/approve-override` has been posted by me.
jakman85 approved these changes 2026-10-30 04:56:04 +00:00
Dismissed
gitea-admin force-pushed feat/image-pipeline from 82235dde30 to 04eb38584a 2026-10-30 05:01:45 +00:00 Compare
gitea-admin dismissed jakman85's review 2026-10-30 05:01:45 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

jakman85 merged commit 0dd57fbaf0 into main 2026-10-30 05:15:16 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
agent
automated-update
blast-radius
breaking
ci-tooling
dependencies
needs-review
own-image
range-widen
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
gitea-admin jakman85 (Keith Jackson) pm-agent renovate-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#81