fix(buildkit): allow buildkitd egress to Gitea so the Gateway stops 403ing the registry push #83

2026-10-30T05:41:22Z

gitea-admin commented

2026-10-30 05:41:22 +00:00

Problem

The first run of the images workflow on main (run 337) built the toolbox image, but the push failed:
HEAD .../toolbox/blobs/sha256:5855... 403 Forbidden: Access denied.

Cause (measured read-only on k3s-prod, 2026-10-30)

The 403 comes from Cilium's Gateway Envoy, not from Gitea. Envoy is enforcing buildkitd's own egress policy, not Gitea's ingress.

Gitea's router log has no entry for the push, and no /v2/ request from buildkitd's IP at all. The CI job's own manifest checks, sent from the act-runner pod through the same Gateway, are logged.
An unauthenticated GET https://gitea.home.keithjackson.dev/v2/ returns 401 from Gitea when sent from act-runner (egress unrestricted). The same request from buildkitd returns 403, and Gitea never logs it. No token was involved, so this is not a package-permission problem.
Hubble logs it as buildkit/buildkitd (ingress) -> gateway/cilium-gateway-shared:443 http-request DROPPED, traffic_direction: EGRESS, with the response a 403 in 0 ms.
Gitea's ingress does not need the source namespace. gitea-actions is not in Gitea's ingress allows, yet its requests through the Gateway arrive. Gitea's BPF policy admits them as reserved:ingress.

Since Cilium 1.16.8/1.17.2 (CVE-2025-30162; prod runs 1.20.1), Gateway L7LB re-checks the source pod's egress against the backend identity. buildkitd's egress allowed only world and the ingress entity. Commit 68690e5 fixed the identical failure for the SSO apps' hop to Keycloak.

Change

platform/buildkit/networkpolicy.yaml: buildkitd-allow-egress adds toEndpoints gitea/app.kubernetes.io/name=gitea on TCP 3000.
platform/buildkit/README.md: the policy row now says the same.
decisions/2026-10-30-gitea-admit-buildkit.md: the handoff entry.

Not changed: catalog/gitea.yaml inClusterClients. Adding buildkit there would open the direct ClusterIP path from build code to Gitea and would not fix this.

Gate

Local toolbox gate, run from the worktree at d2c4bd5: rc 3. The only stage that did not run is gitea-settings (no SETTINGS_API_TOKEN locally). spec-immutability passed, with the spec repo mounted.

Not verified

That the 403 becomes a 401/200 once Flux applies this. That needs the policy live. Proving the push works is an operator step after merge (see the handoff entry).

🤖 Generated with Claude Code

## Problem The first run of the `images` workflow on main (run 337) built the toolbox image, but the push failed: `HEAD .../toolbox/blobs/sha256:5855... 403 Forbidden: Access denied`. ## Cause (measured read-only on k3s-prod, 2026-10-30) The 403 comes from Cilium's Gateway Envoy, not from Gitea. Envoy is enforcing **buildkitd's own egress** policy, not Gitea's ingress. - Gitea's router log has **no** entry for the push, and no `/v2/` request from buildkitd's IP at all. The CI job's own manifest checks, sent from the act-runner pod through the same Gateway, are logged. - An unauthenticated `GET https://gitea.home.keithjackson.dev/v2/` returns **401** from Gitea when sent from act-runner (egress unrestricted). The same request from buildkitd returns **403**, and Gitea never logs it. No token was involved, so this is not a package-permission problem. - Hubble logs it as `buildkit/buildkitd (ingress) -> gateway/cilium-gateway-shared:443 http-request DROPPED`, `traffic_direction: EGRESS`, with the response a `403` in 0 ms. - Gitea's ingress does not need the source namespace. `gitea-actions` is not in Gitea's ingress allows, yet its requests through the Gateway arrive. Gitea's BPF policy admits them as `reserved:ingress`. Since Cilium 1.16.8/1.17.2 (CVE-2025-30162; prod runs 1.20.1), Gateway L7LB re-checks the source pod's egress against the backend identity. buildkitd's egress allowed only `world` and the `ingress` entity. Commit 68690e5 fixed the identical failure for the SSO apps' hop to Keycloak. ## Change - `platform/buildkit/networkpolicy.yaml`: `buildkitd-allow-egress` adds `toEndpoints` gitea/`app.kubernetes.io/name=gitea` on TCP 3000. - `platform/buildkit/README.md`: the policy row now says the same. - `decisions/2026-10-30-gitea-admit-buildkit.md`: the handoff entry. **Not changed:** `catalog/gitea.yaml` `inClusterClients`. Adding buildkit there would open the direct ClusterIP path from build code to Gitea and would not fix this. ## Gate Local toolbox gate, run from the worktree at d2c4bd5: **rc 3**. The only stage that did not run is `gitea-settings` (no `SETTINGS_API_TOKEN` locally). `spec-immutability` passed, with the spec repo mounted. ## Not verified That the 403 becomes a 401/200 once Flux applies this. That needs the policy live. Proving the push works is an operator step after merge (see the handoff entry). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-30 05:41:23 +00:00

fix(buildkit): allow buildkitd egress to Gitea's backend so the Gateway stops 403ing the push

gate / gate (pull_request) Successful in 1m56s

Details

gate / conformance (pull_request) Has been skipped

Details

d2c4bd5194

Images run 337's push failed with "403 Forbidden: Access denied". Measured
read-only: Gitea never logged the request; an unauthenticated GET /v2/ via the
Gateway gets 401 from Gitea when sent from act-runner (egress unrestricted) but
403 from buildkitd; Hubble logs it as an L7 http-request DROPPED with
traffic_direction EGRESS. Cilium >= 1.16.8 (CVE-2025-30162) re-checks the
source pod's egress against the proxied backend identity, and buildkitd's
egress only allowed world and the ingress entity. Same fix as 68690e5 for
Keycloak. Gitea's ingress already admits reserved:ingress and is unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jakman85 merged commit 523cd9118f into main

2026-10-30 05:50:12 +00:00

jakman85 referenced this issue from a commit

2026-10-30 05:50:13 +00:00

Merge pull request 'fix(buildkit): allow buildkitd egress to Gitea so the Gateway stops 403ing the registry push' (#83) from fix/gitea-admit-buildkit into main

gitea-admin referenced this pull request

2026-10-30 06:19:58 +00:00

feat(pm-agent): pull credential + admission rule 7, first published digest pinned, dry-run first start, token scopes fixed #61

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#83