fix(buildkit): allow buildkitd egress to Gitea's backend so the Gateway stops 403ing the push
gate / gate (pull_request) Successful in 1m56s
gate / conformance (pull_request) Has been skipped

Images run 337's push failed with "403 Forbidden: Access denied". Measured
read-only: Gitea never logged the request; an unauthenticated GET /v2/ via the
Gateway gets 401 from Gitea when sent from act-runner (egress unrestricted) but
403 from buildkitd; Hubble logs it as an L7 http-request DROPPED with
traffic_direction EGRESS. Cilium >= 1.16.8 (CVE-2025-30162) re-checks the
source pod's egress against the proxied backend identity, and buildkitd's
egress only allowed world and the ingress entity. Same fix as 68690e5 for
Keycloak. Gitea's ingress already admits reserved:ingress and is unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed 2026-10-30 00:38:54 -05:00
1 parent 0dd57fbaf0
commit d2c4bd5194
3 files changed
+103 -1

No files matched your search

@@ -0,0 +1,81 @@
---
date: 2026-10-30
branch: fix/gitea-admit-buildkit
title: The registry push 403 is Cilium refusing buildkitd's EGRESS at the Gateway — fixed in platform/buildkit, not in Gitea's ingress
---
## What changed
**`platform/buildkit/networkpolicy.yaml`** — `buildkitd-allow-egress` gains one rule:
`toEndpoints {io.kubernetes.pod.namespace: gitea, app.kubernetes.io/name: gitea}` on TCP 3000,
with a comment carrying the measurement. `platform/buildkit/README.md`'s policy row says the same.
**What did NOT change, deliberately:** `catalog/gitea.yaml` `inClusterClients` and
`apps/wave-0/gitea.yaml`. The brief proposed admitting `buildkit` there. The evidence below says
that is the wrong policy: Gitea's ingress is not what refuses the push, and adding it would only
open the direct ClusterIP path from build code to Gitea, which nothing uses.
### Why: the brief's hypothesis was half right
Images run 337 failed its push with `HEAD .../blobs/sha256:5855... 403 Forbidden: Access denied`.
The PM's hypothesis was that this was Cilium's Envoy denying the request, not Gitea. **That part
holds.** The hypothesis also said the denial came from Gitea's *ingress* policy, which does not
admit the `buildkit` namespace. **That part is wrong.** The denial is enforced on buildkitd's own
*egress*. Since Cilium 1.16.8/1.17.2 (CVE-2025-30162; prod runs `cilium:v1.20.1`), the Gateway's
L7 load balancer re-evaluates the source pod's egress policy against the identity of the backend
it resolves to. buildkitd's egress allowed `world` :443/:80 and the `ingress` entity. That gets
the TCP connection to Envoy, but no rule matches Gitea's backend identity, so Envoy answers
`403 Access denied` without forwarding the request. This is the failure commit `68690e5` fixed for
every SSO app's hop to Keycloak, and the fix has the same shape. Keycloak's ingress names no app
namespaces either.
## What I verified
All measurements are read-only, against `k3s-prod`, on 2026-10-30 around 05:27–05:37 UTC.
- **Gitea never saw the failed push.** Its router log for the previous 12h had 316 `/v2/` lines
and **0** that were `toolbox/blobs`, PUT, POST or PATCH. It also had **0** from buildkitd's pod
IP (`10.0.0.7`). In the same window it logged the CI job's own `HEAD .../toolbox/manifests/
sha-0dd57fbaf002 → 404` from the act-runner pod (`10.0.0.38`). Requests that came through the
Gateway reach Gitea and get logged. This one never arrived.
- **A reproduction with no credentials.** An unauthenticated `GET https://gitea.home.keithjackson.dev/v2/`
(Gateway VIP `192.168.2.20`) gave:
- from the buildkitd pod: **`403 Forbidden`**, with no matching line in Gitea's log;
- from the act-runner pod's `dind` container: **`401 Unauthorized`**, which is Gitea's normal
registry challenge, logged by Gitea as `GET /v2/ for 10.0.0.38:0, 401`.
No token was involved, so this cannot be a package-permission problem. The bot account and the
`REGISTRY_PUSH_TOKEN` were not the cause, and the operator does not need to change them.
- **Hubble names the enforcement point.** On `cilium-55jv8`, buildkitd's node:
`buildkit/buildkitd-859d67c8f8-x44nh (ingress) -> gateway/cilium-gateway-shared:443 http-request
DROPPED (HTTP/1.1 GET https://gitea.home.keithjackson.dev/v2/)`, followed by `http-response
FORWARDED (HTTP/1.1 403 0ms)`. In the JSON form, **`traffic_direction: EGRESS`**. Envoy wrote
the 403 itself in 0 ms.
- **Gitea's ingress does not need the source namespace.** The act-runner pod's namespace
(`gitea-actions`) is **not** in Gitea's ingress allows. Its NetworkPolicy
`default-deny-ingress-allow-egress` has an `{}` egress rule, so its egress is unrestricted. Its
requests through the Gateway still reach Gitea. `cilium-dbg bpf policy get` on Gitea's endpoint
(1876 on `cilium-9rk5x`) shows all Gateway traffic admitted as `reserved:ingress 3000/TCP`
(391 MB, 1.19 M packets) and no row for `gitea-actions`. The source-identity check therefore
happens on the source's egress, not on the destination's ingress.
- **The policy gap is visible in the BPF map.** `bpf policy get 83` (buildkitd) has these egress
rows and nothing else: `reserved:world` 80/443, `reserved:ingress ANY`, and kube-dns 53. There
is no Gitea identity.
- The edited YAML parses (three documents, names unchanged).
## What I did NOT verify
- **That this rule turns the 403 into a 401/200. UNVERIFIED.** That needs the policy applied, which
is a Flux/operator action. My confidence is high but not proven, and it rests on two things. The
Hubble verdict is EGRESS. Commit `68690e5` made the identical change for Keycloak and recorded
"Verified live: headlamp-ns -> keycloak discovery 403 -> 200".
- **The port restriction.** The rule names `3000`, Gitea's container port, which is the upstream
port Envoy connects to. The Keycloak precedent has no port. If the 403 survives, the first thing
to test is whether L7LB enforcement uses the frontend :443 instead. I expect it does not.
- **Run 337's own flows.** The Hubble ring buffer had already rotated past them, and buildkitd had
been restarted since (the pod was 18 minutes old). The reproduction above uses the same path, the
same pod spec and the same policy, but it is not the original request.
- Envoy's access logs in the `cilium-envoy` pods. Hubble's L7 flow already shows the drop, so I
did not read them.
- The push itself, the Renovate digest flow, and whether the bot team's `Packages=Write` is enough
for a first push that creates a new package. These only become testable after this lands.