home-network-k3s-impl/decisions/2026-10-30-gitea-admit-buildkit.md

date, branch, title

date	branch	title
2026-10-30	fix/gitea-admit-buildkit	The registry push 403 is Cilium refusing buildkitd's EGRESS at the Gateway — fixed in platform/buildkit, not in Gitea's ingress

What changed

platform/buildkit/networkpolicy.yaml — buildkitd-allow-egress gains one rule: toEndpoints {io.kubernetes.pod.namespace: gitea, app.kubernetes.io/name: gitea} on TCP 3000, with a comment carrying the measurement. platform/buildkit/README.md's policy row says the same.

What did NOT change, deliberately: catalog/gitea.yaml inClusterClients and apps/wave-0/gitea.yaml. The brief proposed admitting buildkit there. The evidence below says that is the wrong policy: Gitea's ingress is not what refuses the push, and adding it would only open the direct ClusterIP path from build code to Gitea, which nothing uses.

Why: the brief's hypothesis was half right

Images run 337 failed its push with HEAD .../blobs/sha256:5855... 403 Forbidden: Access denied. The PM's hypothesis was that this was Cilium's Envoy denying the request, not Gitea. That part holds. The hypothesis also said the denial came from Gitea's ingress policy, which does not admit the buildkit namespace. That part is wrong. The denial is enforced on buildkitd's own egress. Since Cilium 1.16.8/1.17.2 (CVE-2025-30162; prod runs cilium:v1.20.1), the Gateway's L7 load balancer re-evaluates the source pod's egress policy against the identity of the backend it resolves to. buildkitd's egress allowed world :443/:80 and the ingress entity. That gets the TCP connection to Envoy, but no rule matches Gitea's backend identity, so Envoy answers 403 Access denied without forwarding the request. This is the failure commit 68690e5 fixed for every SSO app's hop to Keycloak, and the fix has the same shape. Keycloak's ingress names no app namespaces either.

What I verified

All measurements are read-only, against k3s-prod, on 2026-10-30 around 05:27–05:37 UTC.

• Gitea never saw the failed push. Its router log for the previous 12h had 316 /v2/ lines and 0 that were toolbox/blobs, PUT, POST or PATCH. It also had 0 from buildkitd's pod IP (10.0.0.7). In the same window it logged the CI job's own HEAD .../toolbox/manifests/ sha-0dd57fbaf002 → 404 from the act-runner pod (10.0.0.38). Requests that came through the Gateway reach Gitea and get logged. This one never arrived.

• A reproduction with no credentials. An unauthenticated GET https://gitea.home.keithjackson.dev/v2/ (Gateway VIP 192.168.2.20) gave:

  • from the buildkitd pod: 403 Forbidden, with no matching line in Gitea's log;
  • from the act-runner pod's dind container: 401 Unauthorized, which is Gitea's normal registry challenge, logged by Gitea as GET /v2/ for 10.0.0.38:0, 401.

  No token was involved, so this cannot be a package-permission problem. The bot account and the REGISTRY_PUSH_TOKEN were not the cause, and the operator does not need to change them.

• Hubble names the enforcement point. On cilium-55jv8, buildkitd's node: buildkit/buildkitd-859d67c8f8-x44nh (ingress) -> gateway/cilium-gateway-shared:443 http-request DROPPED (HTTP/1.1 GET https://gitea.home.keithjackson.dev/v2/), followed by http-response FORWARDED (HTTP/1.1 403 0ms). In the JSON form, traffic_direction: EGRESS. Envoy wrote the 403 itself in 0 ms.

• Gitea's ingress does not need the source namespace. The act-runner pod's namespace (gitea-actions) is not in Gitea's ingress allows. Its NetworkPolicy default-deny-ingress-allow-egress has an {} egress rule, so its egress is unrestricted. Its requests through the Gateway still reach Gitea. cilium-dbg bpf policy get on Gitea's endpoint (1876 on cilium-9rk5x) shows all Gateway traffic admitted as reserved:ingress 3000/TCP (391 MB, 1.19 M packets) and no row for gitea-actions. The source-identity check therefore happens on the source's egress, not on the destination's ingress.

• The policy gap is visible in the BPF map. bpf policy get 83 (buildkitd) has these egress rows and nothing else: reserved:world 80/443, reserved:ingress ANY, and kube-dns 53. There is no Gitea identity.

• The edited YAML parses (three documents, names unchanged).

What I did NOT verify

• That this rule turns the 403 into a 401/200. UNVERIFIED. That needs the policy applied, which is a Flux/operator action. My confidence is high but not proven, and it rests on two things. The Hubble verdict is EGRESS. Commit 68690e5 made the identical change for Keycloak and recorded "Verified live: headlamp-ns -> keycloak discovery 403 -> 200".
• The port restriction. The rule names 3000, Gitea's container port, which is the upstream port Envoy connects to. The Keycloak precedent has no port. If the 403 survives, the first thing to test is whether L7LB enforcement uses the frontend :443 instead. I expect it does not.
• Run 337's own flows. The Hubble ring buffer had already rotated past them, and buildkitd had been restarted since (the pod was 18 minutes old). The reproduction above uses the same path, the same pod spec and the same policy, but it is not the original request.
• Envoy's access logs in the cilium-envoy pods. Hubble's L7 flow already shows the drop, so I did not read them.
• The push itself, the Renovate digest flow, and whether the bot team's Packages=Write is enough for a first push that creates a new package. These only become testable after this lands.