decisions/2026-10-30-gitea-admit-buildkit.md

@@ -0,0 +1,81 @@
+---
+date: 2026-10-30
+branch: fix/gitea-admit-buildkit
+title: The registry push 403 is Cilium refusing buildkitd's EGRESS at the Gateway — fixed in platform/buildkit, not in Gitea's ingress
+---
+
+## What changed
+
+**`platform/buildkit/networkpolicy.yaml`** — `buildkitd-allow-egress` gains one rule:
+`toEndpoints {io.kubernetes.pod.namespace: gitea, app.kubernetes.io/name: gitea}` on TCP 3000,
+with a comment carrying the measurement. `platform/buildkit/README.md`'s policy row says the same.
+
+**What did NOT change, deliberately:** `catalog/gitea.yaml` `inClusterClients` and
+`apps/wave-0/gitea.yaml`. The brief proposed admitting `buildkit` there. The evidence below says
+that is the wrong policy: Gitea's ingress is not what refuses the push, and adding it would only
+open the direct ClusterIP path from build code to Gitea, which nothing uses.
+
+### Why: the brief's hypothesis was half right
+
+Images run 337 failed its push with `HEAD .../blobs/sha256:5855... 403 Forbidden: Access denied`.
+The PM's hypothesis was that this was Cilium's Envoy denying the request, not Gitea. **That part
+holds.** The hypothesis also said the denial came from Gitea's *ingress* policy, which does not
+admit the `buildkit` namespace. **That part is wrong.** The denial is enforced on buildkitd's own
+*egress*. Since Cilium 1.16.8/1.17.2 (CVE-2025-30162; prod runs `cilium:v1.20.1`), the Gateway's
+L7 load balancer re-evaluates the source pod's egress policy against the identity of the backend
+it resolves to. buildkitd's egress allowed `world` :443/:80 and the `ingress` entity. That gets
+the TCP connection to Envoy, but no rule matches Gitea's backend identity, so Envoy answers
+`403 Access denied` without forwarding the request. This is the failure commit `68690e5` fixed for
+every SSO app's hop to Keycloak, and the fix has the same shape. Keycloak's ingress names no app
+namespaces either.
+
+## What I verified
+
+All measurements are read-only, against `k3s-prod`, on 2026-10-30 around 05:27–05:37 UTC.
+
+- **Gitea never saw the failed push.** Its router log for the previous 12h had 316 `/v2/` lines
+  and **0** that were `toolbox/blobs`, PUT, POST or PATCH. It also had **0** from buildkitd's pod
+  IP (`10.0.0.7`). In the same window it logged the CI job's own `HEAD .../toolbox/manifests/
+  sha-0dd57fbaf002 → 404` from the act-runner pod (`10.0.0.38`). Requests that came through the
+  Gateway reach Gitea and get logged. This one never arrived.
+- **A reproduction with no credentials.** An unauthenticated `GET https://gitea.home.keithjackson.dev/v2/`
+  (Gateway VIP `192.168.2.20`) gave:
+  - from the buildkitd pod: **`403 Forbidden`**, with no matching line in Gitea's log;
+  - from the act-runner pod's `dind` container: **`401 Unauthorized`**, which is Gitea's normal
+    registry challenge, logged by Gitea as `GET /v2/ for 10.0.0.38:0, 401`.
+
+  No token was involved, so this cannot be a package-permission problem. The bot account and the
+  `REGISTRY_PUSH_TOKEN` were not the cause, and the operator does not need to change them.
+- **Hubble names the enforcement point.** On `cilium-55jv8`, buildkitd's node:
+  `buildkit/buildkitd-859d67c8f8-x44nh (ingress) -> gateway/cilium-gateway-shared:443 http-request
+  DROPPED (HTTP/1.1 GET https://gitea.home.keithjackson.dev/v2/)`, followed by `http-response
+  FORWARDED (HTTP/1.1 403 0ms)`. In the JSON form, **`traffic_direction: EGRESS`**. Envoy wrote
+  the 403 itself in 0 ms.
+- **Gitea's ingress does not need the source namespace.** The act-runner pod's namespace
+  (`gitea-actions`) is **not** in Gitea's ingress allows. Its NetworkPolicy
+  `default-deny-ingress-allow-egress` has an `{}` egress rule, so its egress is unrestricted. Its
+  requests through the Gateway still reach Gitea. `cilium-dbg bpf policy get` on Gitea's endpoint
+  (1876 on `cilium-9rk5x`) shows all Gateway traffic admitted as `reserved:ingress 3000/TCP`
+  (391 MB, 1.19 M packets) and no row for `gitea-actions`. The source-identity check therefore
+  happens on the source's egress, not on the destination's ingress.
+- **The policy gap is visible in the BPF map.** `bpf policy get 83` (buildkitd) has these egress
+  rows and nothing else: `reserved:world` 80/443, `reserved:ingress ANY`, and kube-dns 53. There
+  is no Gitea identity.
+- The edited YAML parses (three documents, names unchanged).
+
+## What I did NOT verify
+
+- **That this rule turns the 403 into a 401/200. UNVERIFIED.** That needs the policy applied, which
+  is a Flux/operator action. My confidence is high but not proven, and it rests on two things. The
+  Hubble verdict is EGRESS. Commit `68690e5` made the identical change for Keycloak and recorded
+  "Verified live: headlamp-ns -> keycloak discovery 403 -> 200".
+- **The port restriction.** The rule names `3000`, Gitea's container port, which is the upstream
+  port Envoy connects to. The Keycloak precedent has no port. If the 403 survives, the first thing
+  to test is whether L7LB enforcement uses the frontend :443 instead. I expect it does not.
+- **Run 337's own flows.** The Hubble ring buffer had already rotated past them, and buildkitd had
+  been restarted since (the pod was 18 minutes old). The reproduction above uses the same path, the
+  same pod spec and the same policy, but it is not the original request.
+- Envoy's access logs in the `cilium-envoy` pods. Hubble's L7 flow already shows the drop, so I
+  did not read them.
+- The push itself, the Renovate digest flow, and whether the bot team's `Packages=Write` is enough
+  for a first push that creates a new package. These only become testable after this lands.

platform/buildkit/README.md

@@ -20,7 +20,7 @@ privileged daemon.** Builds go to this rootless daemon over mTLS instead.
 | `deployment.yaml` | `moby/buildkit:v0.34.1-rootless`, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The `securityContext` is commented line by line with how each item was measured |
 | `certificates.yaml` | cert-manager: a self-signed CA used only here, plus the server cert `buildkitd-tls` and the CI client cert `buildkit-client-ci` |
 | `service.yaml` | ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`, mTLS only. Never on the Gateway |
-| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and the Gateway (`ingress` entity) for the registry push |
+| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and, for the registry push, the Gateway (`ingress` entity) **plus** Gitea's backend identity on :3000 — Cilium's Gateway re-checks this pod's egress against the backend it proxies to, and without that rule answers `403 Access denied` itself (measured 2026-10-30, `decisions/2026-10-30-gitea-admit-buildkit.md`) |
 | `configmap.yaml` | `buildkitd.toml`: cache GC capped at 4 GB, because the cache is an `emptyDir` on node disk |
 | `versions.env` | The CI side: pinned `buildx` (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource |
 

platform/buildkit/networkpolicy.yaml

@@ -64,6 +64,20 @@ spec:
 #     resolves in-cluster to the Gateway VIP (split-horizon CoreDNS), and Cilium hands that
 #     traffic to the Gateway, whose identity is `ingress` — the same identity gitea's own
 #     ingress allow admits.
+#   * gitea's BACKEND identity, :3000 only: ALSO the push, and the half that `ingress` alone does
+#     not cover. Cilium >= 1.16.8/1.17.2 (CVE-2025-30162; this cluster runs 1.20.1) makes the
+#     Gateway's Envoy re-evaluate THIS pod's egress policy against the identity of the backend it
+#     resolves the request to (enforce_policy_on_l7lb). `toEntities: ingress` gets the TCP
+#     connection to Envoy; without this rule Envoy then refuses the HTTP request itself with
+#     `403 Access denied`, and Gitea never sees it. Measured 2026-10-30 (images run 337 failed
+#     this way): Hubble on buildkitd's node logged `buildkit/buildkitd (ingress) ->
+#     gateway/cilium-gateway-shared:443 http-request DROPPED`, traffic_direction EGRESS, and
+#     Gitea's router log had no entry for it — while the same unauthenticated GET /v2/ from the
+#     act-runner pod (egress unrestricted) reached Gitea and got its normal 401. It is the same
+#     fix catalog/render.py applies for every SSO app's hop to Keycloak (commit 68690e5).
+#     Gitea's INGRESS needs nothing: the request arrives there as `reserved:ingress`, already
+#     admitted. Do not add buildkit to catalog/gitea.yaml's inClusterClients — that would open
+#     the ClusterIP path from build code to Gitea, which nothing here uses.
 # Nothing else in the cluster: not the API server, not other pods. A build step is code from a
 # pull request.
 apiVersion: cilium.io/v2
@@ -86,3 +100,10 @@ spec:
             - { port: "80", protocol: TCP }
     - toEntities:
         - ingress
+    - toEndpoints:
+        - matchLabels:
+            io.kubernetes.pod.namespace: gitea
+            app.kubernetes.io/name: gitea
+      toPorts:
+        - ports:
+            - { port: "3000", protocol: TCP }