diff --git a/decisions/2026-10-30-gitea-admit-buildkit.md b/decisions/2026-10-30-gitea-admit-buildkit.md new file mode 100644 index 0000000..8e1e405 --- /dev/null +++ b/decisions/2026-10-30-gitea-admit-buildkit.md @@ -0,0 +1,81 @@ +--- +date: 2026-10-30 +branch: fix/gitea-admit-buildkit +title: The registry push 403 is Cilium refusing buildkitd's EGRESS at the Gateway — fixed in platform/buildkit, not in Gitea's ingress +--- + +## What changed + +**`platform/buildkit/networkpolicy.yaml`** — `buildkitd-allow-egress` gains one rule: +`toEndpoints {io.kubernetes.pod.namespace: gitea, app.kubernetes.io/name: gitea}` on TCP 3000, +with a comment carrying the measurement. `platform/buildkit/README.md`'s policy row says the same. + +**What did NOT change, deliberately:** `catalog/gitea.yaml` `inClusterClients` and +`apps/wave-0/gitea.yaml`. The brief proposed admitting `buildkit` there. The evidence below says +that is the wrong policy: Gitea's ingress is not what refuses the push, and adding it would only +open the direct ClusterIP path from build code to Gitea, which nothing uses. + +### Why: the brief's hypothesis was half right + +Images run 337 failed its push with `HEAD .../blobs/sha256:5855... 403 Forbidden: Access denied`. +The PM's hypothesis was that this was Cilium's Envoy denying the request, not Gitea. **That part +holds.** The hypothesis also said the denial came from Gitea's *ingress* policy, which does not +admit the `buildkit` namespace. **That part is wrong.** The denial is enforced on buildkitd's own +*egress*. Since Cilium 1.16.8/1.17.2 (CVE-2025-30162; prod runs `cilium:v1.20.1`), the Gateway's +L7 load balancer re-evaluates the source pod's egress policy against the identity of the backend +it resolves to. buildkitd's egress allowed `world` :443/:80 and the `ingress` entity. That gets +the TCP connection to Envoy, but no rule matches Gitea's backend identity, so Envoy answers +`403 Access denied` without forwarding the request. This is the failure commit `68690e5` fixed for +every SSO app's hop to Keycloak, and the fix has the same shape. Keycloak's ingress names no app +namespaces either. + +## What I verified + +All measurements are read-only, against `k3s-prod`, on 2026-10-30 around 05:27–05:37 UTC. + +- **Gitea never saw the failed push.** Its router log for the previous 12h had 316 `/v2/` lines + and **0** that were `toolbox/blobs`, PUT, POST or PATCH. It also had **0** from buildkitd's pod + IP (`10.0.0.7`). In the same window it logged the CI job's own `HEAD .../toolbox/manifests/ + sha-0dd57fbaf002 → 404` from the act-runner pod (`10.0.0.38`). Requests that came through the + Gateway reach Gitea and get logged. This one never arrived. +- **A reproduction with no credentials.** An unauthenticated `GET https://gitea.home.keithjackson.dev/v2/` + (Gateway VIP `192.168.2.20`) gave: + - from the buildkitd pod: **`403 Forbidden`**, with no matching line in Gitea's log; + - from the act-runner pod's `dind` container: **`401 Unauthorized`**, which is Gitea's normal + registry challenge, logged by Gitea as `GET /v2/ for 10.0.0.38:0, 401`. + + No token was involved, so this cannot be a package-permission problem. The bot account and the + `REGISTRY_PUSH_TOKEN` were not the cause, and the operator does not need to change them. +- **Hubble names the enforcement point.** On `cilium-55jv8`, buildkitd's node: + `buildkit/buildkitd-859d67c8f8-x44nh (ingress) -> gateway/cilium-gateway-shared:443 http-request + DROPPED (HTTP/1.1 GET https://gitea.home.keithjackson.dev/v2/)`, followed by `http-response + FORWARDED (HTTP/1.1 403 0ms)`. In the JSON form, **`traffic_direction: EGRESS`**. Envoy wrote + the 403 itself in 0 ms. +- **Gitea's ingress does not need the source namespace.** The act-runner pod's namespace + (`gitea-actions`) is **not** in Gitea's ingress allows. Its NetworkPolicy + `default-deny-ingress-allow-egress` has an `{}` egress rule, so its egress is unrestricted. Its + requests through the Gateway still reach Gitea. `cilium-dbg bpf policy get` on Gitea's endpoint + (1876 on `cilium-9rk5x`) shows all Gateway traffic admitted as `reserved:ingress 3000/TCP` + (391 MB, 1.19 M packets) and no row for `gitea-actions`. The source-identity check therefore + happens on the source's egress, not on the destination's ingress. +- **The policy gap is visible in the BPF map.** `bpf policy get 83` (buildkitd) has these egress + rows and nothing else: `reserved:world` 80/443, `reserved:ingress ANY`, and kube-dns 53. There + is no Gitea identity. +- The edited YAML parses (three documents, names unchanged). + +## What I did NOT verify + +- **That this rule turns the 403 into a 401/200. UNVERIFIED.** That needs the policy applied, which + is a Flux/operator action. My confidence is high but not proven, and it rests on two things. The + Hubble verdict is EGRESS. Commit `68690e5` made the identical change for Keycloak and recorded + "Verified live: headlamp-ns -> keycloak discovery 403 -> 200". +- **The port restriction.** The rule names `3000`, Gitea's container port, which is the upstream + port Envoy connects to. The Keycloak precedent has no port. If the 403 survives, the first thing + to test is whether L7LB enforcement uses the frontend :443 instead. I expect it does not. +- **Run 337's own flows.** The Hubble ring buffer had already rotated past them, and buildkitd had + been restarted since (the pod was 18 minutes old). The reproduction above uses the same path, the + same pod spec and the same policy, but it is not the original request. +- Envoy's access logs in the `cilium-envoy` pods. Hubble's L7 flow already shows the drop, so I + did not read them. +- The push itself, the Renovate digest flow, and whether the bot team's `Packages=Write` is enough + for a first push that creates a new package. These only become testable after this lands. diff --git a/platform/buildkit/README.md b/platform/buildkit/README.md index ed49634..4e4bf32 100644 --- a/platform/buildkit/README.md +++ b/platform/buildkit/README.md @@ -20,7 +20,7 @@ privileged daemon.** Builds go to this rootless daemon over mTLS instead. | `deployment.yaml` | `moby/buildkit:v0.34.1-rootless`, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The `securityContext` is commented line by line with how each item was measured | | `certificates.yaml` | cert-manager: a self-signed CA used only here, plus the server cert `buildkitd-tls` and the CI client cert `buildkit-client-ci` | | `service.yaml` | ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`, mTLS only. Never on the Gateway | -| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and the Gateway (`ingress` entity) for the registry push | +| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and, for the registry push, the Gateway (`ingress` entity) **plus** Gitea's backend identity on :3000 — Cilium's Gateway re-checks this pod's egress against the backend it proxies to, and without that rule answers `403 Access denied` itself (measured 2026-10-30, `decisions/2026-10-30-gitea-admit-buildkit.md`) | | `configmap.yaml` | `buildkitd.toml`: cache GC capped at 4 GB, because the cache is an `emptyDir` on node disk | | `versions.env` | The CI side: pinned `buildx` (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource | diff --git a/platform/buildkit/networkpolicy.yaml b/platform/buildkit/networkpolicy.yaml index 1ec331d..d8155c5 100644 --- a/platform/buildkit/networkpolicy.yaml +++ b/platform/buildkit/networkpolicy.yaml @@ -64,6 +64,20 @@ spec: # resolves in-cluster to the Gateway VIP (split-horizon CoreDNS), and Cilium hands that # traffic to the Gateway, whose identity is `ingress` — the same identity gitea's own # ingress allow admits. +# * gitea's BACKEND identity, :3000 only: ALSO the push, and the half that `ingress` alone does +# not cover. Cilium >= 1.16.8/1.17.2 (CVE-2025-30162; this cluster runs 1.20.1) makes the +# Gateway's Envoy re-evaluate THIS pod's egress policy against the identity of the backend it +# resolves the request to (enforce_policy_on_l7lb). `toEntities: ingress` gets the TCP +# connection to Envoy; without this rule Envoy then refuses the HTTP request itself with +# `403 Access denied`, and Gitea never sees it. Measured 2026-10-30 (images run 337 failed +# this way): Hubble on buildkitd's node logged `buildkit/buildkitd (ingress) -> +# gateway/cilium-gateway-shared:443 http-request DROPPED`, traffic_direction EGRESS, and +# Gitea's router log had no entry for it — while the same unauthenticated GET /v2/ from the +# act-runner pod (egress unrestricted) reached Gitea and got its normal 401. It is the same +# fix catalog/render.py applies for every SSO app's hop to Keycloak (commit 68690e5). +# Gitea's INGRESS needs nothing: the request arrives there as `reserved:ingress`, already +# admitted. Do not add buildkit to catalog/gitea.yaml's inClusterClients — that would open +# the ClusterIP path from build code to Gitea, which nothing here uses. # Nothing else in the cluster: not the API server, not other pods. A build step is code from a # pull request. apiVersion: cilium.io/v2 @@ -86,3 +100,10 @@ spec: - { port: "80", protocol: TCP } - toEntities: - ingress + - toEndpoints: + - matchLabels: + io.kubernetes.pod.namespace: gitea + app.kubernetes.io/name: gitea + toPorts: + - ports: + - { port: "3000", protocol: TCP }