fix(buildkit): allow buildkitd egress to Gitea's backend so the Gateway stops 403ing the push

Images run 337's push failed with "403 Forbidden: Access denied". Measured
read-only: Gitea never logged the request; an unauthenticated GET /v2/ via the
Gateway gets 401 from Gitea when sent from act-runner (egress unrestricted) but
403 from buildkitd; Hubble logs it as an L7 http-request DROPPED with
traffic_direction EGRESS. Cilium >= 1.16.8 (CVE-2025-30162) re-checks the
source pod's egress against the proxied backend identity, and buildkitd's
egress only allowed world and the ingress entity. Same fix as 68690e5 for
Keycloak. Gitea's ingress already admits reserved:ingress and is unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

.gitea/workflows/renovate.yaml

@@ -154,13 +154,16 @@ jobs:
           # Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers
           # includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`.
           # This is the one dependency in the repo whose updater is its own updater.
-          configurationFile: renovate.json
+          # MUST match the file's real name. #79 renamed it to renovate.json5 and left this line
+          # saying `renovate.json`; every scheduled run from 2026-10-18 then died in ~1s with
+          # ENOENT before Renovate started, and nothing in the gate noticed. ci/lint.sh now
+          # asserts that every `configurationFile:` here names a file that exists.
+          configurationFile: renovate.json5
           token: ${{ secrets.RENOVATE_TOKEN }}
-          # PINNED 2026-10-29. Unset, this action version runs `ghcr.io/renovatebot/renovate:39`
-          # (its action.yml default), i.e. 39.264.0 — and renovate.json5 is written for, and
-          # validated against, a release that knows `managerFilePatterns` (>= 40.2.0). Under 39.x
-          # the config fails validation with five errors and the run stops at CONFIG_VALIDATION.
-          # Validate renovate.json5 with THIS tag; see its header for the command.
+          # PINNED 2026-10-29. Unset, this action version runs Renovate `39` (action.yml
+          # `renovate-version` default at v41.0.9), i.e. 39.264.0, and renovate.json5 uses
+          # `managerFilePatterns` (>= 40.2.0): under 39.x validation fails with five errors and
+          # the run stops at CONFIG_VALIDATION. #79 validated against `latest`, not this.
           renovate-version: 44.167.0
         env:
           # Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own

ci/lint.sh

@@ -27,3 +27,18 @@ fi
 
 yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${files[@]}"
 ok "yaml/lint passed (${#files[@]} files)"
+
+# A workflow's `configurationFile:` must name a file that exists. Renovate's did not after #79
+# renamed renovate.json -> renovate.json5, and every scheduled run failed with ENOENT for 11 days
+# while the gate stayed green. Valid YAML, so yamllint cannot see it.
+missing=0
+while IFS=: read -r wf _ value; do
+  value="${value#*configurationFile:}"
+  value="$(echo "$value" | sed -E 's/#.*//; s/^[[:space:]"'\'']+//; s/[[:space:]"'\'']+$//')"
+  if [ -n "$value" ] && [ ! -e "$value" ]; then
+    err "${wf}: configurationFile '${value}' does not exist in the repo"
+    missing=1
+  fi
+done < <(grep -nE '^[[:space:]]*configurationFile:' .gitea/workflows/*.y*ml 2>/dev/null || true)
+[ "$missing" -eq 0 ] || exit 1
+ok "workflow configurationFile references resolve"

decisions/2026-10-29-renovate-config-filename.md

@@ -0,0 +1,57 @@
+---
+date: 2026-10-29
+branch: fix/renovate-config-filename
+title: Renovate has failed every run since #79 — the workflow still named renovate.json
+---
+
+## What changed
+
+**`.gitea/workflows/renovate.yaml`** — `configurationFile: renovate.json` → `renovate.json5`.
+
+PR #79 (`fix/renovate-invalid-config`) renamed the config to `renovate.json5` and did not update
+the one line in the workflow that names it. Measured 2026-10-29 from the Gitea Actions log of run
+322 (task 513, schedule, 08:15 UTC):
+
+    Error: ENOENT: no such file or directory, stat '/workspace/Home-Network/home-network-k3s-impl/renovate.json'
+
+Runs 315–322 (2026-10-22 → 10-29, every one listed) are all `failure` on the `schedule` event,
+each failing in about one second. Renovate never started. `RENOVATE_TOKEN` **does** exist as a
+repository Actions secret (listed by name via the API; value not read), so the token preflight
+passes and this filename is the only blocker observed.
+
+**`ci/lint.sh`** — a new assertion after yamllint: every non-comment `configurationFile:` key in
+`.gitea/workflows/*.y*ml` must name a path that exists. This is valid YAML, so yamllint cannot see
+it, and the gate stayed green through 11 days of failures.
+
+**Second blocker, same PR:** the workflow did not pin a Renovate version, so
+`renovatebot/[email protected]` ran its default, Renovate **39** (`action.yml`:
+`renovate-version` `default: '39'`, read at that tag). `renovate.json5` uses `managerFilePatterns`,
+which needs >= 40.2.0. Under 39.264.0 the validator rejects the config with five errors. This was
+found and reproduced by the image-pipeline worker (PR #81). #79 had validated against `latest`. Pinned
+`renovate-version: 44.167.0`, the version the config validates clean against per that worker.
+
+## What I verified
+
+- The root cause, from the run log quoted above, and that `origin/main` has `renovate.json5` and
+  no `renovate.json` (`git ls-tree`).
+- The new guard, both ways, run standalone: on this branch it prints
+  `workflow configurationFile references resolve`, rc 0. With the line reverted to `renovate.json`
+  it prints `configurationFile 'renovate.json' does not exist in the repo`, rc 1. My first version
+  also matched its own explanatory comment in the workflow and failed. That is why the grep is
+  anchored to non-comment lines.
+- Toolbox gate: result and rc are reported in the PR, not predicted here.
+
+## What I did NOT verify
+
+- **The 39.x rejection and the 44.167.0 pass were not reproduced by me.** I confirmed only the
+  action's default (`'39'`). The validation results are the PR #81 worker's.
+- **Renovate has still never completed a run.** This removes the observed blocker. Others may
+  follow it. The test is a `workflow_dispatch` with `dry_run: true` after merge, and its log is
+  the first evidence that Renovate authenticates, extracts dependencies, and honours `.json5`.
+- **The `enabledManagers: ["regex"]` vs `custom.regex` question** (#73, #77) is still open. The
+  same dry-run log answers it: look for the regex/custom manager's dependency-extraction count.
+- **No alert for "Renovate keeps failing".** The guard prevents this specific cause from recurring.
+  Nothing notices a scheduled workflow that is red every day. That gap stays open.
+- **Sensitive path.** `.gitea/**` and `ci/**` are both in `approval-policy.json`. CI will block
+  on `approval-policy` until the operator posts `/approve-override` as a review. The local gate
+  reports that stage as N/A.

decisions/2026-10-30-gitea-admit-buildkit.md

@@ -0,0 +1,81 @@
+---
+date: 2026-10-30
+branch: fix/gitea-admit-buildkit
+title: The registry push 403 is Cilium refusing buildkitd's EGRESS at the Gateway — fixed in platform/buildkit, not in Gitea's ingress
+---
+
+## What changed
+
+**`platform/buildkit/networkpolicy.yaml`** — `buildkitd-allow-egress` gains one rule:
+`toEndpoints {io.kubernetes.pod.namespace: gitea, app.kubernetes.io/name: gitea}` on TCP 3000,
+with a comment carrying the measurement. `platform/buildkit/README.md`'s policy row says the same.
+
+**What did NOT change, deliberately:** `catalog/gitea.yaml` `inClusterClients` and
+`apps/wave-0/gitea.yaml`. The brief proposed admitting `buildkit` there. The evidence below says
+that is the wrong policy: Gitea's ingress is not what refuses the push, and adding it would only
+open the direct ClusterIP path from build code to Gitea, which nothing uses.
+
+### Why: the brief's hypothesis was half right
+
+Images run 337 failed its push with `HEAD .../blobs/sha256:5855... 403 Forbidden: Access denied`.
+The PM's hypothesis was that this was Cilium's Envoy denying the request, not Gitea. **That part
+holds.** The hypothesis also said the denial came from Gitea's *ingress* policy, which does not
+admit the `buildkit` namespace. **That part is wrong.** The denial is enforced on buildkitd's own
+*egress*. Since Cilium 1.16.8/1.17.2 (CVE-2025-30162; prod runs `cilium:v1.20.1`), the Gateway's
+L7 load balancer re-evaluates the source pod's egress policy against the identity of the backend
+it resolves to. buildkitd's egress allowed `world` :443/:80 and the `ingress` entity. That gets
+the TCP connection to Envoy, but no rule matches Gitea's backend identity, so Envoy answers
+`403 Access denied` without forwarding the request. This is the failure commit `68690e5` fixed for
+every SSO app's hop to Keycloak, and the fix has the same shape. Keycloak's ingress names no app
+namespaces either.
+
+## What I verified
+
+All measurements are read-only, against `k3s-prod`, on 2026-10-30 around 05:27–05:37 UTC.
+
+- **Gitea never saw the failed push.** Its router log for the previous 12h had 316 `/v2/` lines
+  and **0** that were `toolbox/blobs`, PUT, POST or PATCH. It also had **0** from buildkitd's pod
+  IP (`10.0.0.7`). In the same window it logged the CI job's own `HEAD .../toolbox/manifests/
+  sha-0dd57fbaf002 → 404` from the act-runner pod (`10.0.0.38`). Requests that came through the
+  Gateway reach Gitea and get logged. This one never arrived.
+- **A reproduction with no credentials.** An unauthenticated `GET https://gitea.home.keithjackson.dev/v2/`
+  (Gateway VIP `192.168.2.20`) gave:
+  - from the buildkitd pod: **`403 Forbidden`**, with no matching line in Gitea's log;
+  - from the act-runner pod's `dind` container: **`401 Unauthorized`**, which is Gitea's normal
+    registry challenge, logged by Gitea as `GET /v2/ for 10.0.0.38:0, 401`.
+
+  No token was involved, so this cannot be a package-permission problem. The bot account and the
+  `REGISTRY_PUSH_TOKEN` were not the cause, and the operator does not need to change them.
+- **Hubble names the enforcement point.** On `cilium-55jv8`, buildkitd's node:
+  `buildkit/buildkitd-859d67c8f8-x44nh (ingress) -> gateway/cilium-gateway-shared:443 http-request
+  DROPPED (HTTP/1.1 GET https://gitea.home.keithjackson.dev/v2/)`, followed by `http-response
+  FORWARDED (HTTP/1.1 403 0ms)`. In the JSON form, **`traffic_direction: EGRESS`**. Envoy wrote
+  the 403 itself in 0 ms.
+- **Gitea's ingress does not need the source namespace.** The act-runner pod's namespace
+  (`gitea-actions`) is **not** in Gitea's ingress allows. Its NetworkPolicy
+  `default-deny-ingress-allow-egress` has an `{}` egress rule, so its egress is unrestricted. Its
+  requests through the Gateway still reach Gitea. `cilium-dbg bpf policy get` on Gitea's endpoint
+  (1876 on `cilium-9rk5x`) shows all Gateway traffic admitted as `reserved:ingress 3000/TCP`
+  (391 MB, 1.19 M packets) and no row for `gitea-actions`. The source-identity check therefore
+  happens on the source's egress, not on the destination's ingress.
+- **The policy gap is visible in the BPF map.** `bpf policy get 83` (buildkitd) has these egress
+  rows and nothing else: `reserved:world` 80/443, `reserved:ingress ANY`, and kube-dns 53. There
+  is no Gitea identity.
+- The edited YAML parses (three documents, names unchanged).
+
+## What I did NOT verify
+
+- **That this rule turns the 403 into a 401/200. UNVERIFIED.** That needs the policy applied, which
+  is a Flux/operator action. My confidence is high but not proven, and it rests on two things. The
+  Hubble verdict is EGRESS. Commit `68690e5` made the identical change for Keycloak and recorded
+  "Verified live: headlamp-ns -> keycloak discovery 403 -> 200".
+- **The port restriction.** The rule names `3000`, Gitea's container port, which is the upstream
+  port Envoy connects to. The Keycloak precedent has no port. If the 403 survives, the first thing
+  to test is whether L7LB enforcement uses the frontend :443 instead. I expect it does not.
+- **Run 337's own flows.** The Hubble ring buffer had already rotated past them, and buildkitd had
+  been restarted since (the pod was 18 minutes old). The reproduction above uses the same path, the
+  same pod spec and the same policy, but it is not the original request.
+- Envoy's access logs in the `cilium-envoy` pods. Hubble's L7 flow already shows the drop, so I
+  did not read them.
+- The push itself, the Renovate digest flow, and whether the bot team's `Packages=Write` is enough
+  for a first push that creates a new package. These only become testable after this lands.

platform/buildkit/README.md

@@ -20,7 +20,7 @@ privileged daemon.** Builds go to this rootless daemon over mTLS instead.
 | `deployment.yaml` | `moby/buildkit:v0.34.1-rootless`, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The `securityContext` is commented line by line with how each item was measured |
 | `certificates.yaml` | cert-manager: a self-signed CA used only here, plus the server cert `buildkitd-tls` and the CI client cert `buildkit-client-ci` |
 | `service.yaml` | ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`, mTLS only. Never on the Gateway |
-| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and the Gateway (`ingress` entity) for the registry push |
+| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and, for the registry push, the Gateway (`ingress` entity) **plus** Gitea's backend identity on :3000 — Cilium's Gateway re-checks this pod's egress against the backend it proxies to, and without that rule answers `403 Access denied` itself (measured 2026-10-30, `decisions/2026-10-30-gitea-admit-buildkit.md`) |
 | `configmap.yaml` | `buildkitd.toml`: cache GC capped at 4 GB, because the cache is an `emptyDir` on node disk |
 | `versions.env` | The CI side: pinned `buildx` (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource |
 

platform/buildkit/networkpolicy.yaml

@@ -64,6 +64,20 @@ spec:
 #     resolves in-cluster to the Gateway VIP (split-horizon CoreDNS), and Cilium hands that
 #     traffic to the Gateway, whose identity is `ingress` — the same identity gitea's own
 #     ingress allow admits.
+#   * gitea's BACKEND identity, :3000 only: ALSO the push, and the half that `ingress` alone does
+#     not cover. Cilium >= 1.16.8/1.17.2 (CVE-2025-30162; this cluster runs 1.20.1) makes the
+#     Gateway's Envoy re-evaluate THIS pod's egress policy against the identity of the backend it
+#     resolves the request to (enforce_policy_on_l7lb). `toEntities: ingress` gets the TCP
+#     connection to Envoy; without this rule Envoy then refuses the HTTP request itself with
+#     `403 Access denied`, and Gitea never sees it. Measured 2026-10-30 (images run 337 failed
+#     this way): Hubble on buildkitd's node logged `buildkit/buildkitd (ingress) ->
+#     gateway/cilium-gateway-shared:443 http-request DROPPED`, traffic_direction EGRESS, and
+#     Gitea's router log had no entry for it — while the same unauthenticated GET /v2/ from the
+#     act-runner pod (egress unrestricted) reached Gitea and got its normal 401. It is the same
+#     fix catalog/render.py applies for every SSO app's hop to Keycloak (commit 68690e5).
+#     Gitea's INGRESS needs nothing: the request arrives there as `reserved:ingress`, already
+#     admitted. Do not add buildkit to catalog/gitea.yaml's inClusterClients — that would open
+#     the ClusterIP path from build code to Gitea, which nothing here uses.
 # Nothing else in the cluster: not the API server, not other pods. A build step is code from a
 # pull request.
 apiVersion: cilium.io/v2
@@ -86,3 +100,10 @@ spec:
             - { port: "80", protocol: TCP }
     - toEntities:
         - ingress
+    - toEndpoints:
+        - matchLabels:
+            io.kubernetes.pod.namespace: gitea
+            app.kubernetes.io/name: gitea
+      toPorts:
+        - ports:
+            - { port: "3000", protocol: TCP }