Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
82235dde30 | ||
|
|
310deb0f7d | ||
|
|
65a7769f55 | ||
|
|
0da34ddbc6 |
No files matched your search
@@ -154,16 +154,13 @@ jobs:
|
||||
# Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers
|
||||
# includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`.
|
||||
# This is the one dependency in the repo whose updater is its own updater.
|
||||
# MUST match the file's real name. #79 renamed it to renovate.json5 and left this line
|
||||
# saying `renovate.json`; every scheduled run from 2026-10-18 then died in ~1s with
|
||||
# ENOENT before Renovate started, and nothing in the gate noticed. ci/lint.sh now
|
||||
# asserts that every `configurationFile:` here names a file that exists.
|
||||
configurationFile: renovate.json5
|
||||
configurationFile: renovate.json
|
||||
token: ${{ secrets.RENOVATE_TOKEN }}
|
||||
# PINNED 2026-10-29. Unset, this action version runs Renovate `39` (action.yml
|
||||
# `renovate-version` default at v41.0.9), i.e. 39.264.0, and renovate.json5 uses
|
||||
# `managerFilePatterns` (>= 40.2.0): under 39.x validation fails with five errors and
|
||||
# the run stops at CONFIG_VALIDATION. #79 validated against `latest`, not this.
|
||||
# PINNED 2026-10-29. Unset, this action version runs `ghcr.io/renovatebot/renovate:39`
|
||||
# (its action.yml default), i.e. 39.264.0 — and renovate.json5 is written for, and
|
||||
# validated against, a release that knows `managerFilePatterns` (>= 40.2.0). Under 39.x
|
||||
# the config fails validation with five errors and the run stops at CONFIG_VALIDATION.
|
||||
# Validate renovate.json5 with THIS tag; see its header for the command.
|
||||
renovate-version: 44.167.0
|
||||
env:
|
||||
# Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
|
||||
|
||||
-15
@@ -27,18 +27,3 @@ fi
|
||||
|
||||
yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${files[@]}"
|
||||
ok "yaml/lint passed (${#files[@]} files)"
|
||||
|
||||
# A workflow's `configurationFile:` must name a file that exists. Renovate's did not after #79
|
||||
# renamed renovate.json -> renovate.json5, and every scheduled run failed with ENOENT for 11 days
|
||||
# while the gate stayed green. Valid YAML, so yamllint cannot see it.
|
||||
missing=0
|
||||
while IFS=: read -r wf _ value; do
|
||||
value="${value#*configurationFile:}"
|
||||
value="$(echo "$value" | sed -E 's/#.*//; s/^[[:space:]"'\'']+//; s/[[:space:]"'\'']+$//')"
|
||||
if [ -n "$value" ] && [ ! -e "$value" ]; then
|
||||
err "${wf}: configurationFile '${value}' does not exist in the repo"
|
||||
missing=1
|
||||
fi
|
||||
done < <(grep -nE '^[[:space:]]*configurationFile:' .gitea/workflows/*.y*ml 2>/dev/null || true)
|
||||
[ "$missing" -eq 0 ] || exit 1
|
||||
ok "workflow configurationFile references resolve"
|
||||
@@ -1,57 +0,0 @@
|
||||
---
|
||||
date: 2026-10-29
|
||||
branch: fix/renovate-config-filename
|
||||
title: Renovate has failed every run since #79 — the workflow still named renovate.json
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**`.gitea/workflows/renovate.yaml`** — `configurationFile: renovate.json` → `renovate.json5`.
|
||||
|
||||
PR #79 (`fix/renovate-invalid-config`) renamed the config to `renovate.json5` and did not update
|
||||
the one line in the workflow that names it. Measured 2026-10-29 from the Gitea Actions log of run
|
||||
322 (task 513, schedule, 08:15 UTC):
|
||||
|
||||
Error: ENOENT: no such file or directory, stat '/workspace/Home-Network/home-network-k3s-impl/renovate.json'
|
||||
|
||||
Runs 315–322 (2026-10-22 → 10-29, every one listed) are all `failure` on the `schedule` event,
|
||||
each failing in about one second. Renovate never started. `RENOVATE_TOKEN` **does** exist as a
|
||||
repository Actions secret (listed by name via the API; value not read), so the token preflight
|
||||
passes and this filename is the only blocker observed.
|
||||
|
||||
**`ci/lint.sh`** — a new assertion after yamllint: every non-comment `configurationFile:` key in
|
||||
`.gitea/workflows/*.y*ml` must name a path that exists. This is valid YAML, so yamllint cannot see
|
||||
it, and the gate stayed green through 11 days of failures.
|
||||
|
||||
**Second blocker, same PR:** the workflow did not pin a Renovate version, so
|
||||
`renovatebot/[email protected]` ran its default, Renovate **39** (`action.yml`:
|
||||
`renovate-version` `default: '39'`, read at that tag). `renovate.json5` uses `managerFilePatterns`,
|
||||
which needs >= 40.2.0. Under 39.264.0 the validator rejects the config with five errors. This was
|
||||
found and reproduced by the image-pipeline worker (PR #81). #79 had validated against `latest`. Pinned
|
||||
`renovate-version: 44.167.0`, the version the config validates clean against per that worker.
|
||||
|
||||
## What I verified
|
||||
|
||||
- The root cause, from the run log quoted above, and that `origin/main` has `renovate.json5` and
|
||||
no `renovate.json` (`git ls-tree`).
|
||||
- The new guard, both ways, run standalone: on this branch it prints
|
||||
`workflow configurationFile references resolve`, rc 0. With the line reverted to `renovate.json`
|
||||
it prints `configurationFile 'renovate.json' does not exist in the repo`, rc 1. My first version
|
||||
also matched its own explanatory comment in the workflow and failed. That is why the grep is
|
||||
anchored to non-comment lines.
|
||||
- Toolbox gate: result and rc are reported in the PR, not predicted here.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The 39.x rejection and the 44.167.0 pass were not reproduced by me.** I confirmed only the
|
||||
action's default (`'39'`). The validation results are the PR #81 worker's.
|
||||
- **Renovate has still never completed a run.** This removes the observed blocker. Others may
|
||||
follow it. The test is a `workflow_dispatch` with `dry_run: true` after merge, and its log is
|
||||
the first evidence that Renovate authenticates, extracts dependencies, and honours `.json5`.
|
||||
- **The `enabledManagers: ["regex"]` vs `custom.regex` question** (#73, #77) is still open. The
|
||||
same dry-run log answers it: look for the regex/custom manager's dependency-extraction count.
|
||||
- **No alert for "Renovate keeps failing".** The guard prevents this specific cause from recurring.
|
||||
Nothing notices a scheduled workflow that is red every day. That gap stays open.
|
||||
- **Sensitive path.** `.gitea/**` and `ci/**` are both in `approval-policy.json`. CI will block
|
||||
on `approval-policy` until the operator posts `/approve-override` as a review. The local gate
|
||||
reports that stage as N/A.
|
||||
@@ -1,81 +0,0 @@
|
||||
---
|
||||
date: 2026-10-30
|
||||
branch: fix/gitea-admit-buildkit
|
||||
title: The registry push 403 is Cilium refusing buildkitd's EGRESS at the Gateway — fixed in platform/buildkit, not in Gitea's ingress
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**`platform/buildkit/networkpolicy.yaml`** — `buildkitd-allow-egress` gains one rule:
|
||||
`toEndpoints {io.kubernetes.pod.namespace: gitea, app.kubernetes.io/name: gitea}` on TCP 3000,
|
||||
with a comment carrying the measurement. `platform/buildkit/README.md`'s policy row says the same.
|
||||
|
||||
**What did NOT change, deliberately:** `catalog/gitea.yaml` `inClusterClients` and
|
||||
`apps/wave-0/gitea.yaml`. The brief proposed admitting `buildkit` there. The evidence below says
|
||||
that is the wrong policy: Gitea's ingress is not what refuses the push, and adding it would only
|
||||
open the direct ClusterIP path from build code to Gitea, which nothing uses.
|
||||
|
||||
### Why: the brief's hypothesis was half right
|
||||
|
||||
Images run 337 failed its push with `HEAD .../blobs/sha256:5855... 403 Forbidden: Access denied`.
|
||||
The PM's hypothesis was that this was Cilium's Envoy denying the request, not Gitea. **That part
|
||||
holds.** The hypothesis also said the denial came from Gitea's *ingress* policy, which does not
|
||||
admit the `buildkit` namespace. **That part is wrong.** The denial is enforced on buildkitd's own
|
||||
*egress*. Since Cilium 1.16.8/1.17.2 (CVE-2025-30162; prod runs `cilium:v1.20.1`), the Gateway's
|
||||
L7 load balancer re-evaluates the source pod's egress policy against the identity of the backend
|
||||
it resolves to. buildkitd's egress allowed `world` :443/:80 and the `ingress` entity. That gets
|
||||
the TCP connection to Envoy, but no rule matches Gitea's backend identity, so Envoy answers
|
||||
`403 Access denied` without forwarding the request. This is the failure commit `68690e5` fixed for
|
||||
every SSO app's hop to Keycloak, and the fix has the same shape. Keycloak's ingress names no app
|
||||
namespaces either.
|
||||
|
||||
## What I verified
|
||||
|
||||
All measurements are read-only, against `k3s-prod`, on 2026-10-30 around 05:27–05:37 UTC.
|
||||
|
||||
- **Gitea never saw the failed push.** Its router log for the previous 12h had 316 `/v2/` lines
|
||||
and **0** that were `toolbox/blobs`, PUT, POST or PATCH. It also had **0** from buildkitd's pod
|
||||
IP (`10.0.0.7`). In the same window it logged the CI job's own `HEAD .../toolbox/manifests/
|
||||
sha-0dd57fbaf002 → 404` from the act-runner pod (`10.0.0.38`). Requests that came through the
|
||||
Gateway reach Gitea and get logged. This one never arrived.
|
||||
- **A reproduction with no credentials.** An unauthenticated `GET https://gitea.home.keithjackson.dev/v2/`
|
||||
(Gateway VIP `192.168.2.20`) gave:
|
||||
- from the buildkitd pod: **`403 Forbidden`**, with no matching line in Gitea's log;
|
||||
- from the act-runner pod's `dind` container: **`401 Unauthorized`**, which is Gitea's normal
|
||||
registry challenge, logged by Gitea as `GET /v2/ for 10.0.0.38:0, 401`.
|
||||
|
||||
No token was involved, so this cannot be a package-permission problem. The bot account and the
|
||||
`REGISTRY_PUSH_TOKEN` were not the cause, and the operator does not need to change them.
|
||||
- **Hubble names the enforcement point.** On `cilium-55jv8`, buildkitd's node:
|
||||
`buildkit/buildkitd-859d67c8f8-x44nh (ingress) -> gateway/cilium-gateway-shared:443 http-request
|
||||
DROPPED (HTTP/1.1 GET https://gitea.home.keithjackson.dev/v2/)`, followed by `http-response
|
||||
FORWARDED (HTTP/1.1 403 0ms)`. In the JSON form, **`traffic_direction: EGRESS`**. Envoy wrote
|
||||
the 403 itself in 0 ms.
|
||||
- **Gitea's ingress does not need the source namespace.** The act-runner pod's namespace
|
||||
(`gitea-actions`) is **not** in Gitea's ingress allows. Its NetworkPolicy
|
||||
`default-deny-ingress-allow-egress` has an `{}` egress rule, so its egress is unrestricted. Its
|
||||
requests through the Gateway still reach Gitea. `cilium-dbg bpf policy get` on Gitea's endpoint
|
||||
(1876 on `cilium-9rk5x`) shows all Gateway traffic admitted as `reserved:ingress 3000/TCP`
|
||||
(391 MB, 1.19 M packets) and no row for `gitea-actions`. The source-identity check therefore
|
||||
happens on the source's egress, not on the destination's ingress.
|
||||
- **The policy gap is visible in the BPF map.** `bpf policy get 83` (buildkitd) has these egress
|
||||
rows and nothing else: `reserved:world` 80/443, `reserved:ingress ANY`, and kube-dns 53. There
|
||||
is no Gitea identity.
|
||||
- The edited YAML parses (three documents, names unchanged).
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **That this rule turns the 403 into a 401/200. UNVERIFIED.** That needs the policy applied, which
|
||||
is a Flux/operator action. My confidence is high but not proven, and it rests on two things. The
|
||||
Hubble verdict is EGRESS. Commit `68690e5` made the identical change for Keycloak and recorded
|
||||
"Verified live: headlamp-ns -> keycloak discovery 403 -> 200".
|
||||
- **The port restriction.** The rule names `3000`, Gitea's container port, which is the upstream
|
||||
port Envoy connects to. The Keycloak precedent has no port. If the 403 survives, the first thing
|
||||
to test is whether L7LB enforcement uses the frontend :443 instead. I expect it does not.
|
||||
- **Run 337's own flows.** The Hubble ring buffer had already rotated past them, and buildkitd had
|
||||
been restarted since (the pod was 18 minutes old). The reproduction above uses the same path, the
|
||||
same pod spec and the same policy, but it is not the original request.
|
||||
- Envoy's access logs in the `cilium-envoy` pods. Hubble's L7 flow already shows the drop, so I
|
||||
did not read them.
|
||||
- The push itself, the Renovate digest flow, and whether the bot team's `Packages=Write` is enough
|
||||
for a first push that creates a new package. These only become testable after this lands.
|
||||
@@ -20,7 +20,7 @@ privileged daemon.** Builds go to this rootless daemon over mTLS instead.
|
||||
| `deployment.yaml` | `moby/buildkit:v0.34.1-rootless`, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The `securityContext` is commented line by line with how each item was measured |
|
||||
| `certificates.yaml` | cert-manager: a self-signed CA used only here, plus the server cert `buildkitd-tls` and the CI client cert `buildkit-client-ci` |
|
||||
| `service.yaml` | ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`, mTLS only. Never on the Gateway |
|
||||
| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and, for the registry push, the Gateway (`ingress` entity) **plus** Gitea's backend identity on :3000 — Cilium's Gateway re-checks this pod's egress against the backend it proxies to, and without that rule answers `403 Access denied` itself (measured 2026-10-30, `decisions/2026-10-30-gitea-admit-buildkit.md`) |
|
||||
| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and the Gateway (`ingress` entity) for the registry push |
|
||||
| `configmap.yaml` | `buildkitd.toml`: cache GC capped at 4 GB, because the cache is an `emptyDir` on node disk |
|
||||
| `versions.env` | The CI side: pinned `buildx` (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource |
|
||||
|
||||
|
||||
@@ -64,20 +64,6 @@ spec:
|
||||
# resolves in-cluster to the Gateway VIP (split-horizon CoreDNS), and Cilium hands that
|
||||
# traffic to the Gateway, whose identity is `ingress` — the same identity gitea's own
|
||||
# ingress allow admits.
|
||||
# * gitea's BACKEND identity, :3000 only: ALSO the push, and the half that `ingress` alone does
|
||||
# not cover. Cilium >= 1.16.8/1.17.2 (CVE-2025-30162; this cluster runs 1.20.1) makes the
|
||||
# Gateway's Envoy re-evaluate THIS pod's egress policy against the identity of the backend it
|
||||
# resolves the request to (enforce_policy_on_l7lb). `toEntities: ingress` gets the TCP
|
||||
# connection to Envoy; without this rule Envoy then refuses the HTTP request itself with
|
||||
# `403 Access denied`, and Gitea never sees it. Measured 2026-10-30 (images run 337 failed
|
||||
# this way): Hubble on buildkitd's node logged `buildkit/buildkitd (ingress) ->
|
||||
# gateway/cilium-gateway-shared:443 http-request DROPPED`, traffic_direction EGRESS, and
|
||||
# Gitea's router log had no entry for it — while the same unauthenticated GET /v2/ from the
|
||||
# act-runner pod (egress unrestricted) reached Gitea and got its normal 401. It is the same
|
||||
# fix catalog/render.py applies for every SSO app's hop to Keycloak (commit 68690e5).
|
||||
# Gitea's INGRESS needs nothing: the request arrives there as `reserved:ingress`, already
|
||||
# admitted. Do not add buildkit to catalog/gitea.yaml's inClusterClients — that would open
|
||||
# the ClusterIP path from build code to Gitea, which nothing here uses.
|
||||
# Nothing else in the cluster: not the API server, not other pods. A build step is code from a
|
||||
# pull request.
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -100,10 +86,3 @@ spec:
|
||||
- { port: "80", protocol: TCP }
|
||||
- toEntities:
|
||||
- ingress
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: gitea
|
||||
app.kubernetes.io/name: gitea
|
||||
toPorts:
|
||||
- ports:
|
||||
- { port: "3000", protocol: TCP }
|
||||
Reference in new issue
Block a user