From b548224db8876fd0b8ae70a3a0a6dda6293f1645 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 8 Nov 2026 00:09:08 -0600 Subject: [PATCH] feat(gitea-actions): act_runner capacity 2, dind sized for two concurrent jobs The in-cluster runner ran at the default capacity 1 with no config file, so gate / images / renovate runs on different refs queued behind each other (29-759s waits; run 472 waited 759s behind a renovate burst). - ConfigMap act-runner-config with ONLY runner.capacity: 2, selected via CONFIG_FILE (honoured by the v0.6.1 image's run.sh). v0.6.1 LoadDefault() applies the same defaults to a partial file as to no file; the generate-config example is deliberately not used (its labels would override the registered ones, force_pull differs from the default). - Pod-template config-revision annotation so ConfigMap-only edits can roll it. - dind: cpu request 100m -> 2 (no limit), memory 256Mi/2Gi -> 1Gi/4Gi. Measured at capacity 1: peak 0.69 core, 447Mi working set (30d), 0 OOMs. - runner: memory 128Mi/512Mi -> 256Mi/1Gi (peak 314Mi at capacity 1). apps/wave-1/gitea-actions.yaml is a hand-owned EXTRA_UNITS file (the renderer only places it in a wave), so it is the source, not generated output. Co-Authored-By: Claude Opus 5.5 --- apps/wave-1/gitea-actions.yaml | 74 ++++++++++- decisions/2026-11-08-runner-capacity-2.md | 155 ++++++++++++++++++++++ 2 files changed, 224 insertions(+), 5 deletions(-) create mode 100644 decisions/2026-11-08-runner-capacity-2.md diff --git a/apps/wave-1/gitea-actions.yaml b/apps/wave-1/gitea-actions.yaml index 1577598..6887105 100644 --- a/apps/wave-1/gitea-actions.yaml +++ b/apps/wave-1/gitea-actions.yaml @@ -77,6 +77,42 @@ spec: requests: storage: 1Gi --- +# act_runner config file — DELTAS ONLY. act_runner v0.6.1 LoadDefault() +# (internal/pkg/config/config.go) unmarshals this file onto a ZERO Config and then fills every +# unset field with the same default it uses when there is no file at all, so a file holding +# only `runner.capacity` changes exactly that and nothing else. Do NOT paste in the output of +# `act_runner generate-config`: that example is NOT the defaults. Its `runner.labels` would +# OVERRIDE the registered labels in /data/.runner (daemon.go: config labels win when non-empty) +# and repoint ubuntu-latest at a different job image; its `force_pull: true` differs from the +# no-file default (false); and its sample `envs` would be injected into every job. +# +# Using a file also makes act_runner IGNORE the legacy GITEA_RUNNER_CAPACITY / _FILE / +# _ENVIRON / _ENV_FILE / GITEA_DEBUG / GITEA_TRACE env vars (deprecated.go). None of them is +# set on this Deployment; the GITEA_INSTANCE_URL / _REGISTRATION_TOKEN / _NAME / _LABELS env +# below is consumed by the image's run.sh for registration, not by that legacy path. +# +# A ConfigMap edit alone does NOT restart the runner (it reads the file once at start). If a +# future change touches ONLY this ConfigMap, also bump the Deployment's pod template (e.g. the +# config-revision annotation below) or the new value stays inert until the next restart. +apiVersion: v1 +kind: ConfigMap +metadata: + name: act-runner-config + namespace: gitea-actions + labels: + app.kubernetes.io/name: act-runner +data: + config.yaml: | + runner: + # Two concurrent jobs (default 1). Measured 2026-11-08: queue wait ~1s normally, but + # 29-759s whenever gate / images / renovate runs collide (run 472 waited 759s behind a + # renovate burst). Both jobs share the ONE dind daemon below; act_runner names each job's + # containers GITEA-ACTIONS-TASK- and creates a network per job, so they do not + # collide. Workflow `concurrency:` groups are enforced by Gitea server-side and are + # unaffected: gate.yml's gate- group still cancels a superseded run on the SAME ref, + # and renovate's group still serialises renovate. The dind resources are sized for TWO jobs. + capacity: 2 +--- apiVersion: apps/v1 kind: Deployment metadata: @@ -95,6 +131,10 @@ spec: metadata: labels: app.kubernetes.io/name: act-runner + annotations: + # Bump when act-runner-config changes without any other pod-template change, so the + # Deployment rolls and act_runner re-reads its config (see the ConfigMap header). + homelab.keithjackson.dev/act-runner-config-revision: "1" # capacity 2 spec: # CI holds no cluster credentials — do not mount a SA token into the runner or # (via the shared netns) the job containers (R-SEC-02, docs/14). @@ -146,12 +186,23 @@ spec: mountPath: /certs - name: dind-storage mountPath: /var/lib/docker + # Sized for runner.capacity: 2. Every job's containers run UNDER this daemon, so this + # container's cgroup carries the whole CI workload. Measured 2026-11-08 (Prometheus, + # capacity 1): peak CPU 0.69 core (1m rate, 30d), peak working set 447Mi (30d), 0 OOMs. + # cpu request 2: two jobs at today's peak is ~1.4 core, and the gate is being + # parallelised, so per-job demand rises. The request is what the CFS share is + # weighted by under contention (100m was a 1/20 share); NO cpu limit, deliberately, + # so an idle node's spare cores are still usable. k3s-prod-2 (8 CPU) had 2945m + # requested; +1.9 core = ~4.85 (61%), and prod-1/prod-3 fit too if it reschedules. + # memory request 1Gi / limit 4Gi: two jobs at today's peak is ~0.9Gi. An OOM kill here + # takes down dockerd and with it BOTH running jobs, so the limit carries ~4x headroom + # over that for the parallelised gate. Nodes have ~31Gi allocatable. resources: requests: - cpu: 100m - memory: 256Mi + cpu: "2" + memory: 1Gi limits: - memory: 2Gi + memory: 4Gi startupProbe: exec: command: @@ -195,23 +246,36 @@ spec: value: /certs/client - name: DOCKER_TLS_VERIFY value: "1" + # The image's run.sh passes `--config ` to BOTH `act_runner register` and + # `act_runner daemon` when CONFIG_FILE is set (act_runner v0.6.1 scripts/run.sh). + - name: CONFIG_FILE + value: /etc/act_runner/config.yaml volumeMounts: - name: data mountPath: /data - name: certs mountPath: /certs readOnly: true + - name: config + mountPath: /etc/act_runner + readOnly: true + # Measured 2026-11-08 at capacity 1: peak CPU 0.24 core, peak working set 314Mi (30d) + # against the old 512Mi limit. It holds per-task state, log buffers and the embedded + # actions/cache server for every concurrent task, so the limit is doubled for capacity 2. resources: requests: cpu: 100m - memory: 128Mi + memory: 256Mi limits: - memory: 512Mi + memory: 1Gi volumes: - name: data persistentVolumeClaim: claimName: act-runner-data - name: certs emptyDir: {} + - name: config + configMap: + name: act-runner-config - name: dind-storage emptyDir: {} diff --git a/decisions/2026-11-08-runner-capacity-2.md b/decisions/2026-11-08-runner-capacity-2.md new file mode 100644 index 0000000..7e7788c --- /dev/null +++ b/decisions/2026-11-08-runner-capacity-2.md @@ -0,0 +1,155 @@ +--- +date: 2026-11-08 +branch: feat/runner-capacity-2 +title: act_runner runs two jobs at once, and dind is sized for two +--- + +## What changed + +`apps/wave-1/gitea-actions.yaml` only. That file is a **hand-owned** non-catalog unit +(`EXTRA_UNITS["gitea-actions"]` in `catalog/render.py`): the renderer only decides which wave +it lands in, it does not generate its bytes, and `ci/render-golden.sh` compares only files the +catalog renders. So it is edited in place, and no re-render was needed or run. + +1. **New ConfigMap `act-runner-config`** holding a config file with ONE key, + `runner.capacity: 2`, mounted read-only at `/etc/act_runner/` and selected with + `CONFIG_FILE=/etc/act_runner/config.yaml`. The image's `run.sh` (act_runner v0.6.1) turns + that into `--config` for both `register` and `daemon`. + - Deltas only, on evidence: v0.6.1 `LoadDefault()` unmarshals the file onto a zero Config and + then applies the same defaults as the no-file path, so the other settings are unchanged. + - The `generate-config` example was deliberately NOT used. Its `runner.labels` would override + the registered labels in `/data/.runner` and repoint `ubuntu-latest` at + `docker.gitea.com/runner-images`. Its `force_pull: true` is not the default. Its sample `envs` + would be injected into every job. + - With a file present, act_runner ignores the legacy `GITEA_RUNNER_CAPACITY` / + `_FILE` / `_ENVIRON` / `_ENV_FILE` / `GITEA_DEBUG` / `GITEA_TRACE` env vars. None of + them is set on this Deployment. +2. **Pod-template annotation `homelab.keithjackson.dev/act-runner-config-revision: "1"`.** + act_runner reads its config once, at start, so a change to the ConfigMap alone would never + take effect. Bump this annotation whenever you change only the ConfigMap. +3. **dind resources:** requests `cpu 100m -> 2` and `memory 256Mi -> 1Gi`; limit + `memory 2Gi -> 4Gi`. There is still no CPU limit, on purpose. +4. **runner resources:** request `memory 128Mi -> 256Mi`, limit `memory 512Mi -> 1Gi`. CPU stays + at `100m`. + +Why these sizes. Measured 2026-11-08 through Prometheus, at capacity 1: + +| container | peak CPU (1m rate, 30d) | peak working set (30d) | OOM events (7d) | +|---|---|---|---| +| dind | 0.69 core | 447 MiB | 0 | +| runner | 0.24 core | 314 MiB | 0 | + +At the moment of measurement the pod was idle (`top`: dind 1m/138Mi, runner 1m/83Mi), on +k3s-prod-2. + +- **CPU request 2.** Two jobs at today's peak is about 1.4 cores. The gate is being parallelised + in other work, so each job will use more than that. The request sets dind's CFS weight under + contention; at 100m it got a 1/20 share. With no limit, dind can still use spare cores on an + idle node. +- **Memory 1Gi / 4Gi.** Two jobs at today's peak is about 0.9 GiB. If dind is OOM-killed, + dockerd goes down and takes both running jobs with it, so the limit is about 4x that. +- **Runner limit 512Mi -> 1Gi.** Its 30-day peak was already 61% of the old limit, and with + capacity 2 it holds state for two tasks. + +Scheduling headroom: CPU requests per node were measured with `describe node`. The pod's total +CPU request goes from 200m to 2100m (+1.9 cores). + +| node | CPU requested now | after the change | allocatable | +|---|---|---|---| +| prod-2 (current) | 2945m | ~4845m (61%) | 8 | +| prod-1 | 3175m | ~5075m | 8 | +| prod-3 | 3840m | ~5740m (72%) | 8 | + +The pod fits on any of the three if the RWO PVC moves it. + +Concurrency semantics. Workflow `concurrency:` groups are enforced by Gitea on the server, not by +the runner, so capacity does not change them: + +- `gate.yml` `gate-${{ github.ref }}` with cancel-in-progress still cancels a superseded run on + the same ref. +- `renovate` (group `renovate`, no cancel) and `images-${{ github.ref }}` still serialise + themselves. + +What capacity 2 does change: runs on *different* refs (a PR gate next to a renovate or images +run, or two PRs) now run side by side instead of queueing. The required check +`gate / gate (pull_request)` is still one job per PR head, and gate.yml's other job +(`conformance`) is `if: false`, so it never runs. + +Both jobs share one dockerd. act_runner gives each task its own container prefix +(`GITEA-ACTIONS-TASK-`) and its own network, and no workflow uses fixed container names +or host ports. + +## What I verified + +- **Source of truth.** `catalog/render.py` `EXTRA_UNITS` places `gitea-actions.yaml` by + anchor/offset only. `ci/render-golden.sh` iterates over rendered files only, so this file is + outside render==golden. No `--emit-waves` was run. +- **act_runner v0.6.1 behaviour, read in its source** (tag v0.6.1): + - `config.LoadDefault()` treats a partial file as described above. + - `daemon.go`: config labels override `.runner` labels only when non-empty. + - `poller.go`: a semaphore of `cfg.Runner.Capacity` bounds concurrent tasks. + - `runner.go`: per-task container name prefix. + - The image's `run.sh`, read from `gitea/act_runner:0.6.1`, honours `CONFIG_FILE`. +- **The live pod has no config file today.** `/data` holds only `.runner` and `lost+found`. + `act_runner --version` reports v0.6.1. No LimitRange or ResourceQuota exists in + `gitea-actions`. +- **Resource numbers.** All figures above come from read-only `kubectl top`, `describe node` and + Prometheus queries against k3s-prod. +- **dind disk.** The emptyDir sits on the node's 11.8G ephemeral filesystem: 4.0G used, 7.2G + free. Image cache is 2.18GB, which is the one `catthehacker/ubuntu:act-22.04` image. Two + concurrent jobs share that image. +- **Embedded config.** It parses as YAML to `{'runner': {'capacity': 2}}`. The only `$` in the + manifest is the existing `${BASE_DOMAIN}`, so nothing new is exposed to Flux postBuild + envsubst. +- **Approval policy.** `apps/wave-1/` matches no `sensitive_paths` glob in + `infra/gitea/approval-policy.json`, so this PR merges on a green gate alone. +- **Local gate.** Pinned toolbox, run from the worktree on the committed tree. The rc and skipped + stages are given in the PR description. + +## What I did NOT verify + +- **UNVERIFIED: that the runner actually starts with the config and runs two jobs at once.** + Nothing is applied until Flux reconciles `main`. +- **Capacity cannot be seen through Gitea's API.** act_runner v0.6.1 does not report capacity to + Gitea: `Declare` sends only version and labels. `GET /api/v1/admin/actions/runners` therefore + cannot show "capacity 2". Concurrency has to be observed instead. +- **UNVERIFIED: whether the parallelised gate fits in these sizes.** That gate has not landed. + Re-measure after it does, with + `max_over_time(sum by (container)(rate(container_cpu_usage_seconds_total{namespace="gitea-actions"}[1m]))[7d:30s])`. +- **Rollout cancels in-flight jobs.** The rollout restarts the runner pod (strategy Recreate, + `shutdown_timeout: 0s`), and any job running at that moment is cancelled, possibly including + the `push`-to-main gate run triggered by this very merge. Re-run it if that happens. +- **dind disk is not bounded by a limit.** With two jobs it holds two jobs' workspaces and layers + on the same 11.8G node filesystem. I did not add an `ephemeral-storage` limit, because an + eviction would kill both jobs. Watch kubelet DiskPressure on the runner's node. +- **Stale comments in sensitive paths.** `.gitea/workflows/gate.yml` (line ~52) and + `.gitea/workflows/README.md` (line ~86) still say "`act_runner` runs at capacity 1". I left + them alone because `.gitea/**` is a sensitive path and editing them would put this PR behind + human approval. Fix them in a separate PR. + +### Operator post-merge checklist + +1. Flux applied it: + `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-1 -o jsonpath='{.status.lastAppliedRevision}'` + equals the merge commit. +2. The pod restarted with the config: + - `kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner` + shows a new pod, 2/2 Running. + - `kubectl ... exec deploy/act-runner -c runner -- cat /etc/act_runner/config.yaml` shows + `capacity: 2`. + - `kubectl ... exec deploy/act-runner -c runner -- ps` shows `act_runner daemon --config /etc/act_runner/config.yaml`. + - The logs show "declare successfully" with the two expected labels (`ubuntu-latest`, + `homelab` -> `catthehacker/ubuntu:act-22.04`). That confirms the labels did not change. +3. The runner is registered and online: + - `GET /api/v1/admin/actions/runners` lists `incluster-dind`, online, with the same ID as + before (no duplicate row). + - Capacity is NOT shown there; see above. +4. Concurrency is real: trigger two runs on different refs (e.g. two PR gates, or a PR gate plus a + `workflow_dispatch` of renovate). Then either: + - in `GET /api/v1/repos/Home-Network/home-network-k3s-impl/actions/tasks`, confirm their + `run_started_at` .. `updated_at` windows overlap; or + - check + `docker --host=tcp://127.0.0.1:2376 --tlsverify ... ps` in the dind container lists two + `GITEA-ACTIONS-TASK-*` container sets at once. +5. Resources: `kubectl --context k3s-prod -n gitea-actions describe pod` shows dind requests + cpu 2 / memory 1Gi, limit memory 4Gi, and runner limit memory 1Gi.