home-network-k3s-impl/decisions/2026-11-08-runner-capacity-2.md

---
date: 2026-11-08
branch: feat/runner-capacity-2
title: act_runner runs two jobs at once, and dind is sized for two
---

## What changed

`apps/wave-1/gitea-actions.yaml` only. That file is a **hand-owned** non-catalog unit
(`EXTRA_UNITS["gitea-actions"]` in `catalog/render.py`): the renderer only decides which wave
it lands in, it does not generate its bytes, and `ci/render-golden.sh` compares only files the
catalog renders. So it is edited in place, and no re-render was needed or run.

1. **New ConfigMap `act-runner-config`** holding a config file with ONE key,
   `runner.capacity: 2`, mounted read-only at `/etc/act_runner/` and selected with
   `CONFIG_FILE=/etc/act_runner/config.yaml`. The image's `run.sh` (act_runner v0.6.1) turns
   that into `--config` for both `register` and `daemon`.
   - Deltas only, on evidence: v0.6.1 `LoadDefault()` unmarshals the file onto a zero Config and
     then applies the same defaults as the no-file path, so the other settings are unchanged.
   - The `generate-config` example was deliberately NOT used. Its `runner.labels` would override
     the registered labels in `/data/.runner` and repoint `ubuntu-latest` at
     `docker.gitea.com/runner-images`. Its `force_pull: true` is not the default. Its sample `envs`
     would be injected into every job.
   - With a file present, act_runner ignores the legacy `GITEA_RUNNER_CAPACITY` /
     `_FILE` / `_ENVIRON` / `_ENV_FILE` / `GITEA_DEBUG` / `GITEA_TRACE` env vars. None of
     them is set on this Deployment.
2. **Pod-template annotation `homelab.keithjackson.dev/act-runner-config-revision: "1"`.**
   act_runner reads its config once, at start, so a change to the ConfigMap alone would never
   take effect. Bump this annotation whenever you change only the ConfigMap.
3. **dind resources:** requests `cpu 100m -> 2` and `memory 256Mi -> 1Gi`; limit
   `memory 2Gi -> 4Gi`. There is still no CPU limit, on purpose.
4. **runner resources:** request `memory 128Mi -> 256Mi`, limit `memory 512Mi -> 1Gi`. CPU stays
   at `100m`.

Why these sizes. Measured 2026-11-08 through Prometheus, at capacity 1:

| container | peak CPU (1m rate, 30d) | peak working set (30d) | OOM events (7d) |
|---|---|---|---|
| dind | 0.69 core | 447 MiB | 0 |
| runner | 0.24 core | 314 MiB | 0 |

At the moment of measurement the pod was idle (`top`: dind 1m/138Mi, runner 1m/83Mi), on
k3s-prod-2.

- **CPU request 2.** Two jobs at today's peak is about 1.4 cores. The gate is being parallelised
  in other work, so each job will use more than that. The request sets dind's CFS weight under
  contention; at 100m it got a 1/20 share. With no limit, dind can still use spare cores on an
  idle node.
- **Memory 1Gi / 4Gi.** Two jobs at today's peak is about 0.9 GiB. If dind is OOM-killed,
  dockerd goes down and takes both running jobs with it, so the limit is about 4x that.
- **Runner limit 512Mi -> 1Gi.** Its 30-day peak was already 61% of the old limit, and with
  capacity 2 it holds state for two tasks.

Scheduling headroom: CPU requests per node were measured with `describe node`. The pod's total
CPU request goes from 200m to 2100m (+1.9 cores).

| node | CPU requested now | after the change | allocatable |
|---|---|---|---|
| prod-2 (current) | 2945m | ~4845m (61%) | 8 |
| prod-1 | 3175m | ~5075m | 8 |
| prod-3 | 3840m | ~5740m (72%) | 8 |

The pod fits on any of the three if the RWO PVC moves it.

Concurrency semantics. Workflow `concurrency:` groups are enforced by Gitea on the server, not by
the runner, so capacity does not change them:

- `gate.yml` `gate-${{ github.ref }}` with cancel-in-progress still cancels a superseded run on
  the same ref.
- `renovate` (group `renovate`, no cancel) and `images-${{ github.ref }}` still serialise
  themselves.

What capacity 2 does change: runs on *different* refs (a PR gate next to a renovate or images
run, or two PRs) now run side by side instead of queueing. The required check
`gate / gate (pull_request)` is still one job per PR head, and gate.yml's other job
(`conformance`) is `if: false`, so it never runs.

Both jobs share one dockerd. act_runner gives each task its own container prefix
(`GITEA-ACTIONS-TASK-<task id>`) and its own network, and no workflow uses fixed container names
or host ports.

## What I verified

- **Source of truth.** `catalog/render.py` `EXTRA_UNITS` places `gitea-actions.yaml` by
  anchor/offset only. `ci/render-golden.sh` iterates over rendered files only, so this file is
  outside render==golden. No `--emit-waves` was run.
- **act_runner v0.6.1 behaviour, read in its source** (tag v0.6.1):
  - `config.LoadDefault()` treats a partial file as described above.
  - `daemon.go`: config labels override `.runner` labels only when non-empty.
  - `poller.go`: a semaphore of `cfg.Runner.Capacity` bounds concurrent tasks.
  - `runner.go`: per-task container name prefix.
  - The image's `run.sh`, read from `gitea/act_runner:0.6.1`, honours `CONFIG_FILE`.
- **The live pod has no config file today.** `/data` holds only `.runner` and `lost+found`.
  `act_runner --version` reports v0.6.1. No LimitRange or ResourceQuota exists in
  `gitea-actions`.
- **Resource numbers.** All figures above come from read-only `kubectl top`, `describe node` and
  Prometheus queries against k3s-prod.
- **dind disk.** The emptyDir sits on the node's 11.8G ephemeral filesystem: 4.0G used, 7.2G
  free. Image cache is 2.18GB, which is the one `catthehacker/ubuntu:act-22.04` image. Two
  concurrent jobs share that image.
- **Embedded config.** It parses as YAML to `{'runner': {'capacity': 2}}`. The only `$` in the
  manifest is the existing `${BASE_DOMAIN}`, so nothing new is exposed to Flux postBuild
  envsubst.
- **Approval policy.** `apps/wave-1/` matches no `sensitive_paths` glob in
  `infra/gitea/approval-policy.json`, so this PR merges on a green gate alone.
- **Local gate.** Pinned toolbox, run from the worktree on the committed tree. The rc and skipped
  stages are given in the PR description.

## What I did NOT verify

- **UNVERIFIED: that the runner actually starts with the config and runs two jobs at once.**
  Nothing is applied until Flux reconciles `main`.
- **Capacity cannot be seen through Gitea's API.** act_runner v0.6.1 does not report capacity to
  Gitea: `Declare` sends only version and labels. `GET /api/v1/admin/actions/runners` therefore
  cannot show "capacity 2". Concurrency has to be observed instead.
- **UNVERIFIED: whether the parallelised gate fits in these sizes.** That gate has not landed.
  Re-measure after it does, with
  `max_over_time(sum by (container)(rate(container_cpu_usage_seconds_total{namespace="gitea-actions"}[1m]))[7d:30s])`.
- **Rollout cancels in-flight jobs.** The rollout restarts the runner pod (strategy Recreate,
  `shutdown_timeout: 0s`), and any job running at that moment is cancelled, possibly including
  the `push`-to-main gate run triggered by this very merge. Re-run it if that happens.
- **dind disk is not bounded by a limit.** With two jobs it holds two jobs' workspaces and layers
  on the same 11.8G node filesystem. I did not add an `ephemeral-storage` limit, because an
  eviction would kill both jobs. Watch kubelet DiskPressure on the runner's node.
- **Stale comments in sensitive paths.** `.gitea/workflows/gate.yml` (line ~52) and
  `.gitea/workflows/README.md` (line ~86) still say "`act_runner` runs at capacity 1". I left
  them alone because `.gitea/**` is a sensitive path and editing them would put this PR behind
  human approval. Fix them in a separate PR.

### Operator post-merge checklist

1. Flux applied it:
   `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-1 -o jsonpath='{.status.lastAppliedRevision}'`
   equals the merge commit.
2. The pod restarted with the config:
   - `kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner`
     shows a new pod, 2/2 Running.
   - `kubectl ... exec deploy/act-runner -c runner -- cat /etc/act_runner/config.yaml` shows
     `capacity: 2`.
   - `kubectl ... exec deploy/act-runner -c runner -- ps` shows `act_runner daemon --config /etc/act_runner/config.yaml`.
   - The logs show "declare successfully" with the two expected labels (`ubuntu-latest`,
     `homelab` -> `catthehacker/ubuntu:act-22.04`). That confirms the labels did not change.
3. The runner is registered and online:
   - `GET /api/v1/admin/actions/runners` lists `incluster-dind`, online, with the same ID as
     before (no duplicate row).
   - Capacity is NOT shown there; see above.
4. Concurrency is real: trigger two runs on different refs (e.g. two PR gates, or a PR gate plus a
   `workflow_dispatch` of renovate). Then either:
   - in `GET /api/v1/repos/Home-Network/home-network-k3s-impl/actions/tasks`, confirm their
     `run_started_at` .. `updated_at` windows overlap; or
   - check
     `docker --host=tcp://127.0.0.1:2376 --tlsverify ... ps` in the dind container lists two
     `GITEA-ACTIONS-TASK-*` container sets at once.
5. Resources: `kubectl --context k3s-prod -n gitea-actions describe pod` shows dind requests
   cpu 2 / memory 1Gi, limit memory 4Gi, and runner limit memory 1Gi.