home-network-k3s-impl/decisions/2026-11-08-runner-capacity-2.md

date, branch, title

date	branch	title
2026-11-08	feat/runner-capacity-2	act_runner runs two jobs at once, and dind is sized for two

What changed

apps/wave-1/gitea-actions.yaml only. That file is a hand-owned non-catalog unit (EXTRA_UNITS["gitea-actions"] in catalog/render.py): the renderer only decides which wave it lands in, it does not generate its bytes, and ci/render-golden.sh compares only files the catalog renders. So it is edited in place, and no re-render was needed or run.

1. New ConfigMap act-runner-config holding a config file with ONE key, runner.capacity: 2, mounted read-only at /etc/act_runner/ and selected with CONFIG_FILE=/etc/act_runner/config.yaml. The image's run.sh (act_runner v0.6.1) turns that into --config for both register and daemon.
   • Deltas only, on evidence: v0.6.1 LoadDefault() unmarshals the file onto a zero Config and then applies the same defaults as the no-file path, so the other settings are unchanged.
   • The generate-config example was deliberately NOT used. Its runner.labels would override the registered labels in /data/.runner and repoint ubuntu-latest at docker.gitea.com/runner-images. Its force_pull: true is not the default. Its sample envs would be injected into every job.
   • With a file present, act_runner ignores the legacy GITEA_RUNNER_CAPACITY / _FILE / _ENVIRON / _ENV_FILE / GITEA_DEBUG / GITEA_TRACE env vars. None of them is set on this Deployment.
2. Pod-template annotation homelab.keithjackson.dev/act-runner-config-revision: "1". act_runner reads its config once, at start, so a change to the ConfigMap alone would never take effect. Bump this annotation whenever you change only the ConfigMap.
3. dind resources: requests cpu 100m -> 2 and memory 256Mi -> 1Gi; limit memory 2Gi -> 4Gi. There is still no CPU limit, on purpose.
4. runner resources: request memory 128Mi -> 256Mi, limit memory 512Mi -> 1Gi. CPU stays at 100m.

Why these sizes. Measured 2026-11-08 through Prometheus, at capacity 1:

container	peak CPU (1m rate, 30d)	peak working set (30d)	OOM events (7d)
dind	0.69 core	447 MiB	0
runner	0.24 core	314 MiB	0

At the moment of measurement the pod was idle (top: dind 1m/138Mi, runner 1m/83Mi), on k3s-prod-2.

• CPU request 2. Two jobs at today's peak is about 1.4 cores. The gate is being parallelised in other work, so each job will use more than that. The request sets dind's CFS weight under contention; at 100m it got a 1/20 share. With no limit, dind can still use spare cores on an idle node.
• Memory 1Gi / 4Gi. Two jobs at today's peak is about 0.9 GiB. If dind is OOM-killed, dockerd goes down and takes both running jobs with it, so the limit is about 4x that.
• Runner limit 512Mi -> 1Gi. Its 30-day peak was already 61% of the old limit, and with capacity 2 it holds state for two tasks.

Scheduling headroom: CPU requests per node were measured with describe node. The pod's total CPU request goes from 200m to 2100m (+1.9 cores).

node	CPU requested now	after the change	allocatable
prod-2 (current)	2945m	~4845m (61%)	8
prod-1	3175m	~5075m	8
prod-3	3840m	~5740m (72%)	8

The pod fits on any of the three if the RWO PVC moves it.

Concurrency semantics. Workflow concurrency: groups are enforced by Gitea on the server, not by the runner, so capacity does not change them:

• gate.yml gate-${{ github.ref }} with cancel-in-progress still cancels a superseded run on the same ref.
• renovate (group renovate, no cancel) and images-${{ github.ref }} still serialise themselves.

What capacity 2 does change: runs on different refs (a PR gate next to a renovate or images run, or two PRs) now run side by side instead of queueing. The required check gate / gate (pull_request) is still one job per PR head, and gate.yml's other job (conformance) is if: false, so it never runs.

Both jobs share one dockerd. act_runner gives each task its own container prefix (GITEA-ACTIONS-TASK-<task id>) and its own network, and no workflow uses fixed container names or host ports.

What I verified

• Source of truth. catalog/render.py EXTRA_UNITS places gitea-actions.yaml by anchor/offset only. ci/render-golden.sh iterates over rendered files only, so this file is outside render==golden. No --emit-waves was run.
• act_runner v0.6.1 behaviour, read in its source (tag v0.6.1):
  • config.LoadDefault() treats a partial file as described above.
  • daemon.go: config labels override .runner labels only when non-empty.
  • poller.go: a semaphore of cfg.Runner.Capacity bounds concurrent tasks.
  • runner.go: per-task container name prefix.
  • The image's run.sh, read from gitea/act_runner:0.6.1, honours CONFIG_FILE.
• The live pod has no config file today. /data holds only .runner and lost+found. act_runner --version reports v0.6.1. No LimitRange or ResourceQuota exists in gitea-actions.
• Resource numbers. All figures above come from read-only kubectl top, describe node and Prometheus queries against k3s-prod.
• dind disk. The emptyDir sits on the node's 11.8G ephemeral filesystem: 4.0G used, 7.2G free. Image cache is 2.18GB, which is the one catthehacker/ubuntu:act-22.04 image. Two concurrent jobs share that image.
• Embedded config. It parses as YAML to {'runner': {'capacity': 2}}. The only $ in the manifest is the existing ${BASE_DOMAIN}, so nothing new is exposed to Flux postBuild envsubst.
• Approval policy. apps/wave-1/ matches no sensitive_paths glob in infra/gitea/approval-policy.json, so this PR merges on a green gate alone.
• Local gate. Pinned toolbox, run from the worktree on the committed tree. The rc and skipped stages are given in the PR description.

What I did NOT verify

• UNVERIFIED: that the runner actually starts with the config and runs two jobs at once. Nothing is applied until Flux reconciles main.
• Capacity cannot be seen through Gitea's API. act_runner v0.6.1 does not report capacity to Gitea: Declare sends only version and labels. GET /api/v1/admin/actions/runners therefore cannot show "capacity 2". Concurrency has to be observed instead.
• UNVERIFIED: whether the parallelised gate fits in these sizes. That gate has not landed. Re-measure after it does, with max_over_time(sum by (container)(rate(container_cpu_usage_seconds_total{namespace="gitea-actions"}[1m]))[7d:30s]).
• Rollout cancels in-flight jobs. The rollout restarts the runner pod (strategy Recreate, shutdown_timeout: 0s), and any job running at that moment is cancelled, possibly including the push-to-main gate run triggered by this very merge. Re-run it if that happens.
• dind disk is not bounded by a limit. With two jobs it holds two jobs' workspaces and layers on the same 11.8G node filesystem. I did not add an ephemeral-storage limit, because an eviction would kill both jobs. Watch kubelet DiskPressure on the runner's node.
• Stale comments in sensitive paths. .gitea/workflows/gate.yml (line ~52) and .gitea/workflows/README.md (line ~86) still say "act_runner runs at capacity 1". I left them alone because .gitea/** is a sensitive path and editing them would put this PR behind human approval. Fix them in a separate PR.

Operator post-merge checklist

1. Flux applied it: kubectl --context k3s-prod -n flux-system get kustomization apps-wave-1 -o jsonpath='{.status.lastAppliedRevision}' equals the merge commit.
2. The pod restarted with the config:
   • kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner shows a new pod, 2/2 Running.
   • kubectl ... exec deploy/act-runner -c runner -- cat /etc/act_runner/config.yaml shows capacity: 2.
   • kubectl ... exec deploy/act-runner -c runner -- ps shows act_runner daemon --config /etc/act_runner/config.yaml.
   • The logs show "declare successfully" with the two expected labels (ubuntu-latest, homelab -> catthehacker/ubuntu:act-22.04). That confirms the labels did not change.
3. The runner is registered and online:
   • GET /api/v1/admin/actions/runners lists incluster-dind, online, with the same ID as before (no duplicate row).
   • Capacity is NOT shown there; see above.
4. Concurrency is real: trigger two runs on different refs (e.g. two PR gates, or a PR gate plus a workflow_dispatch of renovate). Then either:
   • in GET /api/v1/repos/Home-Network/home-network-k3s-impl/actions/tasks, confirm their run_started_at .. updated_at windows overlap; or
   • check docker --host=tcp://127.0.0.1:2376 --tlsverify ... ps in the dind container lists two GITEA-ACTIONS-TASK-* container sets at once.
5. Resources: kubectl --context k3s-prod -n gitea-actions describe pod shows dind requests cpu 2 / memory 1Gi, limit memory 4Gi, and runner limit memory 1Gi.