Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

feat(gitea-actions): act_runner capacity 2, dind sized for two concurrent jobs
gate / gate (pull_request) Successful in 2m59s
Details
gate / conformance (pull_request) Skipped
Details

Browse Source 
The in-cluster runner ran at the default capacity 1 with no config file, so
gate / images / renovate runs on different refs queued behind each other
(29-759s waits; run 472 waited 759s behind a renovate burst).

- ConfigMap act-runner-config with ONLY runner.capacity: 2, selected via
  CONFIG_FILE (honoured by the v0.6.1 image's run.sh). v0.6.1 LoadDefault()
  applies the same defaults to a partial file as to no file; the
  generate-config example is deliberately not used (its labels would override
  the registered ones, force_pull differs from the default).
- Pod-template config-revision annotation so ConfigMap-only edits can roll it.
- dind: cpu request 100m -> 2 (no limit), memory 256Mi/2Gi -> 1Gi/4Gi.
  Measured at capacity 1: peak 0.69 core, 447Mi working set (30d), 0 OOMs.
- runner: memory 128Mi/512Mi -> 256Mi/1Gi (peak 314Mi at capacity 1).

apps/wave-1/gitea-actions.yaml is a hand-owned EXTRA_UNITS file (the
renderer only places it in a wave), so it is the source, not generated output.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed 2026-11-08 00:09:08 -06:00
1 parent e8d26669a4
commit b548224db8
2 files changed
+224 -5
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

apps/wave-1/gitea-actions.yaml
+69 -5
View File
@@ -77,6 +77,42 @@ spec:
requests:
storage: 1Gi
---
# act_runner config file — DELTAS ONLY. act_runner v0.6.1 LoadDefault()
# (internal/pkg/config/config.go) unmarshals this file onto a ZERO Config and then fills every
# unset field with the same default it uses when there is no file at all, so a file holding
# only `runner.capacity` changes exactly that and nothing else. Do NOT paste in the output of
# `act_runner generate-config`: that example is NOT the defaults. Its `runner.labels` would
# OVERRIDE the registered labels in /data/.runner (daemon.go: config labels win when non-empty)
# and repoint ubuntu-latest at a different job image; its `force_pull: true` differs from the
# no-file default (false); and its sample `envs` would be injected into every job.
#
# Using a file also makes act_runner IGNORE the legacy GITEA_RUNNER_CAPACITY / _FILE /
# _ENVIRON / _ENV_FILE / GITEA_DEBUG / GITEA_TRACE env vars (deprecated.go). None of them is
# set on this Deployment; the GITEA_INSTANCE_URL / _REGISTRATION_TOKEN / _NAME / _LABELS env
# below is consumed by the image's run.sh for registration, not by that legacy path.
#
# A ConfigMap edit alone does NOT restart the runner (it reads the file once at start). If a
# future change touches ONLY this ConfigMap, also bump the Deployment's pod template (e.g. the
# config-revision annotation below) or the new value stays inert until the next restart.
apiVersion: v1
kind: ConfigMap
metadata:
name: act-runner-config
namespace: gitea-actions
labels:
app.kubernetes.io/name: act-runner
data:
config.yaml: |
runner:
# Two concurrent jobs (default 1). Measured 2026-11-08: queue wait ~1s normally, but
# 29-759s whenever gate / images / renovate runs collide (run 472 waited 759s behind a
# renovate burst). Both jobs share the ONE dind daemon below; act_runner names each job's
# containers GITEA-ACTIONS-TASK-<task id> and creates a network per job, so they do not
# collide. Workflow `concurrency:` groups are enforced by Gitea server-side and are
# unaffected: gate.yml's gate-<ref> group still cancels a superseded run on the SAME ref,
# and renovate's group still serialises renovate. The dind resources are sized for TWO jobs.
capacity: 2
---
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -95,6 +131,10 @@ spec:
metadata:
labels:
app.kubernetes.io/name: act-runner
annotations:
# Bump when act-runner-config changes without any other pod-template change, so the
# Deployment rolls and act_runner re-reads its config (see the ConfigMap header).
homelab.keithjackson.dev/act-runner-config-revision: "1" # capacity 2
spec:
# CI holds no cluster credentials — do not mount a SA token into the runner or
# (via the shared netns) the job containers (R-SEC-02, docs/14).
@@ -146,12 +186,23 @@ spec:
mountPath: /certs
- name: dind-storage
mountPath: /var/lib/docker
# Sized for runner.capacity: 2. Every job's containers run UNDER this daemon, so this
# container's cgroup carries the whole CI workload. Measured 2026-11-08 (Prometheus,
# capacity 1): peak CPU 0.69 core (1m rate, 30d), peak working set 447Mi (30d), 0 OOMs.
# cpu request 2: two jobs at today's peak is ~1.4 core, and the gate is being
# parallelised, so per-job demand rises. The request is what the CFS share is
# weighted by under contention (100m was a 1/20 share); NO cpu limit, deliberately,
# so an idle node's spare cores are still usable. k3s-prod-2 (8 CPU) had 2945m
# requested; +1.9 core = ~4.85 (61%), and prod-1/prod-3 fit too if it reschedules.
# memory request 1Gi / limit 4Gi: two jobs at today's peak is ~0.9Gi. An OOM kill here
# takes down dockerd and with it BOTH running jobs, so the limit carries ~4x headroom
# over that for the parallelised gate. Nodes have ~31Gi allocatable.
resources:
requests:
cpu: 100m
memory: 256Mi
cpu: "2"
memory: 1Gi
limits:
memory: 2Gi
memory: 4Gi
startupProbe:
exec:
command:
@@ -195,23 +246,36 @@ spec:
value: /certs/client
- name: DOCKER_TLS_VERIFY
value: "1"
# The image's run.sh passes `--config <this>` to BOTH `act_runner register` and
# `act_runner daemon` when CONFIG_FILE is set (act_runner v0.6.1 scripts/run.sh).
- name: CONFIG_FILE
value: /etc/act_runner/config.yaml
volumeMounts:
- name: data
mountPath: /data
- name: certs
mountPath: /certs
readOnly: true
- name: config
mountPath: /etc/act_runner
readOnly: true
# Measured 2026-11-08 at capacity 1: peak CPU 0.24 core, peak working set 314Mi (30d)
# against the old 512Mi limit. It holds per-task state, log buffers and the embedded
# actions/cache server for every concurrent task, so the limit is doubled for capacity 2.
resources:
requests:
cpu: 100m
memory: 128Mi
memory: 256Mi
limits:
memory: 512Mi
memory: 1Gi
volumes:
- name: data
persistentVolumeClaim:
claimName: act-runner-data
- name: certs
emptyDir: {}
- name: config
configMap:
name: act-runner-config
- name: dind-storage
emptyDir: {}
decisions/2026-11-08-runner-capacity-2.md
+155
View File
@@ -0,0 +1,155 @@
---
date: 2026-11-08
branch: feat/runner-capacity-2
title: act_runner runs two jobs at once, and dind is sized for two
---
## What changed
`apps/wave-1/gitea-actions.yaml` only. That file is a **hand-owned** non-catalog unit
(`EXTRA_UNITS["gitea-actions"]` in `catalog/render.py`): the renderer only decides which wave
it lands in, it does not generate its bytes, and `ci/render-golden.sh` compares only files the
catalog renders. So it is edited in place, and no re-render was needed or run.
1. **New ConfigMap `act-runner-config`** holding a config file with ONE key,
`runner.capacity: 2`, mounted read-only at `/etc/act_runner/` and selected with
`CONFIG_FILE=/etc/act_runner/config.yaml`. The image's `run.sh` (act_runner v0.6.1) turns
that into `--config` for both `register` and `daemon`.
- Deltas only, on evidence: v0.6.1 `LoadDefault()` unmarshals the file onto a zero Config and
then applies the same defaults as the no-file path, so the other settings are unchanged.
- The `generate-config` example was deliberately NOT used. Its `runner.labels` would override
the registered labels in `/data/.runner` and repoint `ubuntu-latest` at
`docker.gitea.com/runner-images`. Its `force_pull: true` is not the default. Its sample `envs`
would be injected into every job.
- With a file present, act_runner ignores the legacy `GITEA_RUNNER_CAPACITY` /
`_FILE` / `_ENVIRON` / `_ENV_FILE` / `GITEA_DEBUG` / `GITEA_TRACE` env vars. None of
them is set on this Deployment.
2. **Pod-template annotation `homelab.keithjackson.dev/act-runner-config-revision: "1"`.**
act_runner reads its config once, at start, so a change to the ConfigMap alone would never
take effect. Bump this annotation whenever you change only the ConfigMap.
3. **dind resources:** requests `cpu 100m -> 2` and `memory 256Mi -> 1Gi`; limit
`memory 2Gi -> 4Gi`. There is still no CPU limit, on purpose.
4. **runner resources:** request `memory 128Mi -> 256Mi`, limit `memory 512Mi -> 1Gi`. CPU stays
at `100m`.
Why these sizes. Measured 2026-11-08 through Prometheus, at capacity 1:
| container | peak CPU (1m rate, 30d) | peak working set (30d) | OOM events (7d) |
|---|---|---|---|
| dind | 0.69 core | 447 MiB | 0 |
| runner | 0.24 core | 314 MiB | 0 |
At the moment of measurement the pod was idle (`top`: dind 1m/138Mi, runner 1m/83Mi), on
k3s-prod-2.
- **CPU request 2.** Two jobs at today's peak is about 1.4 cores. The gate is being parallelised
in other work, so each job will use more than that. The request sets dind's CFS weight under
contention; at 100m it got a 1/20 share. With no limit, dind can still use spare cores on an
idle node.
- **Memory 1Gi / 4Gi.** Two jobs at today's peak is about 0.9 GiB. If dind is OOM-killed,
dockerd goes down and takes both running jobs with it, so the limit is about 4x that.
- **Runner limit 512Mi -> 1Gi.** Its 30-day peak was already 61% of the old limit, and with
capacity 2 it holds state for two tasks.
Scheduling headroom: CPU requests per node were measured with `describe node`. The pod's total
CPU request goes from 200m to 2100m (+1.9 cores).
| node | CPU requested now | after the change | allocatable |
|---|---|---|---|
| prod-2 (current) | 2945m | ~4845m (61%) | 8 |
| prod-1 | 3175m | ~5075m | 8 |
| prod-3 | 3840m | ~5740m (72%) | 8 |
The pod fits on any of the three if the RWO PVC moves it.
Concurrency semantics. Workflow `concurrency:` groups are enforced by Gitea on the server, not by
the runner, so capacity does not change them:
- `gate.yml` `gate-${{ github.ref }}` with cancel-in-progress still cancels a superseded run on
the same ref.
- `renovate` (group `renovate`, no cancel) and `images-${{ github.ref }}` still serialise
themselves.
What capacity 2 does change: runs on *different* refs (a PR gate next to a renovate or images
run, or two PRs) now run side by side instead of queueing. The required check
`gate / gate (pull_request)` is still one job per PR head, and gate.yml's other job
(`conformance`) is `if: false`, so it never runs.
Both jobs share one dockerd. act_runner gives each task its own container prefix
(`GITEA-ACTIONS-TASK-<task id>`) and its own network, and no workflow uses fixed container names
or host ports.
## What I verified
- **Source of truth.** `catalog/render.py` `EXTRA_UNITS` places `gitea-actions.yaml` by
anchor/offset only. `ci/render-golden.sh` iterates over rendered files only, so this file is
outside render==golden. No `--emit-waves` was run.
- **act_runner v0.6.1 behaviour, read in its source** (tag v0.6.1):
- `config.LoadDefault()` treats a partial file as described above.
- `daemon.go`: config labels override `.runner` labels only when non-empty.
- `poller.go`: a semaphore of `cfg.Runner.Capacity` bounds concurrent tasks.
- `runner.go`: per-task container name prefix.
- The image's `run.sh`, read from `gitea/act_runner:0.6.1`, honours `CONFIG_FILE`.
- **The live pod has no config file today.** `/data` holds only `.runner` and `lost+found`.
`act_runner --version` reports v0.6.1. No LimitRange or ResourceQuota exists in
`gitea-actions`.
- **Resource numbers.** All figures above come from read-only `kubectl top`, `describe node` and
Prometheus queries against k3s-prod.
- **dind disk.** The emptyDir sits on the node's 11.8G ephemeral filesystem: 4.0G used, 7.2G
free. Image cache is 2.18GB, which is the one `catthehacker/ubuntu:act-22.04` image. Two
concurrent jobs share that image.
- **Embedded config.** It parses as YAML to `{'runner': {'capacity': 2}}`. The only `$` in the
manifest is the existing `${BASE_DOMAIN}`, so nothing new is exposed to Flux postBuild
envsubst.
- **Approval policy.** `apps/wave-1/` matches no `sensitive_paths` glob in
`infra/gitea/approval-policy.json`, so this PR merges on a green gate alone.
- **Local gate.** Pinned toolbox, run from the worktree on the committed tree. The rc and skipped
stages are given in the PR description.
## What I did NOT verify
- **UNVERIFIED: that the runner actually starts with the config and runs two jobs at once.**
Nothing is applied until Flux reconciles `main`.
- **Capacity cannot be seen through Gitea's API.** act_runner v0.6.1 does not report capacity to
Gitea: `Declare` sends only version and labels. `GET /api/v1/admin/actions/runners` therefore
cannot show "capacity 2". Concurrency has to be observed instead.
- **UNVERIFIED: whether the parallelised gate fits in these sizes.** That gate has not landed.
Re-measure after it does, with
`max_over_time(sum by (container)(rate(container_cpu_usage_seconds_total{namespace="gitea-actions"}[1m]))[7d:30s])`.
- **Rollout cancels in-flight jobs.** The rollout restarts the runner pod (strategy Recreate,
`shutdown_timeout: 0s`), and any job running at that moment is cancelled, possibly including
the `push`-to-main gate run triggered by this very merge. Re-run it if that happens.
- **dind disk is not bounded by a limit.** With two jobs it holds two jobs' workspaces and layers
on the same 11.8G node filesystem. I did not add an `ephemeral-storage` limit, because an
eviction would kill both jobs. Watch kubelet DiskPressure on the runner's node.
- **Stale comments in sensitive paths.** `.gitea/workflows/gate.yml` (line ~52) and
`.gitea/workflows/README.md` (line ~86) still say "`act_runner` runs at capacity 1". I left
them alone because `.gitea/**` is a sensitive path and editing them would put this PR behind
human approval. Fix them in a separate PR.
### Operator post-merge checklist
1. Flux applied it:
`kubectl --context k3s-prod -n flux-system get kustomization apps-wave-1 -o jsonpath='{.status.lastAppliedRevision}'`
equals the merge commit.
2. The pod restarted with the config:
- `kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner`
shows a new pod, 2/2 Running.
- `kubectl ... exec deploy/act-runner -c runner -- cat /etc/act_runner/config.yaml` shows
`capacity: 2`.
- `kubectl ... exec deploy/act-runner -c runner -- ps` shows `act_runner daemon --config /etc/act_runner/config.yaml`.
- The logs show "declare successfully" with the two expected labels (`ubuntu-latest`,
`homelab` -> `catthehacker/ubuntu:act-22.04`). That confirms the labels did not change.
3. The runner is registered and online:
- `GET /api/v1/admin/actions/runners` lists `incluster-dind`, online, with the same ID as
before (no duplicate row).
- Capacity is NOT shown there; see above.
4. Concurrency is real: trigger two runs on different refs (e.g. two PR gates, or a PR gate plus a
`workflow_dispatch` of renovate). Then either:
- in `GET /api/v1/repos/Home-Network/home-network-k3s-impl/actions/tasks`, confirm their
`run_started_at` .. `updated_at` windows overlap; or
- check
`docker --host=tcp://127.0.0.1:2376 --tlsverify ... ps` in the dind container lists two
`GITEA-ACTIONS-TASK-*` container sets at once.
5. Resources: `kubectl --context k3s-prod -n gitea-actions describe pod` shows dind requests
cpu 2 / memory 1Gi, limit memory 4Gi, and runner limit memory 1Gi.