jakman85andClaude Opus 5.5
3ae7fb74f8
The act-runner's Docker cache was an unbounded emptyDir on the 11.6 GiB /var partition and evicted k3s-prod-2 on 2026-11-12. It moves to a generic ephemeral volume on longhorn-scratch (off /var, pod-lifetime, single replica), with a threshold-triggered docker prune and Velero opt-out. vuln-scanner's cache sizeLimit and ephemeral-storage limit are cut to fit the partition, and it prefers not to share a node with buildkitd. Co-Authored-By: Claude Opus 5.5 <[email protected]>
411 lines
21 KiB
YAML
411 lines
21 KiB
YAML
---
|
|
# Gitea Actions runner (act_runner) — Phase 6 CI/CD (WS-GITOSS).
|
|
#
|
|
# WHY A DEDICATED, NON-APP NAMESPACE:
|
|
# The runner's job executor is a Docker-in-Docker (DinD) sidecar that MUST run
|
|
# `privileged: true` (dockerd needs it). The Kyverno `restrict-host-namespaces`
|
|
# policy denies privileged containers, but it (and the egress-class / auth-mode /
|
|
# default-deny-baseline policies) scope ONLY to namespaces labeled
|
|
# homelab.keithjackson.dev/app-namespace: "true". This namespace deliberately does
|
|
# NOT carry that label — it is CI plumbing, not a user-facing app — so the privileged
|
|
# DinD sidecar is admitted WITHOUT a policy exception. (Verified: a privileged Pod in
|
|
# a non-app namespace passes server-side admission; Kyverno test fixture
|
|
# `platform-workload` is likewise Excluded from restrict-host-namespaces.)
|
|
#
|
|
# EXECUTOR MODEL — Docker executor + DinD sidecar (privileged):
|
|
# act_runner runs each job step inside a container it launches via a Docker daemon.
|
|
# That daemon is the co-located `dind` sidecar (shared pod netns → tcp://localhost:2376,
|
|
# TLS via a shared emptyDir of certs). Tradeoff vs the host/self-hosted executor: DinD
|
|
# needs `privileged`, but it gives clean per-job isolation and lets the workflow pick its
|
|
# own job image (the pinned toolchain), which the host executor cannot. Privileged is
|
|
# contained to this CI-only namespace and the runner mounts NO ServiceAccount token
|
|
# (automountServiceAccountToken: false) — CI holds no cluster credentials (R-SEC-02).
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: gitea-actions
|
|
labels:
|
|
# NOTE: intentionally NO homelab.keithjackson.dev/app-namespace label (see header).
|
|
homelab.keithjackson.dev/ci: act-runner
|
|
---
|
|
# Default-deny ingress; broad egress (a CI runner legitimately needs the internet:
|
|
# ghcr.io / github.com / docker hub / dl.k8s.io / pypi for job images + pinned tools,
|
|
# and the Gitea ingress VIP for registration + repo clone). Nothing connects INTO the
|
|
# runner, so ingress is denied entirely.
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: NetworkPolicy
|
|
metadata:
|
|
name: default-deny-ingress-allow-egress
|
|
namespace: gitea-actions
|
|
labels:
|
|
homelab.keithjackson.dev/baseline: default-deny
|
|
spec:
|
|
podSelector: {}
|
|
policyTypes:
|
|
- Ingress
|
|
- Egress
|
|
ingress: []
|
|
egress:
|
|
# DNS (cluster + split-horizon resolution of gitea.home.keithjackson.dev).
|
|
- to:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: kube-system
|
|
ports:
|
|
- protocol: UDP
|
|
port: 53
|
|
- protocol: TCP
|
|
port: 53
|
|
# Everything else the runner + DinD need: Gitea VIP + internet registries/tool hosts.
|
|
- {}
|
|
---
|
|
# Runner state (config.yaml + the `.runner` registration file) persists here so the
|
|
# runner registers exactly ONCE and reuses its identity across restarts (no duplicate
|
|
# offline rows in Gitea). RWO → Deployment strategy must be Recreate (fix/recreate-strategy-rwo).
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: act-runner-data
|
|
namespace: gitea-actions
|
|
labels:
|
|
app.kubernetes.io/name: act-runner
|
|
spec:
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
storageClassName: longhorn
|
|
resources:
|
|
requests:
|
|
storage: 1Gi
|
|
---
|
|
# act_runner config file — DELTAS ONLY. act_runner v0.6.1 LoadDefault()
|
|
# (internal/pkg/config/config.go) unmarshals this file onto a ZERO Config and then fills every
|
|
# unset field with the same default it uses when there is no file at all, so a file holding
|
|
# only `runner.capacity` changes exactly that and nothing else. Do NOT paste in the output of
|
|
# `act_runner generate-config`: that example is NOT the defaults. Its `runner.labels` would
|
|
# OVERRIDE the registered labels in /data/.runner (daemon.go: config labels win when non-empty)
|
|
# and repoint ubuntu-latest at a different job image; its `force_pull: true` differs from the
|
|
# no-file default (false); and its sample `envs` would be injected into every job.
|
|
#
|
|
# Using a file also makes act_runner IGNORE the legacy GITEA_RUNNER_CAPACITY / _FILE /
|
|
# _ENVIRON / _ENV_FILE / GITEA_DEBUG / GITEA_TRACE env vars (deprecated.go). None of them is
|
|
# set on this Deployment; the GITEA_INSTANCE_URL / _REGISTRATION_TOKEN / _NAME / _LABELS env
|
|
# below is consumed by the image's run.sh for registration, not by that legacy path.
|
|
#
|
|
# A ConfigMap edit alone does NOT restart the runner (it reads the file once at start). If a
|
|
# future change touches ONLY this ConfigMap, also bump the Deployment's pod template (e.g. the
|
|
# config-revision annotation below) or the new value stays inert until the next restart.
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: act-runner-config
|
|
namespace: gitea-actions
|
|
labels:
|
|
app.kubernetes.io/name: act-runner
|
|
data:
|
|
config.yaml: |
|
|
runner:
|
|
# Two concurrent jobs (default 1). Measured 2026-11-08: queue wait ~1s normally, but
|
|
# 29-759s whenever gate / images / renovate runs collide (run 472 waited 759s behind a
|
|
# renovate burst). Both jobs share the ONE dind daemon below; act_runner names each job's
|
|
# containers GITEA-ACTIONS-TASK-<task id> and creates a network per job, so they do not
|
|
# collide. Workflow `concurrency:` groups are enforced by Gitea server-side and are
|
|
# unaffected: gate.yml's gate-<ref> group still cancels a superseded run on the SAME ref,
|
|
# and renovate's group still serialises renovate. The dind resources are sized for TWO jobs.
|
|
capacity: 2
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: act-runner
|
|
namespace: gitea-actions
|
|
labels:
|
|
app.kubernetes.io/name: act-runner
|
|
spec:
|
|
replicas: 1
|
|
strategy:
|
|
type: Recreate
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: act-runner
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: act-runner
|
|
annotations:
|
|
# Bump when act-runner-config changes without any other pod-template change, so the
|
|
# Deployment rolls and act_runner re-reads its config (see the ConfigMap header).
|
|
homelab.keithjackson.dev/act-runner-config-revision: "1" # capacity 2
|
|
# Velero's daily schedule is opt-OUT (defaultVolumesToFsBackup: true) and its volume policy
|
|
# skips only emptyDir and NFS (platform/velero/volume-policies.yaml). dind-storage is now a
|
|
# Longhorn-backed generic ephemeral volume (see `volumes:`), which that policy does NOT
|
|
# match, so without this the nightly backup would push the whole Docker image cache to R2
|
|
# and race this pod's replacement (the PartiallyFailed failure that file documents). It is a
|
|
# disposable cache: nothing in it is state.
|
|
backup.velero.io/backup-volumes-excludes: dind-storage
|
|
spec:
|
|
# CI holds no cluster credentials — do not mount a SA token into the runner or
|
|
# (via the shared netns) the job containers (R-SEC-02, docs/14).
|
|
automountServiceAccountToken: false
|
|
# ---- Docker-in-Docker: the job executor daemon (privileged) -------------------
|
|
#
|
|
# NATIVE SIDECAR (k3s v1.36; sidecar containers are GA). It is an initContainer with
|
|
# `restartPolicy: Always`, which means: it starts BEFORE the regular containers, keeps
|
|
# running alongside them, and is terminated after them. kubelet holds `runner` until
|
|
# this container's startupProbe passes, so the ordering is enforced by the kubelet
|
|
# instead of hand-rolled shell in the runner.
|
|
#
|
|
# This REPLACES two bounded wait-loops that used to live in the runner's `command:`.
|
|
# Deleting them also deletes an envsubst hazard: clusters/base/apps-wave-1.yaml runs
|
|
# postBuild substitution over this ENTIRE manifest, comments included, and those loops
|
|
# depended on bare dollar-names surviving it untouched. There is now no shell here at
|
|
# all, so there is nothing to escape or mis-escape.
|
|
#
|
|
# WHY THE PROBE IS `exec docker ... info` AND NOT `tcpSocket`. A tcpSocket probe would
|
|
# re-open the exact gap the old Stage 1 existed to close: dockerd writes the TLS certs
|
|
# and only THEN begins listening, so the port can accept a connection while the certs
|
|
# are absent or incomplete -- and act_runner hard-exits on a missing
|
|
# /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs
|
|
# are usable and that the daemon answers.
|
|
#
|
|
# CORRECTION (measured 2026-09-29, after this conversion shipped): the commit that
|
|
# introduced this claimed the conversion would REMOVE the one-per-start dockerd line
|
|
# `http: TLS handshake error from 127.0.0.1: EOF`. IT DOES NOT. Exactly one still
|
|
# appears, immediately after "API listen on [::]:2376" -- it is now this probe's FIRST
|
|
# attempt, landing microseconds after dockerd begins listening and cut off by
|
|
# timeoutSeconds: 1. The next poll succeeds. It is cosmetic, exactly as cosmetic as it
|
|
# was before, and deliberately NOT chased: raising timeoutSeconds to hide one benign
|
|
# startup line would slow every subsequent poll. Left documented rather than silenced.
|
|
#
|
|
# periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
|
|
initContainers:
|
|
- name: dind
|
|
restartPolicy: Always # <- this field is what makes it a SIDECAR
|
|
image: docker:29-dind
|
|
securityContext:
|
|
privileged: true
|
|
env:
|
|
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
|
|
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
|
|
- name: DOCKER_TLS_CERTDIR
|
|
value: /certs
|
|
volumeMounts:
|
|
- name: certs
|
|
mountPath: /certs
|
|
- name: dind-storage
|
|
mountPath: /var/lib/docker
|
|
# Sized for runner.capacity: 2. Every job's containers run UNDER this daemon, so this
|
|
# container's cgroup carries the whole CI workload. Measured 2026-11-08 (Prometheus,
|
|
# capacity 1): peak CPU 0.69 core (1m rate, 30d), peak working set 447Mi (30d), 0 OOMs.
|
|
# cpu request 2: two jobs at today's peak is ~1.4 core, and the gate is being
|
|
# parallelised, so per-job demand rises. The request is what the CFS share is
|
|
# weighted by under contention (100m was a 1/20 share); NO cpu limit, deliberately,
|
|
# so an idle node's spare cores are still usable. k3s-prod-2 (8 CPU) had 2945m
|
|
# requested; +1.9 core = ~4.85 (61%), and prod-1/prod-3 fit too if it reschedules.
|
|
# memory request 1Gi / limit 4Gi: two jobs at today's peak is ~0.9Gi. An OOM kill here
|
|
# takes down dockerd and with it BOTH running jobs, so the limit carries ~4x headroom
|
|
# over that for the parallelised gate. Nodes have ~31Gi allocatable.
|
|
# ephemeral-storage: /var/lib/docker (images, job containers, their logs) is on the
|
|
# dind-storage PVC, NOT the node's /var, so this bounds only dockerd's own rootfs
|
|
# writes and its container log. Before 2026-11-14 the image cache was an unbounded
|
|
# emptyDir on /var (see `volumes:`).
|
|
resources:
|
|
requests:
|
|
cpu: "2"
|
|
memory: 1Gi
|
|
ephemeral-storage: 256Mi
|
|
limits:
|
|
memory: 4Gi
|
|
ephemeral-storage: 2Gi
|
|
startupProbe:
|
|
exec:
|
|
command:
|
|
- docker
|
|
- --host=tcp://127.0.0.1:2376
|
|
- --tlsverify
|
|
- --tlscacert=/certs/client/ca.pem
|
|
- --tlscert=/certs/client/cert.pem
|
|
- --tlskey=/certs/client/key.pem
|
|
- info
|
|
periodSeconds: 1
|
|
failureThreshold: 60
|
|
containers:
|
|
# ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
|
|
- name: runner
|
|
image: gitea/act_runner:0.6.1
|
|
# No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
|
|
# registers from env on first boot and runs `act_runner daemon`. The two wait-loops
|
|
# that used to live here are gone: kubelet now holds this container until the dind
|
|
# sidecar's startupProbe passes (see initContainers above).
|
|
env:
|
|
- name: GITEA_INSTANCE_URL
|
|
value: https://gitea.${BASE_DOMAIN}
|
|
- name: GITEA_RUNNER_REGISTRATION_TOKEN
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: gitea-actions-runner
|
|
key: token
|
|
- name: GITEA_RUNNER_NAME
|
|
value: incluster-dind
|
|
# runs-on label -> job image. Both map to the catthehacker "act" image, which
|
|
# carries bash/git/curl/tar/python3+pip/node so .gitea/workflows scripts can
|
|
# install the pinned CI toolchain (ci/versions.env) and actions/checkout runs.
|
|
- name: GITEA_RUNNER_LABELS
|
|
value: "ubuntu-latest:docker://ghcr.io/catthehacker/ubuntu:act-22.04,homelab:docker://ghcr.io/catthehacker/ubuntu:act-22.04"
|
|
# Reach the DinD daemon over the shared pod network namespace, TLS-verified
|
|
# against the certs the sidecar drops into the shared /certs emptyDir.
|
|
- name: DOCKER_HOST
|
|
value: tcp://localhost:2376
|
|
- name: DOCKER_CERT_PATH
|
|
value: /certs/client
|
|
- name: DOCKER_TLS_VERIFY
|
|
value: "1"
|
|
# The image's run.sh passes `--config <this>` to BOTH `act_runner register` and
|
|
# `act_runner daemon` when CONFIG_FILE is set (act_runner v0.6.1 scripts/run.sh).
|
|
- name: CONFIG_FILE
|
|
value: /etc/act_runner/config.yaml
|
|
volumeMounts:
|
|
- name: data
|
|
mountPath: /data
|
|
- name: certs
|
|
mountPath: /certs
|
|
readOnly: true
|
|
- name: config
|
|
mountPath: /etc/act_runner
|
|
readOnly: true
|
|
# Measured 2026-11-08 at capacity 1: peak CPU 0.24 core, peak working set 314Mi (30d)
|
|
# against the old 512Mi limit. It holds per-task state, log buffers and the embedded
|
|
# actions/cache server for every concurrent task, so the limit is doubled for capacity 2.
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 256Mi
|
|
ephemeral-storage: 128Mi
|
|
limits:
|
|
memory: 1Gi
|
|
ephemeral-storage: 1Gi
|
|
# ---- docker-prune: keeps the dind image cache bounded ----------------------------------
|
|
#
|
|
# dockerd never garbage-collects images. Every new renovate/toolbox digest (~2 GB each) used
|
|
# to stack up in /var/lib/docker until the node's /var filled: on 2026-11-12 that tipped
|
|
# k3s-prod-2 into DiskPressure and the act-runner pod was evicted 63 times. Moving the cache
|
|
# to its own volume (see `volumes:`) stops it hurting the node; this stops it filling that
|
|
# volume and failing jobs with ENOSPC instead.
|
|
#
|
|
# Every 10 minutes: if the volume is at or above 45% used, `docker system prune --all
|
|
# --force` removes every image, stopped container, unused network and build cache entry NOT
|
|
# in use by a container. Images a running job uses are never removed. 45% (not 85%) so the
|
|
# volume stays above the 50%-free gate of PersistentVolumeFillingUpPredicted
|
|
# (platform/monitoring/pvc-capacity-alerts.yaml) and a normal refill never pages anyone.
|
|
# Threshold-triggered, not age-based: `--filter until=` on images is image CREATION time, so
|
|
# an age-based prune would delete the long-lived job images (catthehacker, the toolbox) between
|
|
# every pair of jobs and force a re-pull each time.
|
|
#
|
|
# Accepted race: a prune landing in the milliseconds between a job's image pull and its
|
|
# container create removes that image and fails that one job. It needs the volume over the
|
|
# threshold AT that instant; a re-run passes.
|
|
#
|
|
# NO DOLLAR SIGNS, deliberately: clusters/base/apps-wave-1.yaml runs Flux postBuild
|
|
# substitution over this whole file, comments included, which rewrites shell variables and
|
|
# command substitutions (conformance/tests/l0-static/flux-postbuild-substitution). The
|
|
# percentage test is therefore
|
|
# a regex over `df -P` output rather than arithmetic on a captured value.
|
|
- name: docker-prune
|
|
image: docker:29-cli
|
|
command:
|
|
- /bin/sh
|
|
- -c
|
|
- |
|
|
trap 'exit 0' TERM INT
|
|
echo "docker-prune: started; prunes when /var/lib/docker is at or above 45% used"
|
|
while true; do
|
|
if df -P /var/lib/docker | grep -qE ' (4[5-9]|[5-9][0-9]|100)% '; then
|
|
echo "docker-prune: threshold reached, before:"
|
|
df -P /var/lib/docker | tail -n 1
|
|
docker system prune --all --force || echo "docker-prune: prune failed; retrying next cycle"
|
|
echo "docker-prune: after:"
|
|
df -P /var/lib/docker | tail -n 1
|
|
fi
|
|
sleep 600 &
|
|
wait
|
|
done
|
|
env:
|
|
- name: DOCKER_HOST
|
|
value: tcp://localhost:2376
|
|
- name: DOCKER_CERT_PATH
|
|
value: /certs/client
|
|
- name: DOCKER_TLS_VERIFY
|
|
value: "1"
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
volumeMounts:
|
|
- name: certs
|
|
mountPath: /certs
|
|
readOnly: true
|
|
# Mounted only so `df` can read the volume's usage; the prune itself goes through the
|
|
# daemon's API.
|
|
- name: dind-storage
|
|
mountPath: /var/lib/docker
|
|
readOnly: true
|
|
resources:
|
|
requests:
|
|
cpu: 10m
|
|
memory: 32Mi
|
|
ephemeral-storage: 16Mi
|
|
limits:
|
|
memory: 128Mi
|
|
ephemeral-storage: 128Mi
|
|
volumes:
|
|
- name: data
|
|
persistentVolumeClaim:
|
|
claimName: act-runner-data
|
|
- name: certs
|
|
emptyDir:
|
|
sizeLimit: 16Mi # a handful of PEM files
|
|
- name: config
|
|
configMap:
|
|
name: act-runner-config
|
|
# The Docker daemon's data root: every job image and job container. OFF the node's /var.
|
|
#
|
|
# WHY NOT an emptyDir (what this was until 2026-11-14). kubelet emptyDirs live on /var, a
|
|
# separate 11.8 GiB partition on every node, and containerd's image store is NOT there
|
|
# (k3s --data-dir is /data/k3s, infra/ansible/inventory/group_vars/k3s_prod/vars.yml), so
|
|
# kubelet image GC never frees /var. Measured 2026-11-14: this emptyDir held 3.75 GiB on
|
|
# k3s-prod-2 (renovate 1.99 GB + toolbox 2.04 GB) with ~1.9 GiB of /var left; the
|
|
# 2026-11-12 eviction storm was it tipping the node into DiskPressure. A sizeLimit alone
|
|
# would only trade that for this pod evicting itself, and /var is too small to give the
|
|
# cache the room it needs next to vuln-scanner's and buildkit's.
|
|
#
|
|
# WHY a GENERIC EPHEMERAL volume on `longhorn-scratch` and not a standalone PVC.
|
|
# * Off /var: Longhorn's dataPath is /data/longhorn on the ~424 GB partition (~170 GB free
|
|
# per node, measured 2026-11-14).
|
|
# * Single replica (`longhorn-scratch`, dataLocality best-effort): a disposable cache gains
|
|
# nothing from 3x replication, and replica writes of multi-GB image pulls would land on
|
|
# the device etcd's WAL shares (docs/04 accepted risk). Same reasoning as CDI's scratch,
|
|
# which is what that class exists for.
|
|
# * Pod-lifetime, not persistent: the claim is created with the pod and deleted with it.
|
|
# That keeps this out of R-STATE-02 ("persistent storage MUST be replicated"), which a
|
|
# long-lived single-replica PVC would violate. It also means a replacement pod never
|
|
# waits on a lone replica stranded on a dead node: it gets a fresh, empty volume. The
|
|
# cost, accepted: the cache is cold after every pod replacement (~4 GB of pulls), which
|
|
# is exactly what the emptyDir already did.
|
|
# * Bounded twice: the claim size is a hard ceiling (ENOSPC for jobs, never a node
|
|
# eviction), and the docker-prune container keeps it under 45%.
|
|
# Velero: excluded by the pod annotation above AND the claim label below.
|
|
- name: dind-storage
|
|
ephemeral:
|
|
volumeClaimTemplate:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: act-runner
|
|
velero.io/exclude-from-backup: "true"
|
|
spec:
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
storageClassName: longhorn-scratch
|
|
resources:
|
|
requests:
|
|
storage: 30Gi
|